Extract every URL a domain ever exposed.

A fast Rust CLI that pulls URLs from OSINT archives in parallel, then filters and validates them for recon.

Get started View on GitHub

7Sources

5Keyless

3Output formats

urx · recon

$ urx dalfox.hahwul.com --providers wayback,otx,vt,urlscan --check-status
Domains         [====================] 1/1 processed
Wayback Machine [====================] done
OTX             [====================] done
VirusTotal      [====================] done
Urlscan         [====================] done
Filtering       [====================] 2408 URLs
Testing URLs    [====================] 2408/2408 complete
https://dalfox.hahwul.com/                         [200 OK]
https://dalfox.hahwul.com/.well-known/security.txt [200 OK]
https://dalfox.hahwul.com/docs/getting-started/    [200 OK]
https://dalfox.hahwul.com/assets/js/main.min.js    [200 OK]
https://dalfox.hahwul.com/allposts.html            [404 Not Found]
... found 2408 URLs

Seven archives, one command.

urx queries public web archives and threat-intel feeds at the same time. Five work with no API key at all.

Keyless Wayback Machine --providers wayback The Internet Archive's CDX index. Deep historical coverage, on by default.

Keyless Common Crawl cc

Keyless OTX otx

Keyless Arquivo.pt arquivo

API key VirusTotal vt

Anonymous URLScan urlscan

API key ZoomEye zoomeye

Built for the whole recon loop.

7 sources

Parallel collection

Async requests fan out to every enabled provider at once, then merge and deduplicate into a single URL set. No waiting on one slow archive.

Advanced filtering

Filter by extension, pattern, or presets like no-images. Control URL length and host parts.

$ urx target.com \
-e js,php \
--patterns api,v1

https://target.com/api/v1/auth.js
https://target.com/api/config.php
https://target.com/v1/users.js

Flexible output

Plain text, JSON, or CSV. Stream to the console, a file, or a pipe.

URL validation

Check HTTP status codes, drop dead links, and extract more URLs from live pages.

Caching & incremental scanning

Local SQLite or remote Redis caching skips domains you already scanned. Incremental mode returns only the URLs discovered since last run.

Built in Rust

Async runtime, parallel requests, and a small footprint for maximum throughput.

From a domain to a clean URL set.

One command runs the full pipeline. Each stage maps to a flag you can tune.

--providers

Collect

Query every archive in parallel and merge the results into one stream.

-e · --patterns

Filter

Keep only the extensions, patterns, or presets you care about.

--check-status

Validate

Probe live URLs, check HTTP status, and pull extra links from responses.

-f json · -o

Output

Write text, JSON, or CSV to a file, or pipe it straight into your next tool.

Ready to launch

Install urx and start collecting.

Available on Cargo, Homebrew, and as a container image. No account, no key to get going.

$ cargo install urx Cargo

$ brew install urx Homebrew

$ docker pull ghcr.io/hahwul/urx Docker

Read the docs