Extract every URL a domain ever exposed.

A fast Rust CLI that queries nine OSINT archives in parallel, then filters and validates what comes back.

urx · recon

$ urx dalfox.hahwul.com --providers wayback,otx --check-status
Domains         [====================] 1/1 processed
Wayback Machine [====================] done
OTX             [====================] done
Filtering       [====================] 199 URLs
Testing URLs    [====================] 199/199 complete
https://dalfox.hahwul.com/                         [200 OK]
https://dalfox.hahwul.com/.well-known/security.txt [200 OK]
https://dalfox.hahwul.com/llms.txt                 [200 OK]
https://dalfox.hahwul.com/sitemap.xml              [200 OK]
https://dalfox.hahwul.com/advanced/config/         [404 Not Found]
https://dalfox.hahwul.com/admin                    [404 Not Found]
199 URLs · 61 live · 138 gone

Nine sources, one command.

urx queries public web archives and threat-intel feeds at the same time, then merges and deduplicates the results. Five of them need no API key.

Wayback Machine wayback The Internet Archive CDX index. Deep historical coverage, enabled by default. Keyless

Common Crawl cc The monthly Common Crawl URL index. Keyless

OTX otx AlienVault Open Threat Exchange passive DNS and URLs. Keyless

Arquivo.pt arquivo The Portuguese web archive CDX index. Keyless

Urlscan urlscan Urlscan.io search. Works anonymously; a key only raises the rate limit. Keyless

VirusTotal vt URLs VirusTotal has observed for the domain. API key

ZoomEye zoomeye ZoomEye search results for the target. API key

GitHub github GitHub Code Search, for URLs committed into public repositories. API key

BeVigil bevigil Endpoints pulled out of unpacked Android apps. API key

Beyond the nine, urx also reads the target's own robots.txt and sitemap.xml, and will query any CDX server you point it at with --cdx-endpoint.

Built for the whole recon loop.

9 sources

Parallel collection

Async requests fan out to every enabled provider at once, then merge and deduplicate into a single URL set. Nothing waits on the slowest archive.

Advanced filtering

Filter by extension, pattern or preset. Control URL length and which host parts you keep.

$ urx target.com \ -e js,php \ --patterns api,v1

https://target.com/api/v1/auth.js https://target.com/api/config.php https://target.com/v1/users.js

URL validation

Check status codes, drop dead links, and pull more URLs out of the pages that answer.

Flexible output

Plain text, JSON, CSV or a ready-made wordlist, streamed to the console, a file, or the next tool in the pipe.

Caching and incremental scanning

A local SQLite or remote Redis cache skips domains you already covered, and incremental mode returns only what has appeared since the last run.

From a domain to a clean URL set.

One command runs the whole pipeline, and every stage maps to a flag you can tune.

--providers

Collect

Query every enabled archive in parallel and merge the results into one stream.

-e · --patterns

Filter

Keep only the extensions, patterns or presets you actually care about.

--check-status

Validate

Probe what is live, record the status, and mine responses for more links.

-f json · -o

Output

Write text, JSON, CSV or a wordlist to a file, or pipe it straight into the next tool.

Install urx and start collecting.

Available on Cargo, Homebrew and as a container image. No account, no key to get going.

$ cargo install urx Cargo

$ brew install urx Homebrew

$ docker pull ghcr.io/hahwul/urx Docker