Extract every URL a domain ever exposed.

A fast Rust CLI that queries up to nine OSINT sources in parallel, then filters and validates what comes back.

urx · recon

$ urx dalfox.hahwul.com --providers wayback,otx --check-status

Domains         [====================] 1/1 processed
Wayback Machine [====================] done
OTX             [====================] done
Robots.txt      [====================] done
Sitemap         [====================] done
Filtering       [====================] 199 URLs
Testing         [====================] 199/199 complete

https://dalfox.hahwul.com/                         [200 OK]
https://dalfox.hahwul.com/.well-known/security.txt [200 OK]
https://dalfox.hahwul.com/admin                    [404 Not Found]
https://dalfox.hahwul.com/advanced/config/         [404 Not Found]
https://dalfox.hahwul.com/llms.txt                 [200 OK]
https://dalfox.hahwul.com/sitemap.xml              [200 OK]

Nine sources, one command.

urx queries public web archives and threat-intel feeds at the same time, then merges and deduplicates the results. Five of them need no API key.

Wayback Machine wayback The Internet Archive CDX index. Deep historical coverage, enabled by default. Keyless

Common Crawl cc The monthly Common Crawl URL index, enabled by default. Keyless

OTX otx AlienVault Open Threat Exchange passive DNS and URLs, enabled by default. Keyless

Arquivo.pt arquivo The Portuguese web archive CDX index. Keyless

Urlscan urlscan Urlscan.io search. Works anonymously; a key only raises the rate limit. Keyless

VirusTotal vt URLs VirusTotal has observed for the domain. API key

ZoomEye zoomeye ZoomEye search results for the target. API key

GitHub github GitHub Code Search, for URLs committed into public repositories. API key

BeVigil bevigil Endpoints pulled out of unpacked Android apps. API key

Beyond the nine, urx also reads the target's own robots.txt and sitemap.xml, and will query any CDX server you point it at with --cdx-endpoint.

Built for the whole recon loop.

9 sources

Parallel collection

Async requests fan out to every enabled provider at once, then merge and deduplicate into a single URL set. Nothing waits on the slowest archive.

Advanced filtering

Filter by extension, pattern or preset. Control URL length and which host parts you keep.

$ urx target.com \
    -e js,php \
    --patterns api,v1

https://target.com/api/config.php
https://target.com/api/v1/auth.js
https://target.com/v1/users.js

URL validation

Check status codes, drop dead links, and pull more URLs out of the pages that answer.

Flexible output

Plain text, JSON, JSON Lines, CSV or a ready-made wordlist, written to the console, a file, or streamed into the next tool in the pipe.

Caching and incremental scanning

A local SQLite cache (or a shared Redis one, in builds with the redis-cache feature) reuses recent scans, and incremental mode returns only what has appeared since the last run.

From a domain to a clean URL set.

One command runs the whole pipeline, and every stage maps to a flag you can tune.

--providers

Collect

Query every enabled archive in parallel and merge the results into one stream.

-e · --patterns

Filter

Keep only the extensions, patterns or presets you actually care about.

--check-status

Validate

Probe what is live, record the status, and mine responses for more links.

-f json · -o

Output

Write text, JSON, JSON Lines, CSV or a wordlist to a file, or pipe it straight into the next tool.

Install urx and start collecting.

Available on Cargo, Homebrew, the AUR, Chocolatey, as prebuilt release binaries and as a container image. No account, no key to get going.

$ cargo install urx Cargo

$ brew install urx Homebrew

$ docker pull ghcr.io/hahwul/urx Docker