Collect every URL a domain has exposed.

A Rust CLI that queries up to nine OSINT sources in parallel, then filters and checks the results.

urx · recon

$ urx dalfox.hahwul.com --providers wayback,otx --check-status

Domains         [====================] 1/1 processed
Wayback Machine [====================] done
OTX             [====================] done
Robots.txt      [====================] done
Sitemap         [====================] done
Filtering       [====================] 199 URLs
Testing         [====================] 199/199 complete

https://dalfox.hahwul.com/                         [200 OK]
https://dalfox.hahwul.com/.well-known/security.txt [200 OK]
https://dalfox.hahwul.com/admin                    [404 Not Found]
https://dalfox.hahwul.com/advanced/config/         [404 Not Found]
https://dalfox.hahwul.com/llms.txt                 [200 OK]
https://dalfox.hahwul.com/sitemap.xml              [200 OK]

Nine sources, one command.

Web archives and threat-intel feeds, queried in parallel. Results are merged and deduplicated. Five sources need no API key.

Wayback Machine wayback Internet Archive CDX index. On by default. Keyless

Common Crawl cc Monthly Common Crawl URL index. On by default. Keyless

OTX otx AlienVault Open Threat Exchange URLs. On by default. Keyless

Arquivo.pt arquivo Portuguese web archive CDX index. Keyless

Urlscan urlscan urlscan.io search. Works without a key; a key raises the rate limit. Keyless

VirusTotal vt URLs VirusTotal has seen for the domain. API key

ZoomEye zoomeye ZoomEye search results. API key

GitHub github GitHub Code Search over public repositories. API key

BeVigil bevigil Endpoints extracted from Android apps. API key

urx also reads the target's robots.txt and sitemap.xml, and can query any CDX server with --cdx-endpoint.

Features

9 sources

Parallel collection

All enabled providers are queried at once. Results are merged and deduplicated into one URL set.

Filtering

Filter by extension, pattern or preset. Limit URL length and choose which host parts to keep.

$ urx target.com \
    -e js,php \
    --patterns api,v1

https://target.com/api/config.php
https://target.com/api/v1/auth.js
https://target.com/v1/users.js

URL validation

Check status codes, drop dead links, and extract more URLs from live pages.

Output formats

Text, JSON, JSON Lines, CSV or a wordlist. Write to the console or a file, or stream into a pipe.

Caching and incremental scanning

A local SQLite cache (or Redis, in builds with the redis-cache feature) reuses recent scans. Incremental mode returns only URLs new since the last run.

How it works

One command runs every stage. Each stage has its own flags.

--providers

Collect

Query enabled sources in parallel and merge the results.

-e · --patterns

Filter

Keep only matching extensions, patterns or presets.

--check-status

Validate

Check which URLs are live, record the status, and extract more links.

-f json · -o

Output

Write text, JSON, JSON Lines, CSV or a wordlist to a file or a pipe.

Install

Available via Cargo, Homebrew, the AUR, Chocolatey, release binaries and Docker. The default sources need no account or API key.

$ cargo install urx Cargo

$ brew install urx Homebrew

$ docker pull ghcr.io/hahwul/urx Docker