=======================================
888
888
888
.d8888b888d888 8888b. 888 888 888888
d88P" 888P" "88b888 888 888888
888 888 .d888888888 888 888888
Y88b. 888 888 888Y88b 888 d88P888
"Y8888P888 "Y888888 "Y8888888P" 888
=======================================A fast, concurrent web crawler written in Rust π¦
- π Concurrent crawling with configurable parallelism
- π― Domain scoping to limit crawls to specific domains or TLDs
- π Depth-limited crawling to control how deep to traverse
- π‘ Custom User-Agent support
- π Duplicate detection to avoid revisiting pages
- π Crawl statistics (total pages, time elapsed, average time per page)
- π Quiet mode for piping output (automatic detection or via flag)
- πΎ File output in JSON, plain text, or SQLite (pages, statuses and the link graph)
- πΊ Sitemap discovery via robots.txt or
/sitemap.xml, including index and gzipped sitemaps - π€ robots.txt rules and
Crawl-delayrespected, with retries and backoff for flaky servers
git clone git@github.com:a-mango/crawl.git
cd crawl
cargo build --releaseThe binary will be available at target/release/crawl. You can move it to a directory in your PATH for easier access:
mv target/release/crawl $HOME/.local/bin/Basic usage:
crawl -s https://example.com-s, --start-url <URL>- Required. The URL to start crawling from-d, --depth <NUM>- Maximum depth to crawl (default: 2)--scope <DOMAIN>- Only follow links on this domain and its subdomains (e.g.example.com), or on any host with this suffix if it starts with a dot (e.g..com)--subdomains- Stay on the start URL's site, including all of its subdomains (cannot be combined with--scope)--delay <MS>- Delay between requests in milliseconds (default: 1000)--user-agent <STRING>- Custom User-Agent string (default: "rust-crawler/0.3.0")--concurrency <NUM>- Maximum number of concurrent requests (default: 10)--timeout <SECS>- Per-request timeout in seconds (default: 30)--max-retries <NUM>- Retries for network errors, timeouts, 429 and 5xx responses (default: 3)--ignore-robots- Ignore robots.txt rules andCrawl-delay--sitemap- Also start from every URL listed in the site's sitemaps--max-pages <NUM>- Stop after visiting this many pages-o, --output <FILE>- Output file:.json,.txt, or.db/.sqlite/.sqlite3for SQLite-q, --quiet- Quiet mode - only output URLs (useful for piping)
Crawl a site with depth 3:
crawl -s https://example.com -d 3Crawl only within a specific domain:
crawl -s https://example.com --scope example.comCrawl a site and all of its subdomains (here www.example.com, blog.example.com, β¦).
The site is derived from the start URL using the Public Suffix List,
so www.example.co.uk scopes to example.co.uk:
crawl -s https://www.example.com --subdomainsWithout --scope or --subdomains, the crawler follows links to any host.
Save results to a JSON file:
crawl -s https://example.com -o results.jsonSave results to a text file:
crawl -s https://example.com -o results.txtCrawl all .com sites with custom concurrency:
crawl -s https://example.com --scope .com --concurrency 20Crawl with a custom delay and user-agent:
crawl -s https://example.com --delay 2000 --user-agent "CustomBot/1.0"Piping to other programs (quiet mode activates automatically):
# Filter URLs containing "api"
crawl -s https://example.com | grep "/api/"
# Get first 10 URLs
crawl -s https://example.com | head -n 10Every option can also be set through an environment variable. Command-line flags take precedence.
| Variable | Flag |
|---|---|
CRAWL_START_URL |
--start-url |
CRAWL_DEPTH |
--depth |
CRAWL_SCOPE |
--scope |
CRAWL_SUBDOMAINS |
--subdomains |
CRAWL_DELAY_MS |
--delay |
CRAWL_USER_AGENT |
--user-agent |
CRAWL_CONCURRENCY |
--concurrency |
CRAWL_OUTPUT |
--output |
CRAWL_QUIET |
--quiet |
CRAWL_TIMEOUT_SECS |
--timeout |
CRAWL_MAX_RETRIES |
--max-retries |
CRAWL_IGNORE_ROBOTS |
--ignore-robots |
CRAWL_SITEMAP |
--sitemap |
CRAWL_MAX_PAGES |
--max-pages |
See .env.example for a template.
Build the image and run a crawl:
docker build -t crawl .
docker run --rm crawl -s https://example.com -d 2With Docker Compose, results written to /data inside the container land in ./output:
docker compose run --rm \
-e CRAWL_START_URL=https://example.com \
-e CRAWL_OUTPUT=/data/results.json \
crawlThe Compose service runs as UID/GID 1000 so output files are owned by your user. Set CRAWL_UID and CRAWL_GID if yours differ.
The crawler prints each discovered URL to stdout as it crawls, with an ASCII banner and statistics at the end:
=======================================
888
888
888
.d8888b888d888 8888b. 888 888 888888
d88P" 888P" "88b888 888 888888
888 888 .d888888888 888 888888
Y88b. 888 888 888Y88b 888 d88P888
"Y8888P888 "Y888888 "Y8888888P" 888
=======================================
Starting crawl: https://example.com
Configuration:
Max depth: 2
Delay: 1000ms
User-Agent: rust-crawler/0.3.0
Concurrency: 10
Retries: 3
robots.txt: respected
========================================
https://example.com
https://example.com/about
https://example.com/contact
========================================
Crawl complete!
Statistics:
Total pages: 3
Time elapsed: 2.45s
Avg per page: 816.67ms
When output is piped to another program or when using the -q/--quiet flag, only URLs are printed to stdout (one per line). Diagnostics still go to stderr unless -q is given:
https://example.com
https://example.com/about
https://example.com/contact
This makes it perfect for Unix-style command chaining and processing with standard tools like grep, sort, wc, etc.
When using -o/--output, results are saved in the specified format:
- JSON: Structured format with total count and sorted URL list
- Text: One URL per line, sorted alphabetically
- SQLite (
.db,.sqlite,.sqlite3): every page with its status, content type, depth and errors, plus every link between pages. Each run is appended as a new crawl, so one database can hold a history of crawls.
| Table | Columns |
|---|---|
crawls |
id, start_url, started_at, finished_at, page_count, crawler_version |
pages |
crawl_id, url, depth, status, content_type, error, fetched_at |
links |
crawl_id, source, target (every http(s) link found, followed or not) |
Timestamps are Unix milliseconds. Some example queries:
-- Broken pages in the latest crawl, and which pages link to them
SELECT p.url, p.status, l.source
FROM pages p JOIN links l ON l.crawl_id = p.crawl_id AND l.target = p.url
WHERE p.crawl_id = (SELECT MAX(id) FROM crawls) AND (p.status >= 400 OR p.error IS NOT NULL);
-- Most linked-to pages
SELECT target, COUNT(*) AS inbound FROM links WHERE crawl_id = 1
GROUP BY target ORDER BY inbound DESC LIMIT 10;
-- When each crawl ran
SELECT id, start_url, datetime(started_at / 1000, 'unixepoch') FROM crawls;Non-HTTP schemes (mailto:, ftp:, etc.) are skipped. Diagnostics (fetch errors, retries, robots.txt decisions) go to stderr and are silenced by -q, so stdout only ever contains URLs. Links are only extracted from successful (2xx) HTML responses.
With --sitemap, the crawler reads the start site's sitemaps before crawling
and uses every listed URL as an additional starting point (at depth 0). Sitemaps
are taken from the Sitemap: lines in robots.txt, falling back to
/sitemap.xml. XML, plain-text and gzipped sitemaps are supported, and sitemap
index files are followed (up to 100 sitemap files). Sitemap entries still go
through --scope/--subdomains and robots.txt. Large sites can list tens of
thousands of URLs, so combine it with --max-pages:
crawl -s https://blog.example.com --sitemap --max-pages 500 -o blog.db- robots.txt is fetched once per site and its rules are matched against the
User-Agent's product token (the part before
/). Following RFC 9309, a missing robots.txt (4xx) allows everything, while a server error or unreachable robots.txt blocks the whole site.Crawl-delayis honoured when it is longer than--delay. Use--ignore-robotsonly on sites you own or have permission to crawl. - Retries use exponential backoff starting at 500ms (capped at 60s).
A
Retry-Afterheader on 429/503 responses takes precedence.
use crawl::crawler::{crawl, CrawlConfig};
let report = crawl("https://example.com", CrawlConfig {
max_depth: 3,
scope: Some("example.com".into()),
..CrawlConfig::default()
}).await?;
for page in &report.pages {
println!("{} {:?} ({} links)", page.url, page.status, page.links.len());
}
// Optionally persist it
crawl::storage::save_sqlite("crawl.db", "https://example.com", started, finished, &report)?;The crawler is built with:
- reqwest - Async HTTP client
- tokio - Async runtime
- scraper - HTML parsing and link extraction
- clap - Command-line argument parsing
- texting_robots - robots.txt parsing
- psl - Public Suffix List lookups for
--subdomains - quick-xml / flate2 - Sitemap parsing and decompression
- rusqlite - SQLite output (SQLite is bundled, no system library needed)
The codebase is modular:
crawler.rs- Core crawling logic with concurrency controlextractor.rs- HTML parsing and link extractionsitemap.rs- Sitemap parsingstorage.rs- SQLite outputmain.rs- CLI interface
Run the test suite:
cargo testThe project includes:
- Unit tests for link extraction and URL normalization
- Integration tests for crawling scenarios (circular links, depth limits, scope filtering, etc.)
- No resume/checkpoint functionality
Contributions are welcome! Please feel free to submit issues or pull requests.
This software is licensed under the MIT License. See LICENSE for details.
- Add retry logic with exponential backoff
- Respect robots.txt
- Support for sitemap.xml
- Resume crawling from checkpoint
- Better progress indicators
- Handle rate limiting (429 responses)
- Add CSV output format
- Add option to download crawled content
- Store crawled links as a graph structure
- Store crawl results in a database