web-crawler-node

CLI TypeScript crawler that visits every same-host URL from a seed page, prints links found (including external), and uses a dynamic FIFO frontier with bounded Promise.race() concurrency.

Requirements

• Node.js 20+

How to run

npm install

Quick crawl:

npm run crawl -- https://www.example.com/

With options:

npm run crawl -- https://www.example.com/ --concurrency=2 --timeout=5000

Flag	Default	Meaning
--concurrency	5	Max pages fetched at once
--timeout	5000	Per-request timeout in ms

Develop / check

npm test
npm run test:coverage
npm run typecheck
npm run lint
npm run build

Concurrency

The scheduler is work-conserving and bounded:

1. Fill active from the FIFO queue until active.size === concurrency.
2. await Promise.race(active.values()) for the first completion.
3. Report that result, schedule new same-domain links, refill free slots.
4. Stop when both the queue and active are empty.