A desktop application and CLI tool that recursively crawls websites up to X levels deep, extracts high-resolution images while filtering out blank lazy-loading placeholders, and archives them directly into organized ZIP packages with a built-in archive manager and image previewer.
-
Recursive Multi-Level Crawler ("Worming"):
- Crawl up to
$X$ levels deep (single page, direct links, or recursive exploration). - Domain Lock: Stays strictly within the starting domain to prevent wandering into the wider internet.
- Runaway Safeguards: Set max page limits and visited-page tracking to avoid infinite crawler loops.
- Structured ZIPs: Groups crawled images into tidy subfolders named after each page slug.
- Crawl up to
-
Intelligent Image Extraction:
-
Bypasses Lazy Loading: Prioritizes
data-src,data-original,data-lazy-src, andsrcset(automatically selecting the highest resolution available). - Dummy & Placeholder Filtering: Identifies and drops empty SVG containers, 1x1 spacer GIFs, and blank placeholders.
-
<picture>&<noscript>Support: Captures modern responsive<source>formats (WebP, AVIF) and non-JavaScript fallbacks. -
Anti-Hotlink Referer Spoofing: Supplies authentic
Refererand browser headers so CDNs do not block downloads.
-
Bypasses Lazy Loading: Prioritizes
-
Automatic WebP to PNG Conversion:
- Automatically converts all downloaded WebP images into standard, universal PNG files before archiving.
-
Keep or Trash Original WebP: Choose to retain the original
.webpfiles in a dedicatedwebp_originals/subfolder or trash/discard them to keep your archives lean. - Custom PNG Quality: Select between Lossless (100), High (90), Good (75), or Low (50) to optimize file size vs. visual fidelity.
-
Headless Browser Rendering (Playwright):
- Toggle headless Chromium execution to render dynamic JavaScript single-page apps (SPAs) and trigger scroll-based
IntersectionObserverloaders.
- Toggle headless Chromium execution to render dynamic JavaScript single-page apps (SPAs) and trigger scroll-based
-
Built-in Archive Manager GUI:
- Catalog & History: Automatically tracks all your scraping sessions in a searchable table.
- Direct In-Memory Preview: Click any image in a ZIP to view a scaled thumbnail and file metadata without unpacking the archive to disk.
- Windows Explorer Integration: Open and highlight archives directly in Windows Explorer.
- One-Click Batch Extraction: Extract archives to any chosen destination folder.
├── gui.py # Desktop GUI (Crawler + Archive Manager)
├── crawler.py # Recursive multi-level crawling engine & CLI
├── fetch_images.py # Standalone single-page scraper CLI
├── archive_store.py # Registry manager, ZIP inspection, & preview loader
├── screenshots/ # Application screenshots & UI previews
├── requirements.txt # Python package dependencies
├── .gitignore # Git ignore rules for virtualenvs and archives
├── CONTRIBUTING.md # Contribution guidelines
└── README.md # Project documentation
- Python 3.10+
- Windows, macOS, or Linux
-
Clone this repository:
git clone https://github.com/MagicPlants/web-image-crawler.git cd web-image-crawler -
Create and activate a virtual environment:
python -m venv .venv # On Windows (PowerShell): .venv\Scripts\Activate.ps1 # On macOS/Linux: source .venv/bin/activate
-
Install dependencies:
pip install -r requirements.txt
-
(Optional, for dynamic JavaScript sites) Install the Playwright browser:
playwright install chromium
python gui.py- Tab 1 (Web Crawler & Downloader): Enter your target URL, choose crawl depth and page limits, toggle browser rendering if needed, and hit Start Crawl.
- Tab 2 (Archive Manager & Viewer): Inspect saved archives, browse files inside the ZIP, preview images, or open them in your default photo viewer.
python fetch_images.py https://example.com -o my_images.zip# Crawl 2 levels deep, capping at 30 pages:
python crawler.py https://example.com -d 2 -m 30 -o archives/example_deep.zip
# Use headless Chromium for dynamic JavaScript / infinite-scroll sites:
python crawler.py https://example.com -d 1 -b -o archives/example_js.zip| Flag | Description | Default |
|---|---|---|
-d, --depth |
Number of levels deep to follow links | 0 (current page only) |
-m, --max-pages |
Maximum pages to crawl | 25 |
-o, --output |
Destination ZIP path | images.zip |
-b, --browser |
Use headless browser for JS rendering | Disabled |
--any-domain |
Allow crawling external domains | Disabled (stays on domain) |
--min-size |
Minimum image size in bytes | 200 |
--no-convert-webp |
Disable automatic WebP to PNG conversion | Enabled by default |
--keep-webp |
Keep original WebP files in webp_originals/ |
Disabled (trashes originals) |
--png-quality |
PNG quality: Lossless (100), High (90), Good (75), Low (50) |
Lossless (100) |
Contributions are what make the open-source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated!
If you have an idea for a feature or found a bug:
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Please check out CONTRIBUTING.md for detailed guidelines on code style, issue reporting, and pull request procedures.
Joe Montgomery (Magic Plants)
- Website: magicplants.net
- GitHub: @MagicPlants
Distributed under the MIT License. See LICENSE for more information.
Copyright © 2026 Joe Montgomery (Magic Plants).


