Docprobe is a universal documentation extraction tool built to archive modern documentation websites that do not provide downloadable versions.
It automatically detects common documentation frameworks such as Docusaurus, MkDocs, GitBook, and ReadTheDocs, then extracts content using intelligent fallback strategies.
- Automatic documentation platform detection
- Extracts dynamic SPA documentation sites
- Toolbar crawling and sidebar navigation discovery
- Smart extraction fallback: Markdown → Text → OCR
- Concurrent crawling
- Resume interrupted crawls
- Post-processing: rename files by chapter title, strip nav boilerplate
- PDF export support (per-page or single combined manual)
- OCR support for difficult or image-heavy pages
- Designed for modern JavaScript-rendered documentation portals
- Docusaurus
- MkDocs
- GitBook
- ReadTheDocs
- Custom SPA documentation sites
- PDF-viewer style documentation pages
- Image-heavy documentation pages via OCR fallback
1. Clone the repository
git clone https://github.com/risshe92/docprobe.git
cd docprobe2. Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate3. Install Python dependencies
pip install -r requirements.txt4. Install Playwright browser dependencies
playwright install chromium5. Install system dependencies
Debian/Ubuntu:
sudo apt install tesseract-ocr pandoc texlive-xetexArch:
sudo pacman -S tesseract pandoc texlive-coreExtract a single documentation page:
python3 docprobe.py --url https://docs.example.com/docsCrawl a documentation sidebar automatically:
python3 docprobe.py --url https://docs.example.com/docs --crawl-toolbarResume an interrupted crawl:
python3 docprobe.py --url https://docs.example.com/docs --crawl-toolbar --resumeExport extracted markdown pages to individual PDFs:
python3 docprobe.py --url https://docs.example.com/docs --crawl-toolbar --export-pdf pdfGenerate a single combined PDF manual:
python3 docprobe.py --url https://docs.example.com/docs --crawl-toolbar --export-pdf pdf-single| Argument | Default | Description |
|---|---|---|
--url |
required | Target documentation URL |
--export-as |
md |
Output format: md, txt, html, ocr |
-o, --output-dir |
./output/<format>/ |
Output directory |
--crawl-toolbar |
off | Crawl all links in the sidebar/toolbar |
--max-pages |
0 (no limit) |
Limit number of pages crawled |
--resume |
off | Skip already-processed URLs |
--concurrency |
4 |
Number of concurrent crawl workers |
--export-pdf |
none |
Export to PDF after crawl: none, pdf, pdf-single |
--debug |
off |
Log level: off, error, info, debug |
| Argument | Description |
|---|---|
--post-process |
Run post-processing on an existing directory (no crawl) |
--input-dir |
Input directory to process (required with --post-process) |
--rename |
Rename files using their first heading or title |
--clean |
Strip nav boilerplate and frontmatter (markdown only) |
--export-pdf |
Export to PDF after processing: pdf, pdf-single |
--export-as controls the output format during crawl:
| Value | Output |
|---|---|
md |
Markdown (default) |
txt |
Plain text |
html |
Rendered HTML |
ocr |
OCR text from page screenshots |
Output is written to ./output/<format>/ by default, or to -o if specified.
When using the default md mode, Docprobe analyzes the page and applies a fallback chain automatically:
markdown → text → ocr
This allows Docprobe to recover from poor Markdown extraction and still capture difficult content.
Post-processing runs on an existing directory of extracted files without re-crawling.
Rename files by chapter title:
python3 docprobe.py --post-process --input-dir ./output/md --renameClean nav boilerplate from markdown files:
python3 docprobe.py --post-process --input-dir ./output/md --cleanRename, clean, then export a single combined PDF:
python3 docprobe.py --post-process --input-dir ./output/md --rename --clean --export-pdf pdf-singleoutput/
└── md/ # or txt/, html/, ocr/ depending on --export-as
├── 0001_overview.md
├── 0002_getting_started.md
└── ...
meta/
├── run.json
├── crawl_results.json
├── toolbar.json
├── content.json
└── extraction.json
documentation_manual.pdf # if --export-pdf pdf-single was used
Single page, default markdown mode:
python3 docprobe.py --url https://docs.example.com/docsCrawl toolbar with concurrency:
python3 docprobe.py \
--url https://docs.example.com/docs \
--crawl-toolbar \
--concurrency 8 \
--debug infoCrawl first 20 pages only:
python3 docprobe.py \
--url https://docs.example.com/docs \
--crawl-toolbar \
--max-pages 20Save output to a custom directory:
python3 docprobe.py \
--url https://docs.example.com/docs \
--crawl-toolbar \
-o /home/user/docs_archiveResume a previous crawl:
python3 docprobe.py \
--url https://docs.example.com/docs \
--crawl-toolbar \
--resumeExport as plain text:
python3 docprobe.py \
--url https://docs.example.com/docs \
--crawl-toolbar \
--export-as txtForce OCR mode:
python3 docprobe.py \
--url https://docs.example.com/docs \
--crawl-toolbar \
--export-as ocr- OCR requires
tesseractto be installed on the system. - PDF export requires
pandocand a LaTeX engine (xelatex). - Some sites may require site-specific tuning depending on frontend complexity.
- Large crawls work best with
--resumeto handle interruptions.