PyStates is a fast, minimalist command-line utility that downloads full-length articles from major publications (The New Yorker, Wired, The Atlantic, and hundreds more) and extracts clean, readable text using the industry-leading Trafilatura library.
It automatically removes ads, navigation menus, sidebars, comments, and boilerplate, then saves the result as either beautifully formatted Markdown (.md) or clean Plain Text (.txt).
Designed for researchers, writers, journalists, students, and power users who want to archive high-quality content locally β without distractions or paywalls interfering with readability.
pystates -u https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era- π Blazing Fast β Downloads and processes articles in under 3 seconds
- π‘οΈ 403 Bypass β Smart custom
User-Agentheaders to reliably avoid common blocks - π§Ό Pristine Extraction β Powered by Trafilatura for the cleanest possible text
- π« Furniture Removal β "related articles", newsletter and promo blocks are pruned before extraction, so the file contains only the article (override with
--no-prune) - π Four Output Formats β Markdown (
.md), Plain Text (.txt), Word (.docx) and PDF (.pdf) - π§Ή Intelligent Filename Sanitization β Generates safe OS filenames from article titles without using
re - βοΈ Custom Filenames β Override auto-generated names with the
-oflag - π Modern Python Stack β Built for Python 3.10+ with clean architecture
- π¦ One-Command Installation β
pip install -e .for instant local development - π Broad Compatibility β Works with virtually any news, magazine, or blog site supported by Trafilatura
# 1. Clone the repository
git clone https://github.com/Maksum867/PyStates.git
cd pystates
# 2. Install in editable/development mode
pip install -e .md and txt work out of the box. Word and PDF output need extra dependencies:
pip install "pystates-CLI[word]" # .docx (python-docx)
pip install "pystates-CLI[pdf]" # .pdf (reportlab)
pip install "pystates-CLI[word,pdf]" # bothIf a format is requested without its dependency installed, the CLI exits with a clear message telling you which extra to install.
pystates --help- Python β₯ 3.9
pip
All dependencies (requests, trafilatura) are installed automatically via pyproject.toml. argparse ships with Python.
pystates -u URL [OPTIONS]| Short | Long | Required | Default | Description |
|---|---|---|---|---|
-u |
--url |
Yes | β | The full URL of the article to fetch and process |
-f |
--format |
No | md |
Output format: md, txt, docx (Word) or pdf |
-o |
--output |
No | (auto) | Custom output filename. Any directory component is stripped; a matching extension is normalised |
-d |
--output-dir |
No | . |
Directory to write the file into |
-q |
--quiet |
No | off | Suppress progress output (useful in cron jobs and CI) |
--no-prune |
No | off | Keep "related articles" / newsletter blocks that are stripped by default | |
--version |
No | β | Print the version and exit |
pystates -u "https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era"β Creates the-new-ai-era.md
pystates -u "https://www.wired.com/story/the-future-of-coding/" -f txtβ Creates the-future-of-coding.txt
pystates -u "https://www.theatlantic.com/ideas/archive/2025/07/climate-change/" \
-o "climate_report_july_2025" \
-f mdβ Creates climate_report_july_2025.md
pystates \
--url "https://www.newyorker.com/culture/cultural-comment/the-end-of-the-internet-as-we-know-it" \
--format txt \
--output "end_of_internet"pystates -u "https://www.newyorker.com/books/page-turner/what-is-biography-for" -f docx
pystates -u "https://en.wikipedia.org/wiki/Ada_Lovelace" -f pdfβ Creates What_Is_Biography_For.docx / Ada_Lovelace.pdf, with a proper title,
byline (author Β· date Β· publication) and one paragraph per source paragraph.
pystates --help# The New AI Era
*By Jane Smith Β· 2025-01-20 Β· The New Yorker*
<https://www.newyorker.com/culture/cultural-comment/the-end-of-the-internet-as-we-know-it>
---
Artificial intelligence is no longer a distant promise...
## A New Paradigm
The implications are profound...
PyStates/
βββ pystates/
β βββ __init__.py # Package metadata & version
β βββ __main__.py # Entry point for `python -m pystates`
β βββ cli.py # Argument parsing (argparse)
β βββ exporters.py # Filename sanitization + md/txt/docx/pdf export
β βββ parser.py # HTTP fetching (custom UA) + Trafilatura parsing
β βββ tests/ # 75 pytest tests (unit + real-HTTP integration)
βββ .github/
β βββ workflows/
β β βββ tests.yml # pytest matrix, lint, packaging
β β βββ publish.yml # build + publish to PyPI (Trusted Publishing)
β βββ dependabot.yml # automatic dependency updates
βββ pyproject.toml # Modern packaging + entry points
βββ CHANGELOG.md
βββ .gitignore
βββ README.md
- JavaScript-only pages are not supported. PyStates downloads the HTML the server sends; sites that render the article in the browser (client-side React apps) may return a listing page or a paywall stub instead of the article. When the extracted text is suspiciously short, the CLI prints a warning on stderr and still writes the file.
- Bot protection blocks some sites. With a plain HTTP client, sites behind
Cloudflare/Akamai may answer
403/401. PyStates reports the HTTP error and exits with status1. - PDF fonts. A Unicode TTF (DejaVu/Liberation) is used when present on the system, so Cyrillic and CJK render correctly on most Linux images and macOS; elsewhere ReportLab falls back to Helvetica, which only covers Latin-1.
- The default User-Agent is a fixed desktop Chrome string. It is not rotated, and it is not a "403 bypass".
git clone https://github.com/Maksum867/PyStates.git
cd PyStates
pip install -e ".[dev]"
python -m pytest # 75 tests, no network access required
ruff check . && ruff format --check .The suite includes a real end-to-end run against a local http.server, so it
verifies the whole pipeline (download β extract β sanitize β write) without
touching the internet.
- Bump the version in
pyproject.tomlandpystates/__init__.py. - Update
CHANGELOG.md. - Publish a GitHub Release tagged
vX.Y.Z. .github/workflows/publish.ymlruns the tests, builds the distributions and publishes to PyPI via Trusted Publishing (no API token needed).
This project is released under the MIT License.
See LICENSE for the full text.
PyStates β preserving the web's best writing, one clean file at a time.
Built with care by the open-source community.