Repository navigation
Extract web pages with Defuddle, offline, keeping Readability as a fallback - #69
Merged
Merged
Conversation
… evidence Replace Readability with the pinned Defuddle 0.19.5 browser core for article extraction in Reader and the extension, lazily loaded on save. - Extraction is offline: async fallbacks are off and Defuddle's fetch is replaced with one that throws. A test fails on any fetch or XHR. - Defuddle's default cleanup lost real text on a 39-page comparison (a truncated Wikipedia article, Rust Book code figures, arXiv table figures, hidden sidenotes), so standardisation, content patterns, partial selectors and hidden/small-element removal are disabled. - Readability stays as a lazy fallback when Defuddle's output is suspiciously short; provenance records whichever extractor ran. Existing reading copies are never re-extracted. - Publisher and scholarly metadata still take precedence. - X pages belong to the social-post adapter, so the offline test expects an X shell to be refused there, still without a network request.
callumalpass
force-pushed
the
feat/defuddle-extractor
branch
from
October 11, 2026 09:30
df91538 to
61457d3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Web captures in Reader and the extension now use Defuddle (pinned 0.19.5, browser core) instead of Readability. It is loaded only when saving, as Readability was.
What changes
fetchthrows. A test fails on any fetch or XHR. Extraction depends only on the captured HTML.defuddle@0.19.5ormozilla-readability@0.6.0), and existing reading copies are never re-extracted, so their highlights keep their anchors.Comparison (same captured HTML, real Chromium; words of extracted body)
Defuddle keeps math as MathML (Wikipedia, arXiv) where Readability drops it, which accounts for much of the Wikipedia difference; it also keeps code blocks with captions and arXiv tables, and strips site suffixes from titles. Bylines still differ on some academic pages, which is why publisher metadata stays authoritative. Three small structural fixtures (original prose) are committed as regression tests.
Cost
The lazy Defuddle chunk is about 310 KB (86 KB gzip), fetched only on save; the extension panel does not load it on open. The Readability chunk (34 KB, 11 KB gzip) remains for the fallback.
Verification
pnpm checkpasses; extension and Reader build. Fixture rendering and quote re-anchoring were checked in Chromium, and the built MV3 extension extracted offline. A connected LAB round trip has not run: LAB's daemon currently reportsincompatible_version.