Extract web pages with Defuddle, offline, keeping Readability as a fallback

.gitignore

@@ -11,6 +11,7 @@ test-results/
 playwright-report/
 .playwright-mcp/
 .tmp/
+.scratch/
 apps/reader/public/downloads/
 apps/extension/release/
 artifacts/

apps/extension/src/page-annotations.test.ts

@@ -293,7 +293,7 @@ describe("drawing on the page", () => {
 });
 
 describe("highlights made on the saved reading copy", () => {
-  // The reading copy is Readability's extraction, which drops text the live page shows.
+  // The reading copy is an article extraction, which drops text the live page shows.
   const live = `<!doctype html><html><head><title>Essay</title></head><body><nav>Home News</nav>
     <main><article><h1>A headline about things</h1>
     <p>The first paragraph has a sentence that ends here, with commas, and more commas, for scoring.</p>

apps/reader/src/use-web-capture.ts

@@ -54,7 +54,7 @@ export function useWebCapture(
       setOffer(null);
       setNotice(null);
       try {
-        // Readability, the metadata extractors and PDFium load only when a source is added.
+        // The article extractor, metadata extractors and PDFium load only when a source is added.
         const [flow, { readerSourceAdditionServices }] = await Promise.all([
           import("./source-addition-flow.js"),
           import("./source-addition-services.js"),

docs/extension-capture-improvements.md

@@ -164,7 +164,7 @@ Selection capture targets the top-level page, not cross-origin embedded frames o
 shadow roots (capture includes it; the live selection index does not). Chrome's PDF viewer does not
 expose selections, so PDFs are highlighted in Reader. Live highlights are a projection, not the
 canonical annotation. Same-panel retry identity is not cross-window atomic deduplication. Missing
-passages in Readability's saved copy require opening that copy in Reader and selecting the
+passages in the extracted reading copy require opening that copy in Reader and selecting the
 intended text there; Reader will not silently substitute the live page or overwrite the archive.
 The service worker opens its own Connect session for page status; concurrent use with the panel
 relies on the SDK's grants tolerating two sessions in one extension origin.

docs/web-capture-extraction.md

@@ -0,0 +1,64 @@
+# Web capture extraction
+
+New captures in Reader and the MV3 extension share `webCaptureImport` in
+`packages/web-capture`. It loads the pinned `defuddle@0.19.5` browser core only
+when saving, never `defuddle/full` or `defuddle/node`. The browser bundle is
+self-contained; npm also installs CLI/optional dependencies, but those are not
+part of either application's extractor chunk.
+
+## Preserve the reading evidence
+
+Defuddle runs synchronously on a clone of the captured document. Our scholarly
+metadata, JSON-LD and publisher-tag pipeline still reads the untouched original;
+publisher authors retain their existing precedence. Output remains HTML passed
+through `prepareHtmlDocument`, with a separate unchanged archive.
+
+We disable standardization, content-pattern removal, partial-selector removal,
+hidden-element removal and small-image removal. The comparison found real losses
+with the defaults: truncated Wikipedia sections, code figures and captions in
+the Rust Book, arXiv tables/short summaries, and hidden sidenotes. These settings
+prefer complete evidence and original citation/code/math structure over extra
+cleanup. Exact-selector cleanup and content scoring remain enabled. Some
+boilerplate and duplicate math representations can consequently survive.
+
+As a last-resort safety check, an extraction with fewer than half the page's
+words (at least 400 page words, excluding navigation, header/footer, scripts,
+styles, templates and explicitly hidden elements) tries the separately lazy,
+pinned Readability 0.6.0. It switches only if Readability recovers more than both
+1.5 times the extracted words and 200 additional words. This is a truncation
+heuristic, not proof that every passage survived, and does not detect small losses.
+
+`capture.extractor` records `defuddle@0.19.5` or `mozilla-readability@0.6.0` on the
+new source note. The field is optional for historical captures. There is no
+backfill, opening-time extraction, or migration of existing `readable.html`:
+replacing that text would invalidate its highlights.
+
+## Network audit (0.19.5)
+
+The installed browser-core sources were searched for `fetch`, XHR and other
+network transports. All extractor HTTP calls are in these async paths:
+
+| Extractor | Network paths                                                                                                                                           |
+| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| X oEmbed  | FxTwitter API (including articles), then Twitter publish/oEmbed                                                                                         |
+| Reddit    | old.reddit HTML, then the Reddit Atom feed (`/.rss`)                                                                                                    |
+| YouTube   | InnerTube player (Android, iOS, web clients), next/chapters, caption-track XML URLs; async transcript-panel interaction can also initiate page requests |
+| Bilibili  | view metadata, player WBI/v2 and v2 variants, subtitle-track URLs                                                                                       |
+| C2 wiki   | `c2.com/wiki/remodel/pages/` JSON                                                                                                                       |
+
+They are reachable through `parseAsync()` or `fetchAsyncVariables()`, neither of
+which we call. We additionally set `useAsync: false` and supply a throwing fetch
+override rather than letting any extractor use global fetch. The Node/CLI
+`fetch.js` has direct fetch/proxy HTTP transports, but is not in the browser
+entry point. No extractor XHR calls were found.
+
+Tests spy on fetch/XHR for all five async-extractor families, including empty
+local shells and inline YouTube caption data. Resource/script sanitization and
+original-document immutability are also covered. A separate real Chromium check
+observed zero external requests while extracting the 39-page comparison corpus,
+the regression fixtures and empty X/Reddit/YouTube/Bilibili/C2 shells. Happy DOM's
+iframe loading is disabled in the unit tests to model browsers' inert parsed
+documents; Chromium also checked that generated embeds do not initiate requests.
+
+When upgrading Defuddle, repeat the source audit and comparison rather than
+assuming the version's network or cleanup behavior remains unchanged.

eslint.config.js

@@ -15,6 +15,7 @@ export default tseslint.config(
       "**/coverage/**",
       "**/node_modules/**",
       "**/.wrangler/**",
+      "**/.scratch/**",
       "apps/capacitor/android/**",
       "apps/capacitor/ios/**",
     ],

packages/connect/src/source-frontmatter.ts

@@ -77,6 +77,7 @@ function captureFields(
       captured_at: capture.retrievedAt,
       submitted_url: capture.submittedUrl,
       canonical_url: capture.canonicalUrl,
+      ...(capture.extractor ? { extractor: capture.extractor } : {}),
     },
   };
 }

packages/connect/src/source-imports-web-capture.test.ts

@@ -55,66 +55,70 @@ describe("ConnectSourceImportRepository web capture provenance", () => {
     );
   });
 
-  it("preserves raw and readable web representations with capture metadata", async () => {
-    const create = vi.fn(() => Promise.resolve(success(recordDocument())));
-    const upload = vi.fn((path: string) =>
-      Promise.resolve(
-        fileDescriptor({
-          fileId: path.includes("archive") ? "file-archive" : "file-readable",
-          path,
-          mediaType: "text/html",
-        }),
-      ),
-    );
-    const repository = new ConnectSourceImportRepository(
-      { create } as unknown as ReaderConnectClient,
-      { upload },
-    );
+  it.each([undefined, "defuddle@0.19.5", "mozilla-readability@0.6.0"])(
+    "preserves raw and readable web representations with extractor %s",
+    async (extractor) => {
+      const create = vi.fn(() => Promise.resolve(success(recordDocument())));
+      const upload = vi.fn((path: string) =>
+        Promise.resolve(
+          fileDescriptor({
+            fileId: path.includes("archive") ? "file-archive" : "file-readable",
+            path,
+            mediaType: "text/html",
+          }),
+        ),
+      );
+      const repository = new ConnectSourceImportRepository(
+        { create } as unknown as ReaderConnectClient,
+        { upload },
+      );
 
-    await repository.commitFile(webCapturePlan());
+      await repository.commitFile(webCapturePlan(extractor));
 
-    expect(upload.mock.calls.map(([path]) => path)).toEqual([
-      "files/reader/src_import/example.archive.html",
-      "files/reader/src_import/example.readable.html",
-    ]);
-    expect(create).toHaveBeenCalledWith(
-      expect.objectContaining({
-        frontmatter: expect.objectContaining({
-          kind: "webpage",
-          url: "https://example.com/canonical",
-          original_url: "https://example.com/submitted",
-          authors: ["Jane Example"],
-          published: "2026-08-01",
-          description: "A captured essay.",
-          language: "en",
-          site: "Example Review",
-          capture: {
-            submitted_url: "https://example.com/submitted",
-            canonical_url: "https://example.com/canonical",
-            captured_at: "2026-08-10T11:59:00.000Z",
-            method: "url",
-            application: "dev.mdbase.reader",
-          },
-          documents: [
-            expect.objectContaining({
-              role: "primary",
-              file_id: "file-readable",
-              derived_from_file_id: "file-archive",
-            }),
-            expect.objectContaining({
-              role: "archive",
-              file_id: "file-archive",
-              origin_url: "https://example.com/canonical",
-              retrieved_at: "2026-08-10T11:59:00.000Z",
-            }),
-          ],
+      expect(upload.mock.calls.map(([path]) => path)).toEqual([
+        "files/reader/src_import/example.archive.html",
+        "files/reader/src_import/example.readable.html",
+      ]);
+      expect(create).toHaveBeenCalledWith(
+        expect.objectContaining({
+          frontmatter: expect.objectContaining({
+            kind: "webpage",
+            url: "https://example.com/canonical",
+            original_url: "https://example.com/submitted",
+            authors: ["Jane Example"],
+            published: "2026-08-01",
+            description: "A captured essay.",
+            language: "en",
+            site: "Example Review",
+            capture: {
+              submitted_url: "https://example.com/submitted",
+              canonical_url: "https://example.com/canonical",
+              captured_at: "2026-08-10T11:59:00.000Z",
+              method: "url",
+              application: "dev.mdbase.reader",
+              ...(extractor ? { extractor } : {}),
+            },
+            documents: [
+              expect.objectContaining({
+                role: "primary",
+                file_id: "file-readable",
+                derived_from_file_id: "file-archive",
+              }),
+              expect.objectContaining({
+                role: "archive",
+                file_id: "file-archive",
+                origin_url: "https://example.com/canonical",
+                retrieved_at: "2026-08-10T11:59:00.000Z",
+              }),
+            ],
+          }),
         }),
-      }),
-    );
-  });
+      );
+    },
+  );
 });
 
-function webCapturePlan(): PlannedSourceFileImport {
+function webCapturePlan(extractor?: string): PlannedSourceFileImport {
   const base = plan();
   const representation = base.representations[0]!;
   return {
@@ -131,6 +135,7 @@ function webCapturePlan(): PlannedSourceFileImport {
       submittedUrl: "https://example.com/submitted",
       canonicalUrl: "https://example.com/canonical",
       retrievedAt: dateTime("2026-08-10T11:59:00.000Z"),
+      ...(extractor ? { extractor } : {}),
     },
     representations: [
       {

packages/core/src/application/import-source-file.test.ts

@@ -121,6 +121,7 @@ describe("importSourceFile web and duplicate safeguards", () => {
           submittedUrl: "https://example.com/story#part",
           canonicalUrl: "https://www.example.com/story",
           retrievedAt: dateTime("2026-08-10T11:59:00.000Z"),
+          extractor: "defuddle@0.19.5",
         },
         archive: {
           name: "example-com.archive.html",
@@ -136,6 +137,7 @@ describe("importSourceFile web and duplicate safeguards", () => {
           submittedUrl: "https://example.com/story",
           canonicalUrl: "https://www.example.com/story",
           retrievedAt: "2026-08-10T11:59:00.000Z",
+          extractor: "defuddle@0.19.5",
         },
         representations: [
           expect.objectContaining({ role: "primary", derivedFromRole: "archive" }),

packages/core/src/application/source-import-ports.ts

@@ -56,6 +56,8 @@ export interface SourceCaptureProvenance {
   readonly submittedUrl: string;
   readonly canonicalUrl: string;
   readonly retrievedAt: DateTime;
+  /** Extractor used for this new reading copy; absent on older captures. */
+  readonly extractor?: string;
 }
 
 export interface SourceCaptureArchive {

packages/core/src/application/source-import-values.ts

@@ -21,7 +21,13 @@ export function validateCapture(
 ): NonNullable<SourceFileImportRequest["capture"]> {
   const submittedUrl = httpsUrl(capture.submittedUrl);
   const canonicalUrl = httpsUrl(capture.canonicalUrl);
-  return { submittedUrl, canonicalUrl, retrievedAt: capture.retrievedAt };
+  const extractor = normalizeOptionalText(capture.extractor, 100);
+  return {
+    submittedUrl,
+    canonicalUrl,
+    retrievedAt: capture.retrievedAt,
+    ...(extractor ? { extractor } : {}),
+  };
 }
 
 export function validateMetadata(metadata: SourceImportMetadata): SourceImportMetadata {

packages/web-capture/package.json

@@ -15,6 +15,7 @@
   "dependencies": {
     "@mdbase-reader/core": "workspace:*",
     "@mdbase-reader/renderer-html": "workspace:*",
-    "@mozilla/readability": "^0.6.0"
+    "@mozilla/readability": "0.6.0",
+    "defuddle": "0.19.5"
   }
 }

packages/web-capture/src/fixtures/README.md

@@ -0,0 +1,18 @@
+# Capture fixtures
+
+These small, original HTML pages reproduce structures observed in the public-page
+comparison; they do not copy the articles' text. The full fetched corpus stays in
+`.scratch/corpus` (gitignored).
+
+- `academic.html`: structures reduced from [Llama 3](https://arxiv.org/html/2407.21783v1)
+  and [DeepSeekMath](https://arxiv.org/html/2402.03300v2): short summary paragraphs,
+  MathML, table figures, captions, hidden footnotes, and conflicting
+  publisher/JSON-LD metadata.
+- `code-guide.html`: structures reduced from the
+  [Rust Book ownership chapter](https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html):
+  code inside figures, diagram captions, and ordinary preformatted code.
+- `encyclopedia.html`: structures reduced from
+  [CRISPR gene editing](https://en.wikipedia.org/wiki/CRISPR_gene_editing):
+  Wikipedia-style sections, a data table, and original-format citation anchors.
+
+Tests exercise the final sanitized reading copy, not just raw extractor output.