A heading holding a line break extracts as one line, words beside the marker - #63
Merged
Conversation
… marker The second field wave (#62) found the residue the empty-anchor repair did not reach: Hugging Face model cards write headings as `<h3><span>Use Case: <br></span></h3>`, and the `<br>` — not the anchor — is what breaks extraction. A markdown heading is one line by construction, so trafilatura emits the marker alone and the words follow below it, among the span's source-formatting tabs and newlines emitted verbatim. Any consumer keying on the heading line reads an empty heading. Measured on the two reported pages: 12 and 18 broken headings under 0.1.0; the merged anchor repair cured 12 and 14 of them; the four survivors all carried a `<br>` inside the heading. So preparation now flattens the inside of every heading: `<br>` elements drop (isolated as the root cause — dropping them alone cures all four survivors), and whitespace runs collapse to one space, which is what puts the words beside the marker cleanly instead of behind a run of tabs. Both reported pages now extract zero broken headings, and every page from the first wave is byte-identical to what the merged repair produced. The fixture is the verbatim Hugging Face heading shape — an earlier hand-reduced version passed against the unfixed code, because the `<br>` was the load-bearing detail the reduction dropped. Also records the one new fact from the fourth article-defect occurrence (#61, its fix already merged): an x article's figures live in `article.media_entities`, which the driver does not read — a backlog entry, not a fix, since the prose recovery is whole without it. Fixes #62 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #56, from the second field wave (#62) — filed by an instance still on 0.1.0, but a residue of it survives the merged repair.
What #56 left behind
Hugging Face model cards write headings as
<h3><span>Use Case: <br></span></h3>. The<br>— not the anchor — is what breaks extraction: a markdown heading is one line by construction, so trafilatura emits the marker alone and the words land below it, among the span's source-formatting tabs and newlines emitted verbatim:Measured on the two reported pages against each stage:
The four survivors all carried a
<br>inside the heading — isolated as the root cause by testing each repair separately (br-drop alone cures all four; whitespace collapse is what puts the words beside the marker instead of behind a run of tabs). Both repairs ship.Every page from the first wave (PocketFlow, llama-cpp, HN) extracts byte-identically to the merged repair — no regressions.
A note on the fixture
The first fixture I wrote — the same heading minus the
<br>— passed against the unfixed code: the<br>was the load-bearing detail the hand-reduction dropped. The shipped fixture is the verbatim HF shape, confirmed failing on merged main before the fix.Also in this PR
A backlog entry for the one new fact in #61's local note (the fourth article-defect occurrence, fix already merged in #57): an x article's figures live in
article.media_entities, which the driver doesn't read. The reporter recovered them by hand. Backlogged rather than fixed — the prose recovery is whole without it, and mapping figures into the media pool is its own small design.Four gates green: 2041 passed, ruff, ty, format.
Fixes #62