Skip to content

feat(search): cut result excerpts around the match - #69

Merged
bits-bytes-nn merged 4 commits into
mainfrom
feat/search-match-centred-excerpts
Aug 19, 2026
Merged

feat(search): cut result excerpts around the match#69
bits-bytes-nn merged 4 commits into
mainfrom
feat/search-match-centred-excerpts

Conversation

@bits-bytes-nn

Copy link
Copy Markdown
Owner

Search matched on the full body but showed a fixed excerpt of the first 40
words
— and a search term is almost never in the first 40 words.

Measured against the live index before the change:

query posts matched matched only past char 300
어텐션 19 19
트랜스포머 19 18
GRPO 7 7
ontology 3 3

So for a realistic Korean query, essentially every result showed prose with no
visible match and nothing highlighted. Correct results looked like wrong ones.

What changed

The excerpt is cut around the match. Done in templateMiddleware, a
simple-jekyll-search hook called once per {placeholder} per result: it receives
the whole field value and returns what to render, so the full body reaches the JS
and only the window reaches the DOM. No second fetch, no body in the markup.

  • Multi-word queries try the window at each match and keep whichever covers
    the most distinct query words. Centring on the first match alone gave "vibe
    coding" a window holding only a far-off coding; it now shows both.
  • Word-boundary snapping is capped at 12 characters. Korean has no inter-word
    spaces, so an unbounded search for one runs past the match and swallows the
    window — the same trap as the truncation floor in post_description.rb. The cap
    degrades it to a clean character cut.
  • Escaping is ours. The library inserts middleware output as HTML, so the text
    is escaped here and the only tags introduced are the <mark>s. Verified with a
    > query.

The index is now real plain text (plain_text, new _plugins/search_index.rb).
strip_html | strip_newlines had two defects that both reached the reader:

  • Entities stayed encoded — 29 of 35 posts carried them, so the card printed
    "음료 &gt; 탄산음료" verbatim.
  • Tags were removed with nothing left behind, so <p>끝</p><p>시작</p> indexed as
    끝시작, one token matching neither word.

Order is load-bearing and tested: strip tags before decoding entities, or a
post quoting &lt;script&gt; gets a real element that the strip then deletes
along with the author's words. Inline tags unwrap with no space so <em>강조</em>된
stays one word.

Also: the snippet field is gone (84 KB smaller index); the status line says
"Showing the first 10 matches." instead of reporting a total the library cannot
know, since it stops scanning at limit; and .result-tags was missing from the
dark-mode overrides, sitting at 4.2:1 — below AA — so it moves to $fg-muted
(6.2:1).

Retired along the way: the post-render highlight() pass, its data-original
bookkeeping, escapeRegExp, and the 120 ms debounce that existed to sequence
them. Highlighting happens during render now.

Verification

Driven in a real browser, not reasoned about: a driver script was appended to the
throwaway build so Chrome types into the actual /search/ page against the actual
index. Every result for 어텐션, GRPO, >, attention transformer, Pixtral and
vibe coding came back with at least one <mark> inside a match-centred excerpt.
Checked in both themes.

Gates: 93 tests / 157 assertions, clean build, html-proofer successful, all 16
validate-site checks passed.

Still open, now documented precisely in §9

  • No relevance ranking. The library scans in site.posts order and stops at
    limit, so results are newest-first and a title match does not outrank a
    passing mention. A sort hook can't fix it — sorting happens after truncation.
    Doing it properly means owning the match/sort/slice loop, at which point the
    library (used only for fetch, substring match and template fill) should go.
  • Math shows raw in excerpts. The index is flattened rendered HTML, so MathJax
    delimiters survive as \(O(L^2)\) and appear when the window lands on one.
    Stripping math from the index would fix it at the cost of making characters
    inside formulas unsearchable — left alone rather than decided unilaterally.

`strip_html | strip_newlines` got two things wrong, and both showed up in the
search results:

- Entities stayed encoded. kramdown escapes a literal `>` in prose to `&gt;`, so
  the index held `"음료 &gt; 탄산음료"` and the result card printed that at the
  reader. 29 of 35 posts carried entities.
- Tags were removed with nothing left behind, so `<p>끝</p><p>시작</p>` indexed as
  `끝시작` — one token matching neither word.

`plain_text` in the new _plugins/search_index.rb fixes both. Order is
load-bearing and tested: tags are stripped *before* entities are decoded, because
a post that quotes markup contains `&lt;script&gt;` as text, and decoding first
would make it a real element that the strip then deletes along with the author's
words. Inline tags unwrap with no space so `<em>강조</em>된` stays one word, while
block boundaries become one.

Also drops the `snippet` field. It held the first 40 words, which is not where a
match is; js/search.js now cuts the excerpt out of `content`. The index is 84 KB
smaller for it.
The index is every post's full body, but the result card showed a fixed excerpt
of the first 40 words — and a search term is almost never in the first 40 words.
Measured against the live index: for "어텐션" all 19 matching posts matched only
past character 300, so every card showed prose with no visible match and nothing
highlighted. "트랜스포머" 18 of 19, "GRPO" 7 of 7. Correct results looked wrong.

Now the excerpt is cut around the match, in `templateMiddleware` — a
simple-jekyll-search hook called once per {placeholder} per result, which
receives the whole field value and returns what to render. So the full body
reaches the JS and only the window reaches the DOM.

- Multi-word queries try the window at each match and keep whichever covers the
  most distinct query words. Centring on the first match alone gave "vibe coding"
  a window holding only a far-off `coding`.
- Word-boundary snapping is capped at 12 characters. Korean has no inter-word
  spaces, so an unbounded search for one runs past the match and swallows the
  window; the cap degrades it to a clean character cut. Guards also stop a snap
  from hiding the match it exists to reveal.
- Everything the middleware returns is inserted as HTML by the library, so this
  file escapes the text itself and the only tags it introduces are the <mark>s.
  Verified with a `>` query, which now renders `음료 > 탄산음료` correctly marked.

Highlighting moves into the same pass, retiring the post-render `highlight()`,
its `data-original` bookkeeping, the `escapeRegExp` helper and the 120 ms
debounce that existed to sequence them.

The status line no longer claims a total it cannot know: the library stops
scanning at `limit`, so on a full page it says "Showing the first 10 matches."
rather than reporting 10 as the count.
`_dark.scss` overrides the search UI that css/search.css hardcodes, but
`.result-tags` was not in the list, so it kept `#767676` — a value picked to clear
4.5:1 on white. On #0d1117 that is 4.2:1, and at 0.85em it is normal text, so it
failed AA. `$fg-muted` (#8b949e) is 6.2:1.
The invisible-match problem was written up in tech-doc §5 and §9 as a known
limitation. It is fixed, so both entries are replaced by how the excerpt is
actually chosen — the middleware hook, multi-word window selection, the 12-char
snap cap and why Korean needs it, and why the middleware escapes its own output.

§9 keeps the two limitations that are still real, now stated precisely:

- **No relevance ranking.** The library scans in `site.posts` order and stops at
  `limit`, so results are newest-first and a title match does not outrank a
  passing mention. A sort hook cannot fix it, because sorting happens after the
  truncation. Doing it properly means owning the match/sort/slice loop, at which
  point the library — used only for fetch, substring match and template fill —
  should go.
- **Math shows raw in excerpts.** The index is flattened rendered HTML, so MathJax
  delimiters survive as `\(O(L^2)\)` and appear when the window lands on one.
  Stripping math from the index would clean that up at the cost of making
  characters inside formulas unsearchable.

Also documents the `plain_text` filter and its strip-before-decode ordering, drops
the `snippet` field from the described index shape, and adds `search_index.rb` to
the plugin roll-call in all three documents.
@bits-bytes-nn
bits-bytes-nn merged commit 8319d5d into main Aug 19, 2026
2 checks passed
@bits-bytes-nn
bits-bytes-nn deleted the feat/search-match-centred-excerpts branch August 19, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant