feat(search): cut result excerpts around the match - #69
Merged
Conversation
`strip_html | strip_newlines` got two things wrong, and both showed up in the search results: - Entities stayed encoded. kramdown escapes a literal `>` in prose to `>`, so the index held `"음료 > 탄산음료"` and the result card printed that at the reader. 29 of 35 posts carried entities. - Tags were removed with nothing left behind, so `<p>끝</p><p>시작</p>` indexed as `끝시작` — one token matching neither word. `plain_text` in the new _plugins/search_index.rb fixes both. Order is load-bearing and tested: tags are stripped *before* entities are decoded, because a post that quotes markup contains `<script>` as text, and decoding first would make it a real element that the strip then deletes along with the author's words. Inline tags unwrap with no space so `<em>강조</em>된` stays one word, while block boundaries become one. Also drops the `snippet` field. It held the first 40 words, which is not where a match is; js/search.js now cuts the excerpt out of `content`. The index is 84 KB smaller for it.
The index is every post's full body, but the result card showed a fixed excerpt
of the first 40 words — and a search term is almost never in the first 40 words.
Measured against the live index: for "어텐션" all 19 matching posts matched only
past character 300, so every card showed prose with no visible match and nothing
highlighted. "트랜스포머" 18 of 19, "GRPO" 7 of 7. Correct results looked wrong.
Now the excerpt is cut around the match, in `templateMiddleware` — a
simple-jekyll-search hook called once per {placeholder} per result, which
receives the whole field value and returns what to render. So the full body
reaches the JS and only the window reaches the DOM.
- Multi-word queries try the window at each match and keep whichever covers the
most distinct query words. Centring on the first match alone gave "vibe coding"
a window holding only a far-off `coding`.
- Word-boundary snapping is capped at 12 characters. Korean has no inter-word
spaces, so an unbounded search for one runs past the match and swallows the
window; the cap degrades it to a clean character cut. Guards also stop a snap
from hiding the match it exists to reveal.
- Everything the middleware returns is inserted as HTML by the library, so this
file escapes the text itself and the only tags it introduces are the <mark>s.
Verified with a `>` query, which now renders `음료 > 탄산음료` correctly marked.
Highlighting moves into the same pass, retiring the post-render `highlight()`,
its `data-original` bookkeeping, the `escapeRegExp` helper and the 120 ms
debounce that existed to sequence them.
The status line no longer claims a total it cannot know: the library stops
scanning at `limit`, so on a full page it says "Showing the first 10 matches."
rather than reporting 10 as the count.
`_dark.scss` overrides the search UI that css/search.css hardcodes, but `.result-tags` was not in the list, so it kept `#767676` — a value picked to clear 4.5:1 on white. On #0d1117 that is 4.2:1, and at 0.85em it is normal text, so it failed AA. `$fg-muted` (#8b949e) is 6.2:1.
The invisible-match problem was written up in tech-doc §5 and §9 as a known limitation. It is fixed, so both entries are replaced by how the excerpt is actually chosen — the middleware hook, multi-word window selection, the 12-char snap cap and why Korean needs it, and why the middleware escapes its own output. §9 keeps the two limitations that are still real, now stated precisely: - **No relevance ranking.** The library scans in `site.posts` order and stops at `limit`, so results are newest-first and a title match does not outrank a passing mention. A sort hook cannot fix it, because sorting happens after the truncation. Doing it properly means owning the match/sort/slice loop, at which point the library — used only for fetch, substring match and template fill — should go. - **Math shows raw in excerpts.** The index is flattened rendered HTML, so MathJax delimiters survive as `\(O(L^2)\)` and appear when the window lands on one. Stripping math from the index would clean that up at the cost of making characters inside formulas unsearchable. Also documents the `plain_text` filter and its strip-before-decode ordering, drops the `snippet` field from the described index shape, and adds `search_index.rb` to the plugin roll-call in all three documents.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Search matched on the full body but showed a fixed excerpt of the first 40
words — and a search term is almost never in the first 40 words.
Measured against the live index before the change:
So for a realistic Korean query, essentially every result showed prose with no
visible match and nothing highlighted. Correct results looked like wrong ones.
What changed
The excerpt is cut around the match. Done in
templateMiddleware, asimple-jekyll-search hook called once per
{placeholder}per result: it receivesthe whole field value and returns what to render, so the full body reaches the JS
and only the window reaches the DOM. No second fetch, no body in the markup.
the most distinct query words. Centring on the first match alone gave "vibe
coding" a window holding only a far-off
coding; it now shows both.spaces, so an unbounded search for one runs past the match and swallows the
window — the same trap as the truncation floor in
post_description.rb. The capdegrades it to a clean character cut.
is escaped here and the only tags introduced are the
<mark>s. Verified with a>query.The index is now real plain text (
plain_text, new_plugins/search_index.rb).strip_html | strip_newlineshad two defects that both reached the reader:"음료 > 탄산음료"verbatim.<p>끝</p><p>시작</p>indexed as끝시작, one token matching neither word.Order is load-bearing and tested: strip tags before decoding entities, or a
post quoting
<script>gets a real element that the strip then deletesalong with the author's words. Inline tags unwrap with no space so
<em>강조</em>된stays one word.
Also: the
snippetfield is gone (84 KB smaller index); the status line says"Showing the first 10 matches." instead of reporting a total the library cannot
know, since it stops scanning at
limit; and.result-tagswas missing from thedark-mode overrides, sitting at 4.2:1 — below AA — so it moves to
$fg-muted(6.2:1).
Retired along the way: the post-render
highlight()pass, itsdata-originalbookkeeping,
escapeRegExp, and the 120 ms debounce that existed to sequencethem. Highlighting happens during render now.
Verification
Driven in a real browser, not reasoned about: a driver script was appended to the
throwaway build so Chrome types into the actual
/search/page against the actualindex. Every result for 어텐션, GRPO,
>,attention transformer,Pixtralandvibe codingcame back with at least one<mark>inside a match-centred excerpt.Checked in both themes.
Gates: 93 tests / 157 assertions, clean build, html-proofer successful, all 16
validate-site checks passed.
Still open, now documented precisely in §9
site.postsorder and stops atlimit, so results are newest-first and a title match does not outrank apassing mention. A sort hook can't fix it — sorting happens after truncation.
Doing it properly means owning the match/sort/slice loop, at which point the
library (used only for fetch, substring match and template fill) should go.
delimiters survive as
\(O(L^2)\)and appear when the window lands on one.Stripping math from the index would fix it at the cost of making characters
inside formulas unsearchable — left alone rather than decided unilaterally.