From 3ea50d8768bfebb0a5cef372f2b2c6f13c5b9aae Mon Sep 17 00:00:00 2001 From: Will Scott Date: Sat, 25 Jul 2026 15:12:56 -0500 Subject: [PATCH 1/3] Label overlap + add Chunks-view legend (v3.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Chuck couldn't tell intentional RAG overlap from duplicate content, and the Chunks view assumed vocabulary it never defined. All deterministic — no AI calls. API (api/chunk.js): - addOverlap() records overlap_word_count on each section it modifies; enhanceChunks() carries it into the response - settings.target_words exposes the active {min,max,target} size range so the UI reads it live instead of mirroring CHUNK_SIZES UI (public/index.html): - Overlap prefix renders dimmed with a "↔ Nw overlap" badge + tooltip explaining the intentional carry-over - Collapsible "How to read this" legend defines Chunk, L#, Path, Sections, w/t, live target size range, and overlap - Chunk meta pill gets an explanatory tooltip; "1 sections" plural fixed - Heading-path breadcrumb labeled "PATH" with tooltip + legend entry - Content Summary stat labels swapped (big=Chunks, small=Sections) to match the chunk cards and legend Smoke-tested on the page Chuck screenshotted (Hayes Barton Place assisted-living: Chunk 2's 29-word overlap now renders labeled) plus searchinfluence.com, Wikipedia RAG article, and getchunks itself. Co-Authored-By: Claude Fable 5 --- SESSION_LOG.md | 18 ++++++ api/chunk.js | 11 +++- public/index.html | 142 +++++++++++++++++++++++++++++++++++++++++----- tasks/todo.md | 52 +++++++++++++++++ 4 files changed, 208 insertions(+), 15 deletions(-) diff --git a/SESSION_LOG.md b/SESSION_LOG.md index 1bb41e3..614f5cd 100644 --- a/SESSION_LOG.md +++ b/SESSION_LOG.md @@ -1,5 +1,23 @@ # getchunks Session Log +## 2026-07-24 — Chunk Quality plan (from Chuck's feedback) + repo moved out of archive + +**Session ran from ontologizer-next; planning only, no code changes here yet.** + +**Origin:** Chuck Wilkins Slack feedback — he couldn't interpret the Chunks view on the Hayes Barton Place assisted-living page and mistook the overlap prefix for duplicate content. Verified root cause: the sentence appears exactly once on the live page (`https://www.hayesbartonplace.com/bloomsbury-at-hayes-barton-place/assisted-living`); the "duplicate" is `addOverlap()` prepending the entire 29-word prior section because the overlap window (35w, 10% of medium target) exceeds the section length. `addOverlap()` doesn't record what it prepended, so the UI can't label it. Will's framing: keep everything deterministic — zero AI tokens. + +**Plan written (NOT yet executed):** "Chunk Quality upgrade (v3.2 → v3.4)" at the bottom of `tasks/todo.md`. Summary: v3.2 = overlap labeling + "how to read this" legend (needs `overlap_word_count` from the API); v3.3 = deterministic per-chunk flags (entity anchor, dangling reference, thin/oversized, generic heading, answer-buried, near-duplicate via Jaccard shingles, Flesch-Kincaid) + 0–100 Chunkability score; v3.4 = client-side BM25 retrieval simulator, paste-queries coverage matrix, markdown optimization-report export. Phase 4 parked: template query generator, `?queries=` deep-link from Ontologizer fan-out, compare mode. + +**Repo moved:** `~/Development/Development Archive/getchunks` → `~/Development/getchunks`. Git and Vercel unaffected (deploys from GitHub `searchinfluence/getchunks`). Working tree has exactly one modification: the plan appended to `tasks/todo.md`. + +**Next session:** get Will's approval on the plan, then branch `feature/chunk-quality-v3.2` off `main`. Smoke-test v3.2 against the Hayes Barton Place URL above (the exact page Chuck screenshotted). + +**2026-07-24 (cont.) — Phase 1 (v3.2) EXECUTED on `feature/chunk-quality-v3.2`, uncommitted, awaiting Will's review before PR:** +- `api/chunk.js`: `addOverlap()` now records `overlap_word_count` per small chunk (carried through `enhanceChunks()`); `settings.target_words` ({min,max,target}) added so the UI reads size ranges live instead of mirroring `CHUNK_SIZES`. +- `public/index.html`: overlap prefix rendered dimmed + dotted-underline with `↔ Nw overlap` badge and "intentional RAG carry-over" tooltip (UI splits the prefix with `^(?:\S+\s+){N}` against the reported count); collapsible "How to read this" legend at top of Chunks view (Chunk/L#/Sections/w-t/live size range/overlap); Content Summary stat labels swapped (they were reversed vs the chunk cards: big=Chunks, small=Sections). +- Smoke-tested via `vercel dev --listen 3001` + Playwright: Hayes Barton Place Chunk 2 sec 2 = `overlap_word_count: 29`, renders badged/dimmed exactly as planned; 7 badges total (2w–35w). 3 more URLs (SI home, Wikipedia RAG, getchunks) — 0 prefix mismatches, no warnings, zero-overlap case clean. Screenshots in session scratchpad. +- Will's review feedback, addressed same-branch: (1) tooltip on the chunk meta pill spelling out each part ("Chunk 1 of 4 on this page • starts at a level-2 heading (an

) • …") + fixed "1 sections" plural; (2) breadcrumb now labeled "PATH" with a tooltip + legend bullet explaining it's the heading path (parent headings above the chunk, starting with the page title) stored for retrieval context. (3) His "actionable layer / score" ask = Phase 2 (v3.3 flags + Chunkability score) — confirmed as next. + ## 2026-07-21 (cont. 2) — Dead-CSS cleanup, cross-links, Ontologizer preview fix **getchunks dead-CSS cleanup (#9, merged):** Removed 235 lines of orphaned rules whose classes left the DOM in the marketing-shell rebuild (.header/.header-content/.logo-*/.tagline/.header-description/.main-section/.form-card/.features/.feature*/old .faq-*/.footer + their @media entries). Verified render pixel-identical at 1440px; `