diff --git a/SESSION_LOG.md b/SESSION_LOG.md index 370b8ae..6b1156e 100644 --- a/SESSION_LOG.md +++ b/SESSION_LOG.md @@ -14,6 +14,22 @@ **Next:** Will reviews v3.3 → then Phase 3 (v3.4): BM25 retrieval simulator, query coverage matrix, markdown optimization-report export. +## 2026-07-25 (cont.) — Phase 3 (v3.4) executed; Chunk Quality plan COMPLETE + +**Will said "get through all the outstanding work."** Built v3.4 on `feature/chunk-quality-v3.4` (stacked on v3.3), PR #12: +- Client-side BM25 (k1=1.2/b=0.75, no deps) over sections — heading indexed with text, overlap included, mirroring what the JSONL export feeds a real pipeline. All in-browser. +- "Test retrieval" panel: single query → top-5 ranked sections with scores/bars/matched terms; winner highlighted with its flag chips and a stands-alone verdict (Hayes: "Chunk 2 · section 1 wins but is flagged (Dangling reference)"). +- Coverage check: one query per line → matrix (query / status / best section / score / terms), covered = ≥60% of query terms matched, weak, or gap. Summary counts + Copy CSV. +- "Copy optimization report": markdown deliverable — score/grade, top fixes, flags grouped Fix first / Then improve / Worth knowing, coverage table, content-gaps list, settings. Verified via clipboard read in Playwright (Hayes: 4k chars, "dog boarding rates" correctly a gap, "cost per month" correctly weak). +- State: coverageRows reset per extraction; all buttons delegated (view re-renders per run); Enter submits the single query. + +**Backlog cleanup:** `feature/enhanced-chunking` turned out to be already deleted (backlog item was stale); pruned leftover local tracking refs for the merged `chore/remove-dead-css` + `feature/marketing-shell-parity` branches. Branch list is now just main + the three chunk-quality branches. + +**Open at day's end:** +1. **Merge chain awaiting Will:** #10 (v3.2, base main) → #11 (v3.3, retarget to main after #10) → #12 (v3.4, retarget after #11). Merging main auto-deploys. +2. Fixture/snapshot tests (split/merge/overlap/analyzer) — deferred since v3.1, still the one real backlog item. +3. Phase 4 optional: template query generator, `?queries=` Ontologizer deep-link, compare mode, README rubric docs. + ## 2026-07-24 — Chunk Quality plan (from Chuck's feedback) + repo moved out of archive **Session ran from ontologizer-next; planning only, no code changes here yet.** diff --git a/public/index.html b/public/index.html index 12ec2b2..f1998ea 100644 --- a/public/index.html +++ b/public/index.html @@ -750,6 +750,155 @@ .flag-chip.info { background: rgba(52, 144, 181, 0.1); border-color: rgba(52, 144, 181, 0.4); color: var(--info-blue); } .flag-chip.clean { background: rgba(36, 171, 89, 0.08); border-color: rgba(36, 171, 89, 0.45); color: var(--si-report-green); cursor: default; } + /* Retrieval simulator + coverage matrix (v3.4) */ + .retrieval-panel p { + margin-bottom: 14px; + font-size: 0.95rem; + } + + .retrieval-row { + display: flex; + gap: 10px; + flex-wrap: wrap; + margin-bottom: 16px; + } + + .retrieval-row input, + .retrieval-panel textarea { + flex: 1 1 300px; + background: var(--content-bg); + border: 1px solid var(--border-gray); + border-radius: 8px; + padding: 10px 14px; + font-family: inherit; + font-size: 0.95rem; + color: var(--content-text); + } + + .retrieval-panel textarea { + width: 100%; + resize: vertical; + margin-bottom: 12px; + } + + .panel-btn { + background: var(--si-navy); + border: 1px solid var(--si-navy); + color: var(--white); + padding: 10px 18px; + border-radius: 8px; + font-size: 0.9rem; + font-weight: 700; + font-family: inherit; + cursor: pointer; + } + + .panel-btn.secondary { + background: var(--content-bg); + border-color: var(--border-gray); + color: var(--secondary-content); + } + + .retrieval-actions { + display: flex; + gap: 10px; + flex-wrap: wrap; + } + + .coverage-heading { + font-size: 1.05rem; + font-weight: 800; + margin: 24px 0 8px; + padding-top: 20px; + border-top: 1px solid var(--border-gray); + } + + .retrieval-hit { + background: var(--content-bg); + border: 1px solid var(--border-gray); + border-radius: 8px; + padding: 12px 16px; + margin-bottom: 10px; + font-size: 0.92rem; + } + + .retrieval-hit.winner { + border-color: var(--si-report-green); + border-width: 2px; + } + + .retrieval-hit-head { + display: flex; + justify-content: space-between; + gap: 12px; + flex-wrap: wrap; + font-weight: 700; + margin-bottom: 4px; + } + + .retrieval-bar { + height: 6px; + border-radius: 3px; + background: var(--si-medium-blue); + margin: 6px 0; + } + + .retrieval-snippet { + color: var(--muted-text); + font-size: 0.88rem; + } + + .retrieval-verdict { + margin: 8px 0 14px; + font-size: 0.92rem; + font-weight: 600; + } + + .coverage-table-wrap { + overflow-x: auto; + margin-top: 14px; + } + + .coverage-table { + width: 100%; + border-collapse: collapse; + font-size: 0.9rem; + background: var(--content-bg); + } + + .coverage-table th, + .coverage-table td { + border: 1px solid var(--border-gray); + padding: 8px 12px; + text-align: left; + vertical-align: top; + } + + .coverage-table th { + background: var(--background-gray); + font-weight: 700; + } + + .cov-badge { + display: inline-block; + font-size: 0.78rem; + font-weight: 700; + padding: 2px 10px; + border-radius: 12px; + border: 1px solid; + white-space: nowrap; + } + + .cov-badge.covered { background: rgba(36, 171, 89, 0.1); border-color: rgba(36, 171, 89, 0.5); color: var(--si-report-green); } + .cov-badge.weak { background: rgba(245, 158, 11, 0.1); border-color: rgba(245, 158, 11, 0.55); color: #b45309; } + .cov-badge.gap { background: rgba(231, 76, 60, 0.1); border-color: rgba(231, 76, 60, 0.5); color: #c0392b; } + + .coverage-summary { + margin-top: 12px; + font-weight: 700; + font-size: 0.92rem; + } + /* Pure-CSS tooltips — native title="" is unreliable (long hover delay, never shown in some embedded browsers), so hints render as a styled ::after that appears immediately on hover. */ @@ -2173,14 +2322,32 @@

Frequently asked questions

URL.revokeObjectURL(url); }); - // "Show only flagged" toggle — delegated because the Chunks view - // re-renders its HTML on every extraction + // Chunks-view controls — delegated because the view re-renders + // its HTML on every extraction chunksContent.addEventListener('click', function(e) { - const btn = e.target.closest('#flagged-filter'); - if (!btn) return; - const on = chunksContent.classList.toggle('flags-only'); - btn.classList.toggle('active', on); - btn.textContent = on ? 'Showing flagged chunks — show all' : 'Show only flagged chunks'; + const filterBtn = e.target.closest('#flagged-filter'); + if (filterBtn) { + const on = chunksContent.classList.toggle('flags-only'); + filterBtn.classList.toggle('active', on); + filterBtn.textContent = on ? 'Showing flagged chunks — show all' : 'Show only flagged chunks'; + return; + } + if (e.target.closest('#retrieval-run')) { runRetrievalTest(); return; } + if (e.target.closest('#coverage-run')) { runCoverage(); return; } + const csvBtn = e.target.closest('#copy-coverage-csv'); + if (csvBtn) { copyToClipboard(coverageCsv(), csvBtn, 'Copied ✓'); return; } + const reportBtn = e.target.closest('#copy-report'); + if (reportBtn && currentData) { + trackEvent('report_copied'); + copyToClipboard(buildOptimizationReport(currentData), reportBtn, 'Copied ✓'); + } + }); + + chunksContent.addEventListener('keydown', function(e) { + if (e.key === 'Enter' && e.target.id === 'retrieval-query') { + e.preventDefault(); + runRetrievalTest(); + } }); // New extraction @@ -2289,6 +2456,7 @@

Frequently asked questions

function displayChunksView(data) { let html = ''; chunksContent.classList.remove('flags-only'); // fresh run, fresh filter + coverageRows = null; // coverage results belong to the previous page // Collapsible legend (v3.2) — defines the vocabulary before the data const tw = data.settings?.target_words; @@ -2340,6 +2508,28 @@

Frequently asked questions

html += ``; } + // Retrieval simulator + coverage matrix (v3.4) + if (data.big_chunks && data.big_chunks.length) { + html += `
`; + html += `

🔎 Test retrieval

`; + html += `

Type what a searcher (or an AI assistant) would ask, and see which section wins — ranked with BM25, the same scoring family real retrieval systems use. Nothing leaves your browser.

`; + html += `
`; + html += ``; + html += ``; + html += `
`; + html += `
`; + html += `

Coverage check

`; + html += `

Paste queries — one per line, from Search Console, keyword tools, or an Ontologizer fan-out — to see which are covered, weak, or missing entirely.

`; + html += ``; + html += `
`; + html += ``; + html += ``; + html += ``; + html += `
`; + html += `
`; + html += `
`; + } + // Source metadata block (new in v3) if (data.source) { const s = data.source; @@ -2509,6 +2699,215 @@

Frequently asked questions

chunksContent.innerHTML = html; } + // --- Retrieval simulator (v3.4) — client-side BM25, no deps ---- + let coverageRows = null; // last coverage run, feeds CSV + report + + function bmStem(w) { + return w.length > 4 ? w.replace(/(ings|ing|ers|er|ies|ed|es|s)$/, '') : w.replace(/s$/, ''); + } + + function bm25Tokenize(text) { + return (text || '').toLowerCase().split(/[^a-z0-9]+/).filter(w => w.length >= 2).map(bmStem); + } + + // One doc per section (retrieval unit), heading indexed with it — + // the same shape the JSONL export feeds a real RAG pipeline. + function buildRetrievalDocs(data) { + const docs = []; + (data.big_chunks || []).forEach(bc => { + (bc.small_chunks || []).forEach(sc => { + const terms = bm25Tokenize(`${bc.title} ${sc.text}`); + const tf = {}; + terms.forEach(t => { tf[t] = (tf[t] || 0) + 1; }); + docs.push({ + label: `Chunk ${bc.big_chunk_index} · section ${sc.chunk_index}`, + title: bc.title, + flags: bc.flags || [], + text: sc.text, + len: terms.length, + tf, + }); + }); + }); + return docs; + } + + function bm25Search(docs, query) { + const qTerms = [...new Set(bm25Tokenize(query))]; + if (!qTerms.length || !docs.length) return { qTerms, hits: [] }; + const N = docs.length; + const avgLen = docs.reduce((s, d) => s + d.len, 0) / N || 1; + const k1 = 1.2, b = 0.75; + const df = {}; + qTerms.forEach(q => { df[q] = docs.filter(d => d.tf[q]).length; }); + const hits = docs.map(d => { + let score = 0, matched = 0; + qTerms.forEach(q => { + const f = d.tf[q] || 0; + if (!f) return; + matched++; + const idf = Math.log(1 + (N - df[q] + 0.5) / (df[q] + 0.5)); + score += idf * (f * (k1 + 1)) / (f + k1 * (1 - b + b * d.len / avgLen)); + }); + return { doc: d, score, matched }; + }).filter(h => h.score > 0); + hits.sort((a, b2) => b2.score - a.score); + return { qTerms, hits }; + } + + function coverageStatus(hit, qTermCount) { + if (!hit || hit.score <= 0) return 'gap'; + return hit.matched / qTermCount >= 0.6 ? 'covered' : 'weak'; + } + + function runRetrievalTest() { + const query = document.getElementById('retrieval-query')?.value.trim(); + const out = document.getElementById('retrieval-results'); + if (!query || !out || !currentData) return; + trackEvent('retrieval_tested'); + const { qTerms, hits } = bm25Search(buildRetrievalDocs(currentData), query); + if (!hits.length) { + out.innerHTML = `
✗ No section matches this query — a content gap on this page.
`; + return; + } + let html = ''; + const top = hits.slice(0, 5); + const winner = top[0]; + const highFlags = winner.doc.flags.filter(f => f.severity === 'high'); + html += `
`; + html += highFlags.length + ? `⚠ "${escapeHtml(winner.doc.label)}" wins but is flagged (${escapeHtml(highFlags.map(f => f.label).join(', '))}) — it may not stand alone when retrieved.` + : `✓ "${escapeHtml(winner.doc.label)}" wins and has no high-severity flags.`; + html += `
`; + top.forEach((h, i) => { + const width = Math.max(6, Math.round((h.score / winner.score) * 100)); + html += `
`; + html += `
#${i + 1} ${escapeHtml(h.doc.label)} — ${escapeHtml(h.doc.title)}${h.score.toFixed(2)} · ${h.matched}/${qTerms.length} terms
`; + html += `
`; + if (i === 0 && h.doc.flags.length) { + html += `
${h.doc.flags.map(f => `${escapeHtml(f.label)}`).join(' ')}
`; + } + html += `
${escapeHtml(h.doc.text.slice(0, 160))}${h.doc.text.length > 160 ? '…' : ''}
`; + html += `
`; + }); + out.innerHTML = html; + } + + function runCoverage() { + const box = document.getElementById('coverage-queries'); + const out = document.getElementById('coverage-results'); + if (!box || !out || !currentData) return; + const queries = box.value.split('\n').map(q => q.trim()).filter(Boolean); + if (!queries.length) return; + trackEvent('coverage_run', { queries: queries.length }); + const docs = buildRetrievalDocs(currentData); + coverageRows = queries.map(query => { + const { qTerms, hits } = bm25Search(docs, query); + const best = hits[0] || null; + const status = coverageStatus(best, qTerms.length || 1); + return { + query, + status, + bestLabel: best ? `${best.doc.label} — ${best.doc.title}` : null, + score: best ? best.score.toFixed(2) : '0', + matched: best ? `${best.matched}/${qTerms.length}` : `0/${qTerms.length}`, + }; + }); + const counts = { covered: 0, weak: 0, gap: 0 }; + coverageRows.forEach(r => counts[r.status]++); + let html = `
`; + html += ``; + coverageRows.forEach(r => { + html += ``; + html += ``; + html += ``; + html += ``; + }); + html += `
QueryStatusBest sectionScoreTerms
${escapeHtml(r.query)}${r.status}${r.bestLabel ? escapeHtml(r.bestLabel) : '—'}${escapeHtml(r.score)}${escapeHtml(r.matched)}
`; + html += `
${counts.covered} covered · ${counts.weak} weak · ${counts.gap} ${counts.gap === 1 ? 'gap' : 'gaps'}
`; + out.innerHTML = html; + const csvBtn = document.getElementById('copy-coverage-csv'); + if (csvBtn) csvBtn.hidden = false; + } + + function csvField(value) { + const s = String(value ?? ''); + return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s; + } + + function coverageCsv() { + const lines = ['query,status,best_section,score,matched_terms']; + (coverageRows || []).forEach(r => { + lines.push([r.query, r.status, r.bestLabel || '', r.score, r.matched].map(csvField).join(',')); + }); + return lines.join('\n'); + } + + // Markdown export — the SEO-facing deliverable (JSON stays for devs) + function buildOptimizationReport(data) { + const md = (s) => String(s).replace(/\|/g, '\\|'); + const lines = []; + lines.push(`# Chunk optimization report — ${data.source?.url || ''}`); + lines.push(''); + lines.push('Generated by GetChunks. Deterministic analysis: the same page always produces the same results, so re-run after edits to verify the change. No AI involved.'); + const ca = data.chunkability; + if (ca) { + const flagged = data.big_chunks.filter(c => (c.flags || []).some(f => f.severity !== 'info')).length; + lines.push('', `**Chunkability: ${ca.score}/100 (grade ${ca.grade})** — ${flagged} of ${data.big_chunks.length} chunks need work`); + if (ca.top_fixes?.length) { + lines.push('', '## Top fixes', ''); + ca.top_fixes.forEach((f, i) => { + lines.push(`${i + 1}. **${f.label}${f.count > 1 ? ` (×${f.count})` : ''}** — ${f.fix}${f.points ? ` (up to +${f.points} pts)` : ''}`); + }); + } + } + const groups = { high: [], warn: [], info: [] }; + (data.big_chunks || []).forEach(bc => (bc.flags || []).forEach(f => groups[f.severity]?.push({ bc, f }))); + [['high', 'Fix first'], ['warn', 'Then improve'], ['info', 'Worth knowing']].forEach(([sev, title]) => { + if (!groups[sev].length) return; + lines.push('', `## ${title}`, ''); + groups[sev].forEach(({ bc, f }) => { + lines.push(`- **Chunk ${bc.big_chunk_index} "${bc.title}" — ${f.label}:** ${f.message} *Fix: ${f.fix}*`); + }); + }); + if (coverageRows?.length) { + lines.push('', `## Query coverage (${coverageRows.length} queries)`, ''); + lines.push('| Query | Status | Best section | Score | Terms |'); + lines.push('|---|---|---|---|---|'); + coverageRows.forEach(r => { + lines.push(`| ${md(r.query)} | ${r.status} | ${r.bestLabel ? md(r.bestLabel) : '—'} | ${r.score} | ${r.matched} |`); + }); + const gaps = coverageRows.filter(r => r.status === 'gap'); + if (gaps.length) { + lines.push('', '### Content gaps', ''); + gaps.forEach(g => lines.push(`- "${g.query}" — no section on the page addresses this. Add one, or expand an existing section to cover it.`)); + } + } + if (data.settings) { + lines.push('', `Settings: ${data.settings.chunk_size} chunks (${data.settings.target_words ? `${data.settings.target_words.min}–${data.settings.target_words.max} words` : ''}), ${data.settings.overlap_words}-word overlap, ${data.settings.strategy} strategy.`); + } + return lines.join('\n'); + } + + function copyToClipboard(text, btn, doneLabel) { + const restore = btn.textContent; + const done = () => { + btn.textContent = doneLabel; + setTimeout(() => { btn.textContent = restore; }, 1600); + }; + if (navigator.clipboard?.writeText) { + navigator.clipboard.writeText(text).then(done); + } else { + const ta = document.createElement('textarea'); + ta.value = text; + document.body.appendChild(ta); + ta.select(); + document.execCommand('copy'); + document.body.removeChild(ta); + done(); + } + } + function escapeHtml(text) { // Escapes quotes too — this output lands in attribute values // (title="", href=""), where the textContent/innerHTML trick diff --git a/tasks/todo.md b/tasks/todo.md index 148ef7e..ea72483 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -139,11 +139,11 @@ Branch: `feature/chunk-quality-v3.3` ### Phase 3 — Action (v3.4): retrieval simulator + report export Branch: `feature/chunk-quality-v3.4` -- [ ] Client-side BM25 (~60 lines, no deps) over small chunks -- [ ] "Test retrieval" panel: single question → ranked chunks with scores, winner highlighted with its flags shown (does the winning chunk stand alone?) -- [ ] Multi-query textarea (one per line — from GSC, keyword tools, or an Ontologizer fan-out run) → coverage matrix: query → best chunk → score → covered / weak / gap -- [ ] "Copy optimization report" — markdown export: URL, score, flags grouped by severity with specific edits, coverage gaps. This is the SEO-facing deliverable; JSON stays for devs -- [ ] CSV copy for the coverage matrix +- [x] Client-side BM25 (k1=1.2, b=0.75, ~50 lines, no deps) over sections (heading indexed with text, overlap included — same shape as the JSONL export) +- [x] "Test retrieval" panel: single question → top 5 ranked sections with scores/bars/matched-terms, winner highlighted with its flag chips + stands-alone verdict ("wins but is flagged — may not stand alone") +- [x] Multi-query textarea → coverage matrix: query → best section → score → covered (≥60% terms matched) / weak / gap (no match), with summary counts +- [x] "Copy optimization report" — markdown export: URL, score/grade, top fixes, flags grouped Fix first / Then improve / Worth knowing with specific edits, coverage table + content gaps, settings line +- [x] CSV copy for the coverage matrix (proper quoting) ### Phase 4 — Later / optional - [ ] Template query generator (entity × heading permutations) to prefill the simulator — zero-AI fan-out approximation @@ -151,5 +151,10 @@ Branch: `feature/chunk-quality-v3.4` - [ ] Compare mode: re-run same URL, show score delta - [ ] Docs: README section explaining every flag and the score formula (determinism = explainability; publish the rubric) -### Review -_(fill in after execution)_ +### Review — all 3 phases executed 2026-07-24/25 + +**Shipped (stacked PRs, merge in order):** #10 v3.2 (overlap labeling + legend + schemeless URLs + CSS tooltips) → #11 v3.3 (flags + Chunkability score + nested-extraction fix) → #12 v3.4 (BM25 simulator + coverage matrix + report export). Zero AI calls anywhere; everything deterministic and Playwright-verified against the Hayes Barton Place page Chuck screenshotted (51/F, 29w overlap labeled) plus seasidefl, SI home, Wikipedia RAG (79/C control), Liberty Senior Living. + +**Deviations from plan, all reasoned:** score model became per-chunk-average instead of flat deductions (proportionality); thin-section got an absolute 100w floor; anchor terms gained initialisms; answer-buried requires ≥2 heading content words; the near-duplicate flag surfaced the deferred nested-content extraction bug, fixed at the root in `extractContentPieces`/`stopCondition`. + +**Remaining backlog (not in this plan):** fixture/snapshot tests for split/merge/overlap/analyzer (deferred since v3.1); Phase 4 optional items (template query generator, `?queries=` deep-link from Ontologizer, compare mode, README rubric docs).