Vector retrieval quality fixtures (v3.2 / 60-03 / QUAL-01)

[RichardHightower]

Tracking F4 from docs/plans/v3.2-prove-it-plan.md.

README status table:

• Vector search (HNSW + Candle) is Solid — accurate about the mechanism, silent about whether it returns the right things. Zero quality tests (no recall/precision/MRR/nDCG in memory-vector, memory-search, or e2e).
• Topic graph is Works — clustering quality is not benchmarked, by the README's own admission.
• BM25 is the only layer with labelled relevant items (custom-harness recall@k).

Work (60-03)

• Labelled semantic fixture in the custom harness: queries whose relevant items share meaning but not tokens (jwt vs JSON Web Token) so BM25 alone fails and vector must carry it. Report recall@5 for BM25-only, vector-only, hybrid.
• Topic-graph fixture: known clusters, report purity or adjusted Rand index against the labels.
• Commit results. Downgrade any README row whose result does not support Solid.
• Update the positioning doc's Claims Ledger with each number and its artifact path.

Success

Every Solid row in the README status table cites a committed quality result.

[chengyixu]

Living Memory then Proactive Execution — alongside the synonym fixture, I would add one small labelled currentness slice so a semantically good result is not mistaken for a currently usable result.

Each fixture can contain an older assertion and a semantically similar superseding correction for the same normalized subject/predicate/scope, plus a pinned query cutoff. Evaluate BM25-only, vector-only, and hybrid separately with at least:

• current_head_recall@5: whether the revision valid at the cutoff is retrieved.
• historical_leakage@5: whether an older superseded revision appears in the usable/current set.
• no_current_false_recall@5: a control query where no current fact exists, so a semantically close historical result cannot be counted as success.
• deterministic ordering for equal timestamps, using an explicit stable fixture sequence rather than relying on index insertion timing.

This is distinct from a generic recency boost. The fixture should pin the retrieval state first (fact revisions, cutoff, analyzer/chunking/index configuration, model revision, and seed), then assert the expected ordered IDs. Late-arriving corrections must stay out of historical snapshots, and a hybrid improvement should not be accepted if it only increases retrieval of stale heads.

For the Claims Ledger, commit the fixture digest, corpus/index/config/model revisions, raw per-query output, and the metric artifact path—not just a summary percentage. That makes a README Solid claim inspectable and lets reviewers see whether a result came from lexical coverage, embedding similarity, or hybrid fusion.

If “currentness” is not yet part of the runtime retrieval contract, keep this slice diagnostic-only: surface the two metrics and the leakage examples, but do not silently change ranking or suppress history. The benchmark then informs a separately specified retrieval policy instead of baking a product decision into the test harness.

For Klik’s pre-launch design context, this is the truthful Living Memory then Proactive Execution boundary: retained evidence can inform a proposed next step, while current scoped authorization remains required before any external or judgment-dependent effect. Design context: https://pre.hiklik.ai/

[RichardHightower]

60-03 evidence is in #50.

Committed mock-harness numbers on the 16-test paraphrase set:

• BM25 recall@5 = 0.00 (0/16) — benchmarks/results/semantic-bm25.json
• vector recall@5 = 1.00 (16/16) — benchmarks/results/semantic-vector.json
• hybrid recall@5 = 1.00 (16/16) — benchmarks/results/semantic-hybrid.json

--layers is a mock-backend switch. Mock vector is a committed paraphrase lexicon + TF-IDF cosine, not Candle/HNSW. Live memory search is still RouteQuery hybrid.