Tracking F3 from docs/plans/v3.2-prove-it-plan.md.
The harness is done: memory-bench locomo --backend cli --scorer llm-judge is wired; benchmarks/scripts/download-locomo.sh fetches the real dataset; the judge honors OPENAI_API_KEY or ANTHROPIC_API_KEY and records the model id. The only committed results are mock/mock (benchmarks/results/custom-harness-mock.json, benchmarks/results/locomo-smoke.json).
README Benchmarks section and docs/positioning/agent-memory-vs-competition.md both forbid a comparison claim until a real-backend, real-judge result is committed next to it.
Work
- 60-01 (prerequisite): per-conversation daemon isolation on
--backend cli (fresh --data-dir, wait for outbox drain by polling the checkpoint, not sleep 60). Smoke this in CI against a real daemon with the mock judge.
- 60-02 (maintainer): run
memory-bench locomo --dataset locomo-data --backend cli --scorer llm-judge --output benchmarks/results/locomo-YYYY-MM-DD.json. Record hardware, model id, temperature, dataset SHA. Cap the first run at 200 questions if cost is a concern, then the full set.
Success
benchmarks/results/ contains one locomo_llm_judge result on the real dataset, real backend, hardware and model recorded. Positioning doc and README then cite that path.
Tracking F3 from docs/plans/v3.2-prove-it-plan.md.
The harness is done:
memory-bench locomo --backend cli --scorer llm-judgeis wired;benchmarks/scripts/download-locomo.shfetches the real dataset; the judge honorsOPENAI_API_KEYorANTHROPIC_API_KEYand records the model id. The only committed results are mock/mock (benchmarks/results/custom-harness-mock.json,benchmarks/results/locomo-smoke.json).README Benchmarks section and docs/positioning/agent-memory-vs-competition.md both forbid a comparison claim until a real-backend, real-judge result is committed next to it.
Work
--backend cli(fresh--data-dir, wait for outbox drain by polling the checkpoint, notsleep 60). Smoke this in CI against a real daemon with the mock judge.memory-bench locomo --dataset locomo-data --backend cli --scorer llm-judge --output benchmarks/results/locomo-YYYY-MM-DD.json. Record hardware, model id, temperature, dataset SHA. Cap the first run at 200 questions if cost is a concern, then the full set.Success
benchmarks/results/contains onelocomo_llm_judgeresult on the real dataset, real backend, hardware and model recorded. Positioning doc and README then cite that path.