Repository navigation
v1.10.0: AI Eval — grounding eval for AI Analysis - #6
Merged
Merged
Conversation
New developer-only view (showView('ai-eval')) that runs each of the 8
AI_PROMPTS 3 times on the synthetic samples/ data and classifies every
dollar amount and percentage in the answers as grounded, derived
(verified in code), math error, forecast or unsupported, with a
chance-match calibration and EVAL.md / results JSON export.
- samples/ only, verified by SHA-256; parsed in a synchronous State
sandbox so the user's data is never read, sent or overwritten
- callClaude() raw mode so the eval sends the AI tab's exact request
- version bump to v1.10.0; README Evaluation section; CLAUDE.md §21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
5 tasks done
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QoR1wxcaDjVVk3xmd5rRCt
…nding-eval Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QoR1wxcaDjVVk3xmd5rRCt
This was referenced Oct 25, 2026
Only version strings and the changelog conflicted. The 1.10.0 entry stays on top of master's 1.9.1–1.9.6 entries and is re-dated 2026-10-25 so it isn't older than 1.9.6 below it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JUVH7ZouV4pEBT7ytKaZzN
…fer re-scoring - EVAL.md Method now matches the code: 2.5% rounding cap (said 5%), the context check for rounded/small grounded matches, the real derived tolerances, and anchored-not-just-grounded operands - Forecast figures: say "treated as derived, not unsupported, reported separately" instead of "counted as derived" while excluding them - "Literal grounding rule" replaces the misleading "strict tolerance" - Load results JSON refuses runs made with different AI_PROMPTS (runs are mapped to prompts by index) or without run meta; headline and header use the run's own repeats and model - Escape HTML in model excerpts written to EVAL.md tables - escape instead of a literal BOM in the source - Tagging: whole-word matches for short advice words (cap, aim, try…) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
Opened from disk, the browser blocks fetch() of samples/, so the view used to try anyway and Chrome/Edge printed a CORS error and a failed GET in red in the console. It looked like a crash although the picker then appeared. Now it skips the fetch on file:// and explains why the three files need to be picked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
…g it Run: claude-sonnet-5-5, 8 prompts x 3, samples/ only, 24/24 answered. 703 figures: 48.6% grounded, 20.2% derived and verified (68.8% traceable), 28.4% unsupported, 2.7% forecast, no math errors. Reading the real answers exposed three matcher bugs, fixed and disclosed in EVAL.md with before/after numbers (48.6/19.8/28.9 -> 48.6/20.2/28.4): - table cells are now checked with their column header - a restated figure must match within its written precision (was +-0.5 pts, so "-0.4 pts" matched "-0.1%") - advice/benchmark tags: whole-word regressions (capping, trimming) and common phrasing; excerpt highlight no longer shifts on indented lines EVAL.md's Run line is now in UTC so re-scoring anywhere gives the same text. docs/EVAL.md adds a hand check of 75 random figures (all 50 grounded and derived correct; 4 of 25 unsupported were correct derivations the matcher can't verify) and two findings about the AI feature: 15/24 answers hit max_tokens, and monthly figures rest on a guessed period length because the summary gives no dates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
gitluis
marked this pull request as ready for review
October 26, 2026 03:32
This was referenced Oct 26, 2026
Owner
Author
Review — PR #6 at
|
Owner
Author
|
Approval recorded (chat), per CLAUDE.md §0
Merging next. #12 will be retargeted to Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base:
master. Merge before #10 (v1.11.0).Summary
A developer-only AI Eval view that measures how well AI Analysis answers are grounded in the data the model was sent. Open it from DevTools with
showView('ai-eval'); like DB Inspector, it isn't in the sidebar. docs/EVAL.md has the full method and results.Results — first full run
claude-sonnet-5-5, 8 prompts × 3,samples/only. All 24 calls answered on the first attempt. Out of 703 dollar and percentage figures:Unsupported figures are mostly recommendations and benchmarks: 139 advice/targets and 29 benchmarks, against 11 claims about the data.
Hand check of 75 random figures:
So the number errs low rather than high.
Chance-match rate: 12.3%.
Stability across repeats: grounded 47.0–50.2%.
Found by the eval, not fixed here (separate PRs, if you want them):
max_tokens, and the AI tab shows them cut off mid-sentence.What changed
js/views-eval.js(new) — theAIEvalnamespace and view (the only new top-level name).js/views-ai.js—callClaude(q, label, { raw:true, summary }). Sends the same request but touches no DOM, busy flag or history. The AI tab now uses the same innersend().index.html— loadsviews-eval.jsbeforeinit.js.docs/EVAL.md,docs/eval-results.json— results, with the raw answers and every figure's classification.index.htmlfooter, README andviews-settings.js.Revisions since opening
bd2e824 — review fixes:
AI_PROMPTSis refused;087452c —
file://: goes straight to the sample picker, with no red CORS error in the console.14c2879 — real results, plus three matcher bugs found by reading the answers. Each is disclosed in EVAL.md with before and after numbers (48.6/19.8/28.9 → 48.6/20.2/28.4):
No rule was added just to raise the score.
How it was tested
samples/;State, localStorage, IndexedDB and AI history are unchanged after a full run;file://there are no console errors, and the picker works;file://) on 2026-10-26 03:19–03:25 UTC. Re-scoring his exported JSON here reproduced his browser's numbers exactly before the matcher fixes. The committed JSON regenerates EVAL.md exactly, apart from the hand-written section, and an independent Python recount matches every number.Please check before merging
send()).https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
Generated by Claude Code