Skip to content

v1.10.0: AI Eval — grounding eval for AI Analysis - #6

Merged
gitluis merged 7 commits into
masterfrom
ai-grounding-eval
Oct 26, 2026
Merged

gitluis merged 7 commits into
masterfrom
ai-grounding-eval

Conversation

@gitluis

@gitluis gitluis commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Base: master. Merge before #10 (v1.11.0).

Summary

A developer-only AI Eval view that measures how well AI Analysis answers are grounded in the data the model was sent. Open it from DevTools with showView('ai-eval'); like DB Inspector, it isn't in the sidebar. docs/EVAL.md has the full method and results.

Results — first full run

claude-sonnet-5-5, 8 prompts × 3, samples/ only. All 24 calls answered on the first attempt. Out of 703 dollar and percentage figures:

Grounded Derived ✓ Traceable Math error Forecast Unsupported
48.6% 20.2% 68.8% 0 2.7% 28.4%
  • Unsupported figures are mostly recommendations and benchmarks: 139 advice/targets and 29 benchmarks, against 11 claims about the data.

  • Hand check of 75 random figures:

    • every grounded and derived figure checked was correct;
    • 4 of 25 unsupported were actually correct derivations the matcher can't verify.

    So the number errs low rather than high.

  • Chance-match rate: 12.3%.

  • Stability across repeats: grounded 47.0–50.2%.

Found by the eval, not fixed here (separate PRs, if you want them):

  • 15 of 24 answers hit max_tokens, and the AI tab shows them cut off mid-sentence.
  • The summary gives no date span, so the model guesses how many months the totals cover. One answer gave groceries as ~$470/month; the actual figure is ~$545.

What changed

  • js/views-eval.js (new) — the AIEval namespace and view (the only new top-level name).
  • js/views-ai.js — callClaude(q, label, { raw:true, summary }). Sends the same request but touches no DOM, busy flag or history. The AI tab now uses the same inner send().
  • index.html — loads views-eval.js before init.js.
  • docs/EVAL.md, docs/eval-results.json — results, with the raw answers and every figure's classification.
  • v1.10.0 in CHANGELOG, the index.html footer, README and views-settings.js.
  • README: new Evaluation section with the result. CLAUDE.md: §1, §3, §12, §13, and a new §21, including a rule for changing the matcher after a run.

Revisions since opening

  • bd2e824 — review fixes:

    • EVAL.md's Method section now matches the code;
    • forecast wording is consistent;
    • JSON from different AI_PROMPTS is refused;
    • model text is HTML-escaped in EVAL.md;
    • the literal BOM is now an escape sequence.
  • 087452c — file://: goes straight to the sample picker, with no red CORS error in the console.

  • 14c2879 — real results, plus three matcher bugs found by reading the answers. Each is disclosed in EVAL.md with before and after numbers (48.6/19.8/28.9 → 48.6/20.2/28.4):

    • table cells are now checked with their column header;
    • a restated figure must match within its written precision;
    • tagging regressions are fixed.

    No rule was added just to raise the score.

How it was tested

  • Headless Chromium, mocked API (on master v1.9.6 + this PR):
    • the sandbox summary is byte-identical to a normal import of samples/;
    • your State, localStorage, IndexedDB and AI history are unchanged after a full run;
    • no user data and no key appear in any request body or export;
    • a 529 error is retried;
    • 36 classifier unit cases pass;
    • JSON with edited prompts is refused, and HTML in model text is escaped;
    • BOM- and CRLF-encoded sample copies are accepted, and a tampered one is refused;
    • on file:// there are no console errors, and the picker works;
    • the AI tab behaves the same on success and on 401.
  • Real run: done by Luis in Edge (file://) on 2026-10-26 03:19–03:25 UTC. Re-scoring his exported JSON here reproduced his browser's numbers exactly before the matcher fixes. The committed JSON regenerates EVAL.md exactly, apart from the hand-written section, and an independent Python recount matches every number.

Please check before merging

  • Read EVAL.md's Hand check and Changes made after the run sections. The resume number rests on them.
  • Spot-check a few rows in the unsupported-figures table.
  • The AI Analysis tab still works normally with your key (it now goes through send()).

https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq


Generated by Claude Code

New developer-only view (showView('ai-eval')) that runs each of the 8
AI_PROMPTS 3 times on the synthetic samples/ data and classifies every
dollar amount and percentage in the answers as grounded, derived
(verified in code), math error, forecast or unsupported, with a
chance-match calibration and EVAL.md / results JSON export.

- samples/ only, verified by SHA-256; parsed in a synchronous State
  sandbox so the user's data is never read, sent or overwritten
- callClaude() raw mode so the eval sends the AI tab's exact request
- version bump to v1.10.0; README Evaluation section; CLAUDE.md §21

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
@gitluis gitluis removed their assignment Oct 25, 2026
…nding-eval

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QoR1wxcaDjVVk3xmd5rRCt
Base automatically changed from data-cleanup-and-reset to master October 25, 2026 03:41
gitluis pushed a commit that referenced this pull request Oct 26, 2026
Only version strings and the changelog conflicted. The 1.10.0 entry
stays on top of master's 1.9.1–1.9.6 entries and is re-dated
2026-10-25 so it isn't older than 1.9.6 below it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JUVH7ZouV4pEBT7ytKaZzN
…fer re-scoring

- EVAL.md Method now matches the code: 2.5% rounding cap (said 5%),
  the context check for rounded/small grounded matches, the real
  derived tolerances, and anchored-not-just-grounded operands
- Forecast figures: say "treated as derived, not unsupported, reported
  separately" instead of "counted as derived" while excluding them
- "Literal grounding rule" replaces the misleading "strict tolerance"
- Load results JSON refuses runs made with different AI_PROMPTS (runs
  are mapped to prompts by index) or without run meta; headline and
  header use the run's own repeats and model
- Escape HTML in model excerpts written to EVAL.md tables
-  escape instead of a literal BOM in the source
- Tagging: whole-word matches for short advice words (cap, aim, try…)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
Opened from disk, the browser blocks fetch() of samples/, so the view
used to try anyway and Chrome/Edge printed a CORS error and a failed
GET in red in the console. It looked like a crash although the picker
then appeared. Now it skips the fetch on file:// and explains why the
three files need to be picked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
…g it

Run: claude-sonnet-5-5, 8 prompts x 3, samples/ only, 24/24 answered.
703 figures: 48.6% grounded, 20.2% derived and verified (68.8%
traceable), 28.4% unsupported, 2.7% forecast, no math errors.

Reading the real answers exposed three matcher bugs, fixed and
disclosed in EVAL.md with before/after numbers (48.6/19.8/28.9 ->
48.6/20.2/28.4):
- table cells are now checked with their column header
- a restated figure must match within its written precision
  (was +-0.5 pts, so "-0.4 pts" matched "-0.1%")
- advice/benchmark tags: whole-word regressions (capping, trimming)
  and common phrasing; excerpt highlight no longer shifts on
  indented lines
EVAL.md's Run line is now in UTC so re-scoring anywhere gives the
same text.

docs/EVAL.md adds a hand check of 75 random figures (all 50 grounded
and derived correct; 4 of 25 unsupported were correct derivations
the matcher can't verify) and two findings about the AI feature:
15/24 answers hit max_tokens, and monthly figures rest on a guessed
period length because the summary gives no dates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
@gitluis

gitluis commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

Review — PR #6 at 14c2879

Verdict: ready to merge. No blocking issues. Merge before #10; #12 and #13 build on top of it.

Checked

  • Scope matches the request:
    • a developer-only ai-eval view, not in the sidebar;
    • samples/ only, date range All Time;
    • the 8 AI_PROMPTS × 3 through buildFinancialSummary() → callClaude() with AI_MODEL;
    • figures classified grounded / derived / unsupported, with forecast treated as derived;
    • per-prompt and overall results, and every unsupported figure with its excerpt;
    • docs/EVAL.md and a README Evaluation section.
  • Project rules: no npm or build step; one new top-level name (AIEval); v1.10.0 in all four places; CLAUDE.md §3, §12, §13 and a new §21.
  • Safety: samples are SHA-256-checked; parsing runs in a synchronous State sandbox; nothing is persisted. The API key appears only in the request header, never in the summary, the results JSON or EVAL.md.
  • Results: real run, 24/24 answered. Re-scoring the uploaded JSON reproduced the browser's numbers, the committed JSON regenerates EVAL.md, and an independent recount matches every figure.
  • Tests: every suite in the description passes on the merged branch (master v1.9.6 + this PR). Merge state is clean.

Non-blocking notes

  1. Percentages are matched more loosely than dollars: 19.8% of deliberately wrong percentages were still accepted, vs 10.9% of dollar amounts. One hand-checked "49%" was credited through a coincidental top 7 categories ÷ income rule. EVAL.md discloses both. Tighten this in a later eval revision, before a new run.
  2. EVAL.md puts groceries at "about $545/month" (173 days = 5.7 months). Counting six calendar months gives $516. The real point is that the model's guesses differed between answers. v1.10.2: AI summary states its dates; stop the model guessing the period length #13 removes the guess, and its re-run replaces that section.
  3. The advice / benchmark / claim tags are a keyword heuristic (1 of the 11 "claims" is really advice). They affect the breakdown only, not the headline.
  4. views-eval.js is about 1,200 lines. That's fine for a developer tool; split the classifier into its own file if it grows.

Bugs this eval found

gitluis commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

Approval recorded (chat), per CLAUDE.md §0

Merging next. #12 will be retargeted to master, and #13 stays stacked on #12.


Generated by Claude Code

@gitluis
gitluis merged commit b961257 into master Oct 26, 2026
@gitluis
gitluis deleted the ai-grounding-eval branch October 26, 2026 04:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants