Skip to content

v1.10.2: AI summary states its dates; stop the model guessing the period length - #13

Merged
gitluis merged 5 commits into
masterfrom
fix-ai-summary-date-range
Oct 26, 2026
Merged

gitluis merged 5 commits into
masterfrom
fix-ai-summary-date-range

Conversation

@gitluis

@gitluis gitluis commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Base: master. #12 merged on 2026-10-26 and GitHub retargeted this PR. Ready for review: the eval re-run is in (0163c12). Merge order: this (v1.10.2) → #14 (v1.10.3) → #15 (v1.10.4) → #10 (v1.11.0), per Luis on 2026-10-26.

Summary

The summary FinLens sends to Claude now states the dates it covers, and marks a partial month. The model no longer has to guess how long "All time" is.

Cause

The summary said only Period: All time. In the v1.10.0 eval (#6), every answer that gave a monthly figure had to guess how many months the totals covered, and the guesses varied:

  • "I'm assuming the data covers ~6 months, based on housing of $8,100 ÷ $1,350."
  • "About 6–7 months at ~$4,900/month income." That's $32,525 ÷ $4,900 = 6.6. It overcounts because biweekly pay puts three paychecks in some months; May had three in the sample data.
  • "Roughly 6.5 months … I divided category totals by 6.5."
  • One run guessed 9 months and asked you to confirm the time span.

So the same data gave different numbers: groceries were ~$470/month in one answer and ~$516 in another.

Related: the monthly trend listed October as if it were complete, but the data stops on Oct 20.

What changed

  • js/views-ai.js
    • summarySpan(): the period line now reads Period: All time (transactions from May 1, 2026 to Oct 20, 2026: 173 days, about 5.7 months). It works for every date range, and spans under 45 days give days only.
    • partialMonthNote(): a month the data only partly covers is marked, e.g. 2026-10 (through Oct 20) or (from May 15).
  • js/views-eval.js
    • Parses the new period line and month notes.
    • New check, added before any run with the new summary (§21): per-month figures over the period, meaning each category, income, spending and net, ÷ the stated 5.7 months or ÷ days/30.44. Dividing by any other month count stays unsupported, because the summary now says how long the period is.
    • Re-scoring the v1.10.0 run is unchanged, since its summary has no dates.
  • docs/EVAL.md — the eval re-run on this branch (below), with the Runs table extended to all three runs. The v1.10.1 report moved to docs/eval-runs/v1.10.1/. README's Evaluation numbers are updated.
  • v1.10.2 (patch release; v1.17.0: Settings → Database tab (DB Inspector, DB Schema), rebuilt on the Clarity redesign #10 keeps v1.11.0) in CHANGELOG, index.html, README and views-settings.js. CLAUDE.md §13 now says "never send totals without their dates"; §21 says to keep the parser in step with the summary format.

Summary diff for the sample data (exactly two lines):

-Period: All time
+Period: All time (transactions from May 1, 2026 to Oct 20, 2026: 173 days, about 5.7 months)
-  - 2026-10: income $4,900.00, expense $2,500.29, net $2,399.71
+  - 2026-10 (through Oct 20): income $4,900.00, expense $2,500.29, net $2,399.71

How it was tested

Headless Chromium, mocked API, on this branch (#6 + #12 + this change):

  • Summary: a normal import of samples/ produces the new lines. The eval's sandbox summary still equals the real import path byte for byte.

  • Span and partial months: checked on synthetic data, including data starting May 15, a 20-day range, a single day and no data.

  • Parser: reads days = 173, months = 5.7, and October's "(through Oct 20)". The 30 summary facts are unchanged.

  • New checks:

    • "$543 a month" and "~$545/month" on groceries count as derived;
    • "$516/month" (÷ 6) stays unsupported;
    • spending $2,946/month, income $5,706/month and shopping $325/month count as derived.
  • Regression:

    All pass.

Re-tested on 0163c12: 30/30 headless checks pass, including the dated summary, every version string saying v1.10.2, and re-scoring all three committed runs. The committed EVAL.md is exactly what the view generates from docs/eval-results.json, plus the hand-written sections. Re-scoring the v1.10.1 answers with this PR's matcher gives exactly the v1.10.1 numbers: the new per-month rule applies only when the summary states its period.

Eval re-run

Run by Luis on 2026-10-26, 12:44–12:50 UTC, on this branch (v1.10.2). Same model, prompts, samples and max_tokens as #12's run; the summary differs by exactly the two lines above. No errors or retries.

v1.10.1 (#12) v1.10.2 (this PR)
Answers cut off 0 of 24 0 of 24
Monthly groceries (same $3,098 total) ~$470, ~$515, ~$516 $540–545 in every answer
Output tokens per answer (thinking) 1,560 (669) 1,790 (858)
Slowest call 19.5 s 20.0 s
Traceable (grounded / derived) 66.9% (45.7 / 21.2) 72.6% (43.5 / 29.1)
Unsupported 29.2% 23.3%
Chance-match rate 11.9% 13.4%
Unsupported "claims about the data" 46 56
  • Monthly figures agree, and no answer guesses the period length any more. v1.10.1: fix AI answers cut off mid-sentence; flag cut-off answers #12's run assumed 6, 6–7 or 6.6 months; these answers cite 5.7 months.
  • Unsupported fell most where monthly figures matter: from 52.0% to 34.1% in Where am I overspending?, and from 69.6% to 39.7% in Realistic budget suggestion.
  • October is treated as partial. This month vs last month now compares daily rates and projects how October will finish. That's correct arithmetic or projection the matcher doesn't model, so the prompt's unsupported share rose from 4.0% to 22.6%.
  • "Claims about the data" rose from 46 to 56, but they are different figures. I read all 56; none divides by a guessed month count:
    • about 32 are correct arithmetic the matcher doesn't model (May–Jul income average, daily rates, shares of monthly income, rent over 6 payments);
    • 9 are projections;
    • 15 are advice or benchmarks the keyword tagger labeled as claims.
  • The chance-match rate rose 1.5 points. The new per-month checks give the matcher more values to match by chance. That rise is much smaller than the 5.7-point gain in traceable figures.
  • Your own data: in Luis's screenshot of the AI tab on Last 3 Months, the answer said its monthly figures were the 87-day totals divided by 2.9.

Full results: docs/EVAL.md.

Please check before merging

  • The re-run in EVAL.md shows monthly figures consistent across repeats. Note that unsupported "claims about the data" went from 46 to 56, not down; see above and EVAL.md for what they are.
  • The new period line reads well for your own data in the AI tab, for example with "This Month" or a custom range.

Opened in https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq; re-versioned in the session below.


https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR


Generated by Claude Code

The summary said only "Period: All time", so in the v1.10.0 eval the
model guessed how many months the totals covered (6, 6-7, 6.5, 9) and
gave different monthly figures for the same data (groceries ~$470 vs
~$516/month).

- summarySpan(): "Period: All time (transactions from May 1, 2026 to
  Oct 20, 2026: 173 days, about 5.7 months)"
- partialMonthNote(): "2026-10 (through Oct 20)" in the monthly trend
- AI Eval parses the new format and checks per-month figures over the
  period (total / stated months or / days/30.44), added before any run
  with the new summary; v1.10.0 re-scores unchanged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
…on to v1.10.2

#12 and #13 now merge before #10 (Luis, 2026-10-26), so #12 is v1.10.1
and this fix is v1.10.2; #10 keeps v1.11.0. Conflicts were only the
version strings. Every v1.11.2 is now v1.10.2 and every v1.11.1 is
v1.10.1 (CHANGELOG, index.html, README, views-settings.js, CLAUDE.md,
and comments in views-ai.js and views-eval.js). No code change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
gitluis pushed a commit that referenced this pull request Oct 26, 2026
Luis decided on 2026-10-26 that #12 and #13 merge before #10. Master is
at v1.10.0, so this bug fix is the patch release v1.10.1 (CLAUDE.md
§17); #10 keeps v1.11.0. Renumbered in CHANGELOG, index.html, README,
views-settings.js, CLAUDE.md §1/§13/§21/last-updated, and the
views-eval.js comment on runs that didn't record maxTokens. No code
change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
@gitluis gitluis changed the title v1.11.2: AI summary states its dates; stop the model guessing the period length v1.10.2: AI summary states its dates; stop the model guessing the period length Oct 26, 2026
gitluis pushed a commit that referenced this pull request Oct 26, 2026
Base automatically changed from fix-ai-truncated-answers to master October 26, 2026 12:15
…ve v1.10.1

Luis ran the AI Eval on this branch (v1.10.2) on 2026-10-26. Same
model, prompts, samples and max_tokens as the v1.10.1 run; the summary
now states its dates (exactly the two-line difference this PR makes).

- Every monthly groceries figure is $540-545 ($3,098 / 5.7 months);
  v1.10.1 said ~$470, ~$515, ~$516. No answer guesses the period.
- Traceable 66.9% -> 72.6%, unsupported 29.2% -> 23.3%, 0 cut off;
  chance-match rate 11.9% -> 13.4% (new per-month derivations)
- Unsupported "claims about the data" 46 -> 56, read by hand: none
  divides by a guessed month count; ~32 correct arithmetic the matcher
  doesn't model, 9 projections (October is now partial), 15 mis-tagged
  advice or benchmarks
- docs/EVAL.md generated from the new results JSON plus a three-run
  Runs table; v1.10.1 report archived in docs/eval-runs/v1.10.1/
  (relative links fixed); README numbers, CLAUDE.md §21, CHANGELOG 1.10.2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
gitluis pushed a commit that referenced this pull request Oct 26, 2026
@gitluis
gitluis marked this pull request as ready for review October 26, 2026 12:59

gitluis commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

Approved by Luis in chat on 2026-10-26. His words:

#13 is approved. Merge the PR.

Recorded here before merging, per CLAUDE.md §0. Merging next as a merge commit.


🤖 Generated with Claude Code


Generated by Claude Code

@gitluis
gitluis merged commit 17fc0c2 into master Oct 26, 2026
@gitluis
gitluis deleted the fix-ai-summary-date-range branch October 26, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants