Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,26 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

---

## [1.10.1] — 2026-10-25

### AI Analysis
- **Answers no longer get cut off mid-sentence.** In the v1.10.0 eval, 15 of 24 answers stopped at `max_tokens` (1,500) and the AI tab showed them as if complete.
- **Cause:** Sonnet 5.5 thinks before answering (default effort `high`), and its thinking counts toward `max_tokens` even though FinLens only shows the text. The eval's own usage data shows it: thinking took on average 747 of the 1,500 tokens in the cut-off answers, versus 458 in the ones that finished.
- **Fix:** `max_tokens` is now `AI_MAX_TOKENS` = 4,096, room for both thinking and the answer.
- **Cut-off answers are flagged.** If an answer still stops early (`stop_reason` `max_tokens` or `model_context_window_exceeded`), the tab says so under the answer. It's saved to history with `truncated: true`, so reopening it shows the warning, and copies and saved notes end with a "Cut off" line. `send()` returns `truncated` alongside `stopReason`.
- **Answers cut off before any text say so.** If Claude reaches the limit while it is still thinking, the response has no text. The tab now says no answer was written, instead of showing "(no response)" with a note that the end is missing. Nothing is saved to history or left to copy, since there is no answer (`aiNoAnswer()`, `AI_NO_TEXT`).

### AI Eval
- **Reports tokens.** EVAL.md gains a *Tokens* row: output tokens per answer, how many were thinking, and the thinking in the cut-off answers. The *Calls* row shows the run's real `max_tokens`. Runs now record `meta.maxTokens`; for older runs it is inferred from where the cut-off answers stopped.

### Docs
- **`docs/EVAL.md` — the eval re-run on v1.10.1.** **0 of 24 answers cut off** (was 15 of 24 at 1,500). The longest used 2,054 of the 4,096 tokens. Output tokens per answer rose about 10% (1,417 → 1,560), and the slowest call took 19.5 s. Grounding moved slightly: 45.7% grounded and 66.9% traceable (was 48.6% and 68.8%), about the spread between repeats. The change comes from the endings that used to be missing, mostly monthly figures over a guessed period length. A new **Runs** table compares the two runs.
- **`docs/eval-runs/v1.10.0/`** — the v1.10.0 report and raw answers, moved there unchanged except for an "Archived" banner.
- **README** — Evaluation numbers from the new run.
- **CLAUDE.md §13** — the output budget and cut-off answers; **§3** and **§21** — what each eval run records, and how to publish a new run (`docs/eval-runs/`).

---

## [1.10.0] — 2026-10-25

### AI Eval (new developer view, `views-eval.js`)
Expand Down
14 changes: 9 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Until then, this section is the rule.

FinLens is a **single-origin, zero-build, vanilla JavaScript personal finance dashboard** that runs entirely in the browser. There is no server, no npm, no bundler, no framework, and no TypeScript. The entry point is `index.html`; everything else is a plain `.js` file in the `js/` folder.

**Current version:** v1.10.0 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release).
**Current version:** v1.10.1 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release).

**What it does:** Imports CSV exports from Ally Bank (Checking + Savings) and Chase credit cards, parses and normalizes them into a unified transaction model, and renders a multi-tab financial dashboard with budgeting, debt tracking, savings forecasting, investment planning, and Claude-powered AI analysis.

Expand Down Expand Up @@ -102,8 +102,9 @@ FinLens/
│ ├── ally_savings_sample.csv
│ └── chase_card_sample.csv
├── docs/
│ ├── EVAL.md ← AI Analysis grounding eval: method + results (exported by the AI Eval view)
│ ├── EVAL.md ← AI Analysis grounding eval: method + results (exported by the AI Eval view) + hand-written Runs table
│ ├── eval-results.json ← Raw answers + per-figure classifications for that run
│ ├── eval-runs/vX.Y.Z/ ← Earlier runs: each one's EVAL.md + eval-results.json, as published
│ └── archive/
│ ├── INDEXEDDB_PLAN.md ← Completed spec (v1.4.0); retained for reference
│ └── IMPLEMENTATION_STATUS.md ← Historical implementation notes
Expand Down Expand Up @@ -665,7 +666,8 @@ Views build a fresh DOM on every `showView()` call. This is intentional. Do not
- `buildFinancialSummary()` assembles a plain-text snapshot of current State. It is sent inside `<financial_data>` tags, and the system prompt tells the model to treat that content as data, not instructions (account names and merchant descriptions come from CSVs and are untrusted).
- Responses are read from every `text` content block (don't assume `content[0]` is text) and rendered with `renderMarkdown()` — never `marked.parse()`.
- AI history is capped at 50 entries (`State.aiHistory`).
- **Raw mode:** `callClaude(question, label, { raw:true, summary })` sends the identical request through the same inner `send()` but touches no DOM, busy flag or history; it resolves to `{ text, stopReason, usage, model }` or throws (`.status`/`.detail` on HTTP errors). The AI Eval (§21) depends on it — keep one request path for both, so the eval measures what the AI tab actually sends.
- **Output budget and cut-off answers (v1.10.1).** `max_tokens` is `AI_MAX_TOKENS` (4,096). It caps the model's thinking *and* the answer combined. Sonnet 5.5 thinks by default (effort `high`), and thinking counts toward `max_tokens` even though FinLens shows only the text. At 1,500, thinking took ~750 tokens and 15 of 24 eval answers were cut off. Asking the model in the prompt to think less doesn't reliably work; raise the budget or set `output_config.effort` instead. `send()` returns `truncated` when `stop_reason` is `max_tokens` or `model_context_window_exceeded`. The AI tab then shows a warning, and the answer is saved to history with `truncated: true` and copied or saved with `AI_CUT_OFF_NOTE`. If it was cut off before any text (the text is `AI_NO_TEXT`; `aiNoAnswer()`), the tab says no answer was written and nothing goes to history. Never show a cut-off answer as if it were complete.
- **Raw mode:** `callClaude(question, label, { raw:true, summary })` sends the identical request through the same inner `send()` but touches no DOM, busy flag or history; it resolves to `{ text, stopReason, truncated, usage, model }` or throws (`.status`/`.detail` on HTTP errors). The AI Eval (§21) depends on it — keep one request path for both, so the eval measures what the AI tab actually sends.
- If updating the Claude model string, update `AI_MODEL` in `views-ai.js` and mention it in CHANGELOG.md. Changing the model, the system prompt or `buildFinancialSummary()` changes what the eval measures — re-run it and replace `docs/EVAL.md`.

---
Expand Down Expand Up @@ -790,7 +792,7 @@ Multiple Chase card files can be loaded simultaneously. Each gets a unique `acco

---

*Last updated: v1.10.0 — 2026-10-25 (§21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)*
*Last updated: v1.10.1 — 2026-10-25 (§13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)*

---

Expand Down Expand Up @@ -858,15 +860,17 @@ CSV files on disk are **never** modified.

## 21. AI Eval (views-eval.js)

A developer-only view (`showView('ai-eval')`) that measures how well AI Analysis answers are grounded in the data the model was sent. Everything lives on the `AIEval` namespace object. Method and results: `docs/EVAL.md`.
A developer-only view (`showView('ai-eval')`) that measures how well AI Analysis answers are grounded in the data the model was sent. Everything lives on the `AIEval` namespace object. Method and results: `docs/EVAL.md`; earlier runs: `docs/eval-runs/`.

### Rules

- **Samples only, never real data.** The eval reads only the three files in `samples/`, and only if their SHA-256 (after normalizing line endings) matches `AIEval.SAMPLES`. If you change a sample file on purpose, update its hash there (the error message shows the new one) and re-run the eval.
- **Sandbox, synchronously.** `_withSampleState(fn)` swaps a fresh, sample-only set of `State` keys in, runs `fn`, and restores every key in `finally`. `fn` must be synchronous — no `await` — so no render, `lsSet()` or debounced save can run while sample data is in `State`. If `buildFinancialSummary()`, `normalizeRow()`, `mkTx()` or `detectTransfers()` start reading a new `State` key, add it to the sandbox or the user's value will leak into the eval.
- **Same parsing as an import.** `buildDataset()` mirrors `parseCsvFile()` (schema detection, account ids, occurrence suffixes) without touching IndexedDB. If `parseCsvFile()` changes, mirror it here; the sandbox summary must equal what a normal import of `samples/` produces.
- **Same request as the AI tab** — `callClaude()` raw mode (§13). Never build a second request path.
- **Each run records** `meta.maxTokens` and every call's `usage` (including `output_tokens_details.thinking_tokens`), and EVAL.md reports cut-off answers and thinking per answer. Runs from before v1.10.1 didn't record `maxTokens`, so it is inferred from the size the cut-off answers stopped at.
- **Nothing persisted.** Results live in memory only and leave via the download buttons, so §20 needs no entry. Don't add `lsSet()` calls here. The API key is never put in results, exports or logs.
- **A summary loaded from a results JSON is never sent.** It can be re-scored locally (`loadResults()` sets `dataset.fromFile`), but Run stays disabled until samples are reloaded and verified.
- **Publishing a new run:** move the current `docs/EVAL.md` and `docs/eval-results.json` to `docs/eval-runs/v<the old run's version>/` unchanged except for a one-line "Archived" banner (and its JSON link label). Put the new results JSON in `docs/`, generate EVAL.md by loading it in the view (**Load results JSON** → **Download EVAL.md**), then add the hand-written sections before *Method*: the **Runs** table (one column per run: `max_tokens`, cut-off answers, tokens, figures, grounded, derived, traceable, unsupported), what changed between runs, and *Changes made after the run*. Keep README's Evaluation numbers in step.
- **Changing the matcher after a run:** fix only bugs against the method EVAL.md states. Record each fix in EVAL.md's "Changes made after the run" with before and after numbers. Never add a rule just because it raises the score on answers you've already seen.
- **Classification** (`classifyText()`): grounded → explicit equation (a wrong one is a math error) → standard derivations (`_candidates()`, each with context keys) → restatements, correct equations and same-line arithmetic on anchored figures → forecast (the 12-month prediction prompt only) or unsupported. When adding a derivation, give it context keys; check the chance-match rate in the report doesn't rise noticeably — it is the matcher's false-positive rate and is published alongside the headline.
22 changes: 12 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> A zero-install, browser-native finance dashboard. Drop in your bank CSVs, get instant clarity on your money.

**Version:** v1.10.0 &nbsp;·&nbsp; **Stack:** Vanilla JS, HTML, CSS &nbsp;·&nbsp; **Install:** None
**Version:** v1.10.1 &nbsp;·&nbsp; **Stack:** Vanilla JS, HTML, CSS &nbsp;·&nbsp; **Install:** None

---

Expand Down Expand Up @@ -176,14 +176,15 @@ FinLens uses `claude-sonnet-5-5`.

How far can you trust the numbers in an AI Analysis answer? FinLens has a grounding eval: each of the 8 quick-analysis prompts is run 3 times on the synthetic data in `samples/`, and every dollar amount and percentage in the answers is checked against the financial summary the model was sent.

**Result** (`claude-sonnet-5-5`, 24 answers, 703 dollar and percentage figures):
**Result** (`claude-sonnet-5-5`, v1.10.1, 24 answers, 761 dollar and percentage figures):

- **68.8% traceable to the data:**
- **48.6%** matched the data the model was sent;
- **20.2%** were arithmetic on that data, verified in code.
- **66.9% traceable to the data:**
- **45.7%** matched the data the model was sent;
- **21.2%** were arithmetic on that data, verified in code.
- **No math errors.**
- **28.4% unsupported.** These are mostly recommendations and benchmarks (budgets, rules of thumb) rather than claims about the data.
- **Hand check:** of 75 random figures, every grounded and derived one checked was correct, so the matcher errs on the conservative side.
- **29.2% unsupported.** These are mostly recommendations and benchmarks (budgets, rules of thumb). Most of the rest are monthly amounts the model worked out by guessing how many months the data covers.
- **No answers cut off.** The v1.10.0 run cut off 15 of 24 at a 1,500-token limit; [docs/EVAL.md](docs/EVAL.md) compares the two runs.
- **Hand check** (v1.10.0 run, same matcher): of 75 random figures, every grounded and derived one checked was correct, so the matcher errs on the conservative side.

Each figure is classified as **grounded** (in the data), **derived** (arithmetic on the data, verified in code), a **math error**, an unverifiable **forecast** assumption, or **unsupported**. A chance-match calibration measures how often the matcher would accept a wrong number. **[docs/EVAL.md](docs/EVAL.md)** has the method, per-prompt results and every unsupported figure with its context.

Expand All @@ -207,8 +208,9 @@ FinLens/
│ ├── ally_savings_sample.csv
│ └── chase_card_sample.csv
├── docs/
│ ├── EVAL.md ← AI Analysis grounding eval: method and results
│ └── eval-results.json ← Raw answers and per-figure classifications from that run
│ ├── EVAL.md ← AI Analysis grounding eval: method, results, and a table comparing runs
│ ├── eval-results.json ← Raw answers and per-figure classifications from that run
│ └── eval-runs/ ← Earlier runs, one folder per version (EVAL.md + eval-results.json)
└── js/
├── core.js ← State, helpers, CSV parsing, bank schema registry
├── aggregation.js ← Pure data aggregation: sums, groupings, recurring detection
Expand Down Expand Up @@ -258,4 +260,4 @@ See [CHANGELOG.md](CHANGELOG.md) for the full release history.

---

*FinLens v1.10.0 — Built for one person's finances, designed to be useful for anyone's.*
*FinLens v1.10.1 — Built for one person's finances, designed to be useful for anyone's.*
Loading