Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,30 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

---

## [1.10.2] — 2026-10-25

### AI Analysis
- **The summary sent to Claude now states its dates.** Before, it said only "Period: All time".
- **What went wrong:** in the v1.10.0 eval the model had to guess how many months the totals covered. It guessed 6, 6–7, 6.5 and even 9 months, from rent ($8,100 ÷ $1,350) or income ($32,525 ÷ ~$4,900, which overcounts because biweekly pay puts three paychecks in some months). So the same data gave different monthly figures: groceries came out at ~$470/month in one answer and ~$516 in another.
- **Fix:** `summarySpan()` adds the span to the period line: "Period: All time (transactions from May 1, 2026 to Oct 20, 2026: 173 days, about 5.7 months)". Spans under 45 days give days only.
- **Partial months are marked** in the monthly trend: "2026-10 (through Oct 20)" (`partialMonthNote()`). The latest month no longer looks like a complete month.

### AI Eval
- **Reads the new summary format**: the period line and partial-month notes.
- **Checks per-month figures over the period**: each category, income, spending and net, ÷ the months as stated (5.7) or ÷ days/30.44. These were added before any run with the new summary (§21). Dividing by a different month count stays unsupported, since the summary now says how long the period is. Re-scoring the v1.10.0 run is unchanged; its summary has no dates.

### Docs
- **`docs/EVAL.md` — the eval re-run on v1.10.2.**
- **Monthly figures agree:** every answer gives groceries as $540–545 a month ($3,098 over 5.7 months). In v1.10.1 they said ~$470, ~$515 and ~$516, and no answer guesses the period length any more.
- **Traceable 72.6%** (was 66.9%) and **unsupported 23.3%** (was 29.2%), still with no answers cut off. The chance-match rate is 13.4% (was 11.9%).
- **October is treated as partial:** answers compare daily rates and project how it will finish.
- **The Runs table** now has a column for each of the three runs.
- **`docs/eval-runs/v1.10.1/`** — the v1.10.1 report and raw answers, archived.
- **README** — Evaluation numbers from the new run.
- **CLAUDE.md §13** — the summary always states its dates; **§21** — keep `parseSummary()` in step with the summary format, and which links to fix when archiving a run.

---

## [1.10.1] — 2026-10-25

### AI Analysis
Expand Down
14 changes: 10 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Until then, this section is the rule.

FinLens is a **single-origin, zero-build, vanilla JavaScript personal finance dashboard** that runs entirely in the browser. There is no server, no npm, no bundler, no framework, and no TypeScript. The entry point is `index.html`; everything else is a plain `.js` file in the `js/` folder.

**Current version:** v1.10.1 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release).
**Current version:** v1.10.2 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release).

**What it does:** Imports CSV exports from Ally Bank (Checking + Savings) and Chase credit cards, parses and normalizes them into a unified transaction model, and renders a multi-tab financial dashboard with budgeting, debt tracking, savings forecasting, investment planning, and Claude-powered AI analysis.

Expand Down Expand Up @@ -663,7 +663,12 @@ Views build a fresh DOM on every `showView()` call. This is intentional. Do not
- Uses the model in `AI_MODEL` (currently `claude-sonnet-5-5`) via direct browser fetch to `https://api.anthropic.com/v1/messages`. `claude-sonnet-4-20250514` was retired 2026-06-15 — check Anthropic's model deprecations page before changing models.
- Requires `anthropic-dangerous-direct-browser-access: true` header (CORS bypass).
- API key is stored in `State.apiKey` (localStorage + IndexedDB `app_state` only — **never** in `finlens-state.json`, see §4.5). It is entered in **Settings → General → Anthropic API Key**; the settings card never renders the stored value back into the DOM. **Never log or expose it.**
- `buildFinancialSummary()` assembles a plain-text snapshot of current State. It is sent inside `<financial_data>` tags, and the system prompt tells the model to treat that content as data, not instructions (account names and merchant descriptions come from CSVs and are untrusted).
- `buildFinancialSummary()` assembles a plain-text snapshot of current State. **It always states its dates (v1.10.2):**
- **The period line** gives the span of the transactions it covers (`summarySpan()`), e.g. "Period: All time (transactions from May 1, 2026 to Oct 20, 2026: 173 days, about 5.7 months)".
- **A partial month** in the monthly trend is marked (`partialMonthNote()`), e.g. "2026-10 (through Oct 20)".

Without them the model guessed the period length (6, 6.5 or even 9 months) to work out monthly figures, so the same data gave different answers. Never send totals without the dates they cover.
- The summary is sent inside `<financial_data>` tags, and the system prompt tells the model to treat that content as data, not instructions (account names and merchant descriptions come from CSVs and are untrusted).
- Responses are read from every `text` content block (don't assume `content[0]` is text) and rendered with `renderMarkdown()` — never `marked.parse()`.
- AI history is capped at 50 entries (`State.aiHistory`).
- **Output budget and cut-off answers (v1.10.1).** `max_tokens` is `AI_MAX_TOKENS` (4,096). It caps the model's thinking *and* the answer combined. Sonnet 5.5 thinks by default (effort `high`), and thinking counts toward `max_tokens` even though FinLens shows only the text. At 1,500, thinking took ~750 tokens and 15 of 24 eval answers were cut off. Asking the model in the prompt to think less doesn't reliably work; raise the budget or set `output_config.effort` instead. `send()` returns `truncated` when `stop_reason` is `max_tokens` or `model_context_window_exceeded`. The AI tab then shows a warning, and the answer is saved to history with `truncated: true` and copied or saved with `AI_CUT_OFF_NOTE`. If it was cut off before any text (the text is `AI_NO_TEXT`; `aiNoAnswer()`), the tab says no answer was written and nothing goes to history. Never show a cut-off answer as if it were complete.
Expand Down Expand Up @@ -792,7 +797,7 @@ Multiple Chase card files can be loaded simultaneously. Each gets a unique `acco

---

*Last updated: v1.10.1 — 2026-10-25 (§13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)*
*Last updated: v1.10.2 — 2026-10-25 (§13 summary states its dates; §21 per-month derivations; v1.10.1: §13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)*

---

Expand Down Expand Up @@ -871,6 +876,7 @@ A developer-only view (`showView('ai-eval')`) that measures how well AI Analysis
- **Each run records** `meta.maxTokens` and every call's `usage` (including `output_tokens_details.thinking_tokens`), and EVAL.md reports cut-off answers and thinking per answer. Runs from before v1.10.1 didn't record `maxTokens`, so it is inferred from the size the cut-off answers stopped at.
- **Nothing persisted.** Results live in memory only and leave via the download buttons, so §20 needs no entry. Don't add `lsSet()` calls here. The API key is never put in results, exports or logs.
- **A summary loaded from a results JSON is never sent.** It can be re-scored locally (`loadResults()` sets `dataset.fromFile`), but Run stays disabled until samples are reloaded and verified.
- **Publishing a new run:** move the current `docs/EVAL.md` and `docs/eval-results.json` to `docs/eval-runs/v<the old run's version>/` unchanged except for a one-line "Archived" banner (and its JSON link label). Put the new results JSON in `docs/`, generate EVAL.md by loading it in the view (**Load results JSON** → **Download EVAL.md**), then add the hand-written sections before *Method*: the **Runs** table (one column per run: `max_tokens`, cut-off answers, tokens, figures, grounded, derived, traceable, unsupported), what changed between runs, and *Changes made after the run*. Keep README's Evaluation numbers in step.
- **The summary format is parsed** by `parseSummary()`: the period line (`S.days`, `S.periodMonths`) and month lines with an optional `(through …)` / `(from …)` note. If `buildFinancialSummary()` changes, update the parser and its derivations *before* the next run. Example: the per-month figures over the period (total ÷ months as stated, or ÷ days/30.44) were added in v1.10.2 together with the dates.
- **Publishing a new run:** move the current `docs/EVAL.md` and `docs/eval-results.json` to `docs/eval-runs/v<the old run's version>/` unchanged except for a one-line "Archived" banner, its JSON link label, and relative links the move would break (`eval-runs/vX/…` becomes `../vX/…`). Put the new results JSON in `docs/`, generate EVAL.md by loading it in the view (**Load results JSON** → **Download EVAL.md**), then add the hand-written sections before *Method*: the **Runs** table (one column per run: `max_tokens`, cut-off answers, tokens, figures, grounded, derived, traceable, unsupported), what changed between runs, and *Changes made after the run*. Keep README's Evaluation numbers in step.
- **Changing the matcher after a run:** fix only bugs against the method EVAL.md states. Record each fix in EVAL.md's "Changes made after the run" with before and after numbers. Never add a rule just because it raises the score on answers you've already seen.
- **Classification** (`classifyText()`): grounded → explicit equation (a wrong one is a math error) → standard derivations (`_candidates()`, each with context keys) → restatements, correct equations and same-line arithmetic on anchored figures → forecast (the 12-month prediction prompt only) or unsupported. When adding a derivation, give it context keys; check the chance-match rate in the report doesn't rise noticeably — it is the matcher's false-positive rate and is published alongside the headline.
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> A zero-install, browser-native finance dashboard. Drop in your bank CSVs, get instant clarity on your money.

**Version:** v1.10.1 &nbsp;·&nbsp; **Stack:** Vanilla JS, HTML, CSS &nbsp;·&nbsp; **Install:** None
**Version:** v1.10.2 &nbsp;·&nbsp; **Stack:** Vanilla JS, HTML, CSS &nbsp;·&nbsp; **Install:** None

---

Expand Down Expand Up @@ -176,15 +176,15 @@ FinLens uses `claude-sonnet-5-5`.

How far can you trust the numbers in an AI Analysis answer? FinLens has a grounding eval: each of the 8 quick-analysis prompts is run 3 times on the synthetic data in `samples/`, and every dollar amount and percentage in the answers is checked against the financial summary the model was sent.

**Result** (`claude-sonnet-5-5`, v1.10.1, 24 answers, 761 dollar and percentage figures):
**Result** (`claude-sonnet-5-5`, v1.10.2, 24 answers, 855 dollar and percentage figures):

- **66.9% traceable to the data:**
- **45.7%** matched the data the model was sent;
- **21.2%** were arithmetic on that data, verified in code.
- **72.6% traceable to the data:**
- **43.5%** matched the data the model was sent;
- **29.1%** were arithmetic on that data, verified in code.
- **No math errors.**
- **29.2% unsupported.** These are mostly recommendations and benchmarks (budgets, rules of thumb). Most of the rest are monthly amounts the model worked out by guessing how many months the data covers.
- **No answers cut off.** The v1.10.0 run cut off 15 of 24 at a 1,500-token limit; [docs/EVAL.md](docs/EVAL.md) compares the two runs.
- **Hand check** (v1.10.0 run, same matcher): of 75 random figures, every grounded and derived one checked was correct, so the matcher errs on the conservative side.
- **23.3% unsupported.** These are mostly recommendations and benchmarks (budgets, rules of thumb), plus correct arithmetic the matcher can't verify, such as daily spending rates.
- **Complete, consistent answers.** None was cut off. Every answer gives the same monthly figures, because the summary states the dates it covers. [docs/EVAL.md](docs/EVAL.md) compares all three runs.
- **Hand check** (v1.10.0 run): of 75 random figures, every grounded and derived one checked was correct, so the matcher errs on the conservative side.

Each figure is classified as **grounded** (in the data), **derived** (arithmetic on the data, verified in code), a **math error**, an unverifiable **forecast** assumption, or **unsupported**. A chance-match calibration measures how often the matcher would accept a wrong number. **[docs/EVAL.md](docs/EVAL.md)** has the method, per-prompt results and every unsupported figure with its context.

Expand Down Expand Up @@ -260,4 +260,4 @@ See [CHANGELOG.md](CHANGELOG.md) for the full release history.

---

*FinLens v1.10.1 — Built for one person's finances, designed to be useful for anyone's.*
*FinLens v1.10.2 — Built for one person's finances, designed to be useful for anyone's.*
Loading