Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,37 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

---

## [1.10.0] — 2026-10-25

### AI Eval (new developer view, `views-eval.js`)
- **Grounding eval for AI Analysis.** Open with `showView('ai-eval')` in DevTools — not in the sidebar, like DB Inspector. It runs each of the 8 `AI_PROMPTS` 3 times (24 calls) and checks every dollar amount and percentage in the answers against the financial summary the model received. Method and results: `docs/EVAL.md`.
- **Synthetic data only.** Loads the three CSVs in `samples/` (`fetch()` when served over HTTP, a file picker on `file://`) and refuses any file whose SHA-256 differs from the committed sample. They're parsed inside a sandboxed `State` (no overrides, rules, manual balances, budgets or debts; date range All Time) that is swapped in and restored synchronously, so your own data is never read, sent or overwritten. Verified equal to the summary a normal import of `samples/` produces.
- **Same request as the AI tab.** Each call goes through `callClaude()` with `AI_MODEL` and the key saved in Settings → General. Retries network errors, timeouts, 429 and 5xx up to 3 times; stops on 400/401/403/404. Nothing is persisted and AI history is untouched.
- **Classification** of every figure:
- **Grounded** — matches a summary figure within ±$1 / ±0.5 pts, or the figure's own rounding (`$4.5K`) when the item it stands for is named nearby.
- **Derived ✓** — arithmetic on summary figures, verified in code. Covers explicit equations in the answer, same-line arithmetic on grounded figures, and ~100 standard derivations (category shares, monthly savings rates, month-over-month changes, averages, ×12, projections), each tied to its context words.
- **Math error** — an explicit equation that doesn't hold.
- **Forecast** — projections in the 12-month prediction prompt that rest on unverifiable assumptions. Treated as derived (never unsupported) per the eval spec, but reported separately and kept out of the verified share.
- **Unsupported** — everything else, tagged as advice/target, benchmark, assumption or a claim about the data.
- **Table-aware:** a table cell is checked with its column header, so a `Sep`/`Oct` column names the month. A figure that restates an earlier one must match it within its written precision.
- **Report** per prompt and overall, stability across the 3 repeats, a strict literal-tolerance variant, and a chance-match calibration (figures perturbed 3–25% and re-scored) that estimates the matcher's false-positive rate. Every unsupported figure is listed with its excerpt.
- **Export** — Download EVAL.md and a results JSON (raw answers + per-figure classifications). "Load results JSON" re-scores a saved run without new calls. It refuses files made from other samples or other `AI_PROMPTS`, and a summary loaded from JSON can be re-scored but never sent.

### AI Analysis
- **`callClaude()` raw mode.** `callClaude(question, label, { raw:true, summary })` sends the identical request but touches no DOM, busy flag or history; it resolves to `{ text, stopReason, usage, model }` or throws (HTTP errors carry `.status`/`.detail`). The UI path is unchanged and uses the same inner `send()`.

### Docs
- **`docs/EVAL.md`** — method and results of the first full run (`claude-sonnet-5-5`, 703 figures):
- 48.6% grounded and 20.2% derived (68.8% traceable), 28.4% unsupported, no math errors;
- a 75-figure hand check;
- a record of three matcher bugs fixed after the run, with before and after numbers.
Raw answers are in `docs/eval-results.json`.
- **Found by the eval, not fixed here:** 15 of 24 AI Analysis answers hit `max_tokens` and are shown cut off. The summary also gives no date span, so monthly figures rest on a guessed period length.
- **README** — new Evaluation section.
- **CLAUDE.md** — §1 version, §3 structure and load order, §12 view list, §13 raw mode, §21 AI Eval rules.

---

## [1.9.6] — 2026-10-25

### Docs
Expand Down
32 changes: 27 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Until then, this section is the rule.

FinLens is a **single-origin, zero-build, vanilla JavaScript personal finance dashboard** that runs entirely in the browser. There is no server, no npm, no bundler, no framework, and no TypeScript. The entry point is `index.html`; everything else is a plain `.js` file in the `js/` folder.

**Current version:** v1.9.6 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release).
**Current version:** v1.10.0 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release).

**What it does:** Imports CSV exports from Ally Bank (Checking + Savings) and Chase credit cards, parses and normalizes them into a unified transaction model, and renders a multi-tab financial dashboard with budgeting, debt tracking, savings forecasting, investment planning, and Claude-powered AI analysis.

Expand Down Expand Up @@ -97,11 +97,13 @@ FinLens/
│ ├── ally_checking_transactions.csv
│ ├── ally_savings_transactions.csv
│ └── chase_transactions.csv
├── samples/ ← Synthetic CSVs (fake data) for demos and manual testing — committed
├── samples/ ← Synthetic CSVs (fake data) for demos, manual testing and the AI eval — committed
│ ├── ally_checking_sample.csv
│ ├── ally_savings_sample.csv
│ └── chase_card_sample.csv
├── docs/
│ ├── EVAL.md ← AI Analysis grounding eval: method + results (exported by the AI Eval view)
│ ├── eval-results.json ← Raw answers + per-figure classifications for that run
│ └── archive/
│ ├── INDEXEDDB_PLAN.md ← Completed spec (v1.4.0); retained for reference
│ └── IMPLEMENTATION_STATUS.md ← Historical implementation notes
Expand All @@ -119,6 +121,7 @@ FinLens/
├── views-settings.js ← Settings shell (tab bar) + General tab (display, Anthropic API key)
├── views-cleanup.js ← Settings → Data & Reset: clean up imported data, DB, logs; start over (§20)
├── views-db.js ← DB Schema diagram, DB Inspector, Changelog views (developer only)
├── views-eval.js ← AI Eval: grounding eval for AI Analysis (developer only, §21)
├── views-database.js ← Unused — not loaded by index.html; retained for reference
└── init.js ← App boot IIFE — event wiring, DB.open(), DB.loadAll(), first render
```
Expand All @@ -128,7 +131,7 @@ FinLens/
```
core.js → aggregation.js → db.js → ui.js → views-main.js → views-plan.js → views-invest.js
→ views-forecast.js → views-ai.js → views-data.js
→ views-settings.js → views-cleanup.js → views-db.js → init.js
→ views-settings.js → views-cleanup.js → views-db.js → views-eval.js → init.js
```

All scripts share the **global `window` scope**. A function defined in `core.js` is callable in any `views-*.js` without import statements.
Expand Down Expand Up @@ -613,6 +616,7 @@ View registrations by file:
- `views-settings.js` — settings (internal use only, not in nav)
- `views-cleanup.js` — data-cleanup (Settings → Data & Reset tab)
- `views-db.js` — db-inspector, db-schema, changelog (not in nav; developer reference only)
- `views-eval.js` — ai-eval (not in nav; developer only — `showView('ai-eval')` from DevTools, §21)

The `data-section` attribute values are `finances`, `plan`, `insights` — the toggle JS derives section IDs as `ns-{data-section}`. Settings is a standalone `<a class="nav">` link at the bottom of the sidebar (no section wrapper, no sub-items).

Expand Down Expand Up @@ -661,7 +665,8 @@ Views build a fresh DOM on every `showView()` call. This is intentional. Do not
- `buildFinancialSummary()` assembles a plain-text snapshot of current State. It is sent inside `<financial_data>` tags, and the system prompt tells the model to treat that content as data, not instructions (account names and merchant descriptions come from CSVs and are untrusted).
- Responses are read from every `text` content block (don't assume `content[0]` is text) and rendered with `renderMarkdown()` — never `marked.parse()`.
- AI history is capped at 50 entries (`State.aiHistory`).
- If updating the Claude model string, update `AI_MODEL` in `views-ai.js` and mention it in CHANGELOG.md.
- **Raw mode:** `callClaude(question, label, { raw:true, summary })` sends the identical request through the same inner `send()` but touches no DOM, busy flag or history; it resolves to `{ text, stopReason, usage, model }` or throws (`.status`/`.detail` on HTTP errors). The AI Eval (§21) depends on it — keep one request path for both, so the eval measures what the AI tab actually sends.
- If updating the Claude model string, update `AI_MODEL` in `views-ai.js` and mention it in CHANGELOG.md. Changing the model, the system prompt or `buildFinancialSummary()` changes what the eval measures — re-run it and replace `docs/EVAL.md`.

---

Expand Down Expand Up @@ -785,7 +790,7 @@ Multiple Chase card files can be loaded simultaneously. Each gets a unique `acco

---

*Last updated: v1.9.6 — 2026-10-25 (§14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)*
*Last updated: v1.10.0 — 2026-10-25 (§21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)*

---

Expand Down Expand Up @@ -848,3 +853,20 @@ CSV files on disk are **never** modified.
- **Permission prompts need a fresh user gesture.** Destructive buttons use click-again-to-confirm (no `confirm()` dialog, which can outlast the ~5 s activation window), and actions ask for folder access before doing slow work.
- **Start over must leave nothing behind.** It sets `State._resetInProgress` so a pending debounced save can't re-create the file, and `DB.open()` registers `onversionchange` so other tabs release the database. `idbGet`/`idbSet`/`idbDel` close their connections so `finlens_fs` can be deleted.
- `clearAllData()` is kept as a wrapper for `DataCleanup.startOver('archive')`. Up to v1.8.1 it only cleared `localStorage`, which IndexedDB and the state file then restored on reload.

---

## 21. AI Eval (views-eval.js)

A developer-only view (`showView('ai-eval')`) that measures how well AI Analysis answers are grounded in the data the model was sent. Everything lives on the `AIEval` namespace object. Method and results: `docs/EVAL.md`.

### Rules

- **Samples only, never real data.** The eval reads only the three files in `samples/`, and only if their SHA-256 (after normalizing line endings) matches `AIEval.SAMPLES`. If you change a sample file on purpose, update its hash there (the error message shows the new one) and re-run the eval.
- **Sandbox, synchronously.** `_withSampleState(fn)` swaps a fresh, sample-only set of `State` keys in, runs `fn`, and restores every key in `finally`. `fn` must be synchronous — no `await` — so no render, `lsSet()` or debounced save can run while sample data is in `State`. If `buildFinancialSummary()`, `normalizeRow()`, `mkTx()` or `detectTransfers()` start reading a new `State` key, add it to the sandbox or the user's value will leak into the eval.
- **Same parsing as an import.** `buildDataset()` mirrors `parseCsvFile()` (schema detection, account ids, occurrence suffixes) without touching IndexedDB. If `parseCsvFile()` changes, mirror it here; the sandbox summary must equal what a normal import of `samples/` produces.
- **Same request as the AI tab** — `callClaude()` raw mode (§13). Never build a second request path.
- **Nothing persisted.** Results live in memory only and leave via the download buttons, so §20 needs no entry. Don't add `lsSet()` calls here. The API key is never put in results, exports or logs.
- **A summary loaded from a results JSON is never sent.** It can be re-scored locally (`loadResults()` sets `dataset.fromFile`), but Run stays disabled until samples are reloaded and verified.
- **Changing the matcher after a run:** fix only bugs against the method EVAL.md states. Record each fix in EVAL.md's "Changes made after the run" with before and after numbers. Never add a rule just because it raises the score on answers you've already seen.
- **Classification** (`classifyText()`): grounded → explicit equation (a wrong one is a math error) → standard derivations (`_candidates()`, each with context keys) → restatements, correct equations and same-line arithmetic on anchored figures → forecast (the 12-month prediction prompt only) or unsupported. When adding a derivation, give it context keys; check the chance-match rate in the report doesn't rise noticeably — it is the matcher's false-positive rate and is published alongside the headline.
29 changes: 26 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> A zero-install, browser-native finance dashboard. Drop in your bank CSVs, get instant clarity on your money.

**Version:** v1.9.6 &nbsp;·&nbsp; **Stack:** Vanilla JS, HTML, CSS &nbsp;·&nbsp; **Install:** None
**Version:** v1.10.0 &nbsp;·&nbsp; **Stack:** Vanilla JS, HTML, CSS &nbsp;·&nbsp; **Install:** None

---

Expand Down Expand Up @@ -172,6 +172,25 @@ FinLens uses `claude-sonnet-5-5`.

---

## Evaluation

How far can you trust the numbers in an AI Analysis answer? FinLens has a grounding eval: each of the 8 quick-analysis prompts is run 3 times on the synthetic data in `samples/`, and every dollar amount and percentage in the answers is checked against the financial summary the model was sent.

**Result** (`claude-sonnet-5-5`, 24 answers, 703 dollar and percentage figures):

- **68.8% traceable to the data:**
- **48.6%** matched the data the model was sent;
- **20.2%** were arithmetic on that data, verified in code.
- **No math errors.**
- **28.4% unsupported.** These are mostly recommendations and benchmarks (budgets, rules of thumb) rather than claims about the data.
- **Hand check:** of 75 random figures, every grounded and derived one checked was correct, so the matcher errs on the conservative side.

Each figure is classified as **grounded** (in the data), **derived** (arithmetic on the data, verified in code), a **math error**, an unverifiable **forecast** assumption, or **unsupported**. A chance-match calibration measures how often the matcher would accept a wrong number. **[docs/EVAL.md](docs/EVAL.md)** has the method, per-prompt results and every unsupported figure with its context.

To re-run it: save an API key in Settings, open DevTools, run `showView('ai-eval')` and click **Run eval** (24 calls, about 6 minutes). If FinLens is opened as a file, pick the three CSVs in `samples/` when asked. It only ever uses `samples/` — your own data, range and settings are untouched.

---

## Project Structure

```
Expand All @@ -183,10 +202,13 @@ FinLens/
│ ├── ally_checking_transactions.csv
│ ├── ally_savings_transactions.csv
│ └── chase_transactions.csv
├── samples/ ← Synthetic CSVs (fake data) for trying the app
├── samples/ ← Synthetic CSVs (fake data) for trying the app and the AI eval
│ ├── ally_checking_sample.csv
│ ├── ally_savings_sample.csv
│ └── chase_card_sample.csv
├── docs/
│ ├── EVAL.md ← AI Analysis grounding eval: method and results
│ └── eval-results.json ← Raw answers and per-figure classifications from that run
└── js/
├── core.js ← State, helpers, CSV parsing, bank schema registry
├── aggregation.js ← Pure data aggregation: sums, groupings, recurring detection
Expand All @@ -201,6 +223,7 @@ FinLens/
├── views-settings.js ← Settings shell + General tab (theme, API key)
├── views-cleanup.js ← Settings → Data & Reset (clean up data, start over)
├── views-db.js ← DB Inspector, DB Schema diagram, Changelog views
├── views-eval.js ← AI Eval: grounding eval for AI Analysis (developer view)
└── init.js ← App boot and event wiring
```

Expand Down Expand Up @@ -235,4 +258,4 @@ See [CHANGELOG.md](CHANGELOG.md) for the full release history.

---

*FinLens v1.9.6 — Built for one person's finances, designed to be useful for anyone's.*
*FinLens v1.10.0 — Built for one person's finances, designed to be useful for anyone's.*
Loading