From 9b062a7a3076db2a2af643e86c1248d300b911d5 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 26 Oct 2026 11:48:23 +0000 Subject: [PATCH] Fix declined requests shown as an answer; say Claude declined (v1.10.3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the API's safety classifiers decline a request, the response is a normal HTTP 200 with stop_reason "refusal" and no text. The AI tab showed "(no response)" as the answer and saved it to history; the eval counted it as an answered run. The bug was already on master. - send() returns refusal ({ category, explanation } from stop_details, or null) and drops any partial text, as the API docs advise - the AI tab shows aiRefusalHtml(): Claude declined, plus the escaped explanation when there is one; nothing goes to history or the copy buffer - AI Eval counts declined runs in the Calls row, leaves them out of scoring and token averages, and stores each run's refusal - v1.10.3; CLAUDE.md §13 and §21 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR --- CHANGELOG.md | 14 ++++++++++++++ CLAUDE.md | 8 +++++--- README.md | 4 ++-- index.html | 2 +- js/views-ai.js | 19 ++++++++++++++++--- js/views-eval.js | 14 +++++++++----- js/views-settings.js | 2 +- 7 files changed, 48 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e326b3..c785d6f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,20 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). --- +## [1.10.3] — 2026-10-26 + +### AI Analysis +- **Declined questions say so.** When the API's safety classifiers decline a request, it comes back as a normal HTTP 200 with `stop_reason` `refusal` and no answer. The AI tab showed "(no response)" as if that were the answer and saved it to history. It now says Claude declined, shows the reason from `stop_details.explanation` when there is one, and keeps nothing in history or for copying (`aiRefusalHtml()`). Any partial text is discarded, as the API docs advise. The bug was already on `master`. +- **`send()` returns `refusal`**: `{ category, explanation }` for a declined request, otherwise `null`. + +### AI Eval +- **Declined answers are counted, not scored.** The *Calls* row reports them as declined, like failed calls, and they're left out of the figures and the token averages. Each run stores its `refusal`. The v1.10.0 run had none, so it re-scores unchanged. + +### Docs +- **CLAUDE.md §13** — refusals and the raw-mode result; **§21** — declined runs. + +--- + ## [1.10.2] — 2026-10-25 ### AI Analysis diff --git a/CLAUDE.md b/CLAUDE.md index 052b8ec..ddc59df 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -59,7 +59,7 @@ Until then, this section is the rule. FinLens is a **single-origin, zero-build, vanilla JavaScript personal finance dashboard** that runs entirely in the browser. There is no server, no npm, no bundler, no framework, and no TypeScript. The entry point is `index.html`; everything else is a plain `.js` file in the `js/` folder. -**Current version:** v1.10.2 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release). +**Current version:** v1.10.3 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release). **What it does:** Imports CSV exports from Ally Bank (Checking + Savings) and Chase credit cards, parses and normalizes them into a unified transaction model, and renders a multi-tab financial dashboard with budgeting, debt tracking, savings forecasting, investment planning, and Claude-powered AI analysis. @@ -671,7 +671,8 @@ Views build a fresh DOM on every `showView()` call. This is intentional. Do not - Responses are read from every `text` content block (don't assume `content[0]` is text) and rendered with `renderMarkdown()` — never `marked.parse()`. - AI history is capped at 50 entries (`State.aiHistory`). - **Output budget and cut-off answers (v1.10.1).** `max_tokens` is `AI_MAX_TOKENS` (4,096). It caps the model's thinking *and* the answer combined. Sonnet 5.5 thinks by default (effort `high`), and thinking counts toward `max_tokens` even though FinLens shows only the text. At 1,500, thinking took ~750 tokens and 15 of 24 eval answers were cut off. Asking the model in the prompt to think less doesn't reliably work; raise the budget or set `output_config.effort` instead. `send()` returns `truncated` when `stop_reason` is `max_tokens` or `model_context_window_exceeded`. The AI tab then shows a warning, and the answer is saved to history with `truncated: true` and copied or saved with `AI_CUT_OFF_NOTE`. If it was cut off before any text (the text is `AI_NO_TEXT`; `aiNoAnswer()`), the tab says no answer was written and nothing goes to history. Never show a cut-off answer as if it were complete. -- **Raw mode:** `callClaude(question, label, { raw:true, summary })` sends the identical request through the same inner `send()` but touches no DOM, busy flag or history; it resolves to `{ text, stopReason, truncated, usage, model }` or throws (`.status`/`.detail` on HTTP errors). The AI Eval (§21) depends on it — keep one request path for both, so the eval measures what the AI tab actually sends. +- **Refusals (v1.10.3).** The API's safety classifiers can decline a request. That arrives as HTTP 200 with `stop_reason: "refusal"`, not as an error, so check `stop_reason`, not the content. `send()` then returns `refusal` (`{ category, explanation }` from `stop_details`) and drops any partial text. The AI tab shows `aiRefusalHtml()` — Claude declined, plus `explanation` when present (display it, never parse it) — and nothing goes to history. Never show a declined request as an answer. +- **Raw mode:** `callClaude(question, label, { raw:true, summary })` sends the identical request through the same inner `send()` but touches no DOM, busy flag or history; it resolves to `{ text, stopReason, truncated, refusal, usage, model }` or throws (`.status`/`.detail` on HTTP errors). The AI Eval (§21) depends on it — keep one request path for both, so the eval measures what the AI tab actually sends. - If updating the Claude model string, update `AI_MODEL` in `views-ai.js` and mention it in CHANGELOG.md. Changing the model, the system prompt or `buildFinancialSummary()` changes what the eval measures — re-run it and replace `docs/EVAL.md`. --- @@ -796,7 +797,7 @@ Multiple Chase card files can be loaded simultaneously. Each gets a unique `acco --- -*Last updated: v1.10.2 — 2026-10-25 (§13 summary states its dates; §21 per-month derivations; v1.10.1: §13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)* +*Last updated: v1.10.3 — 2026-10-26 (§13 refusals; §21 declined runs; v1.10.2: §13 summary states its dates; §21 per-month derivations; v1.10.1: §13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)* --- @@ -873,6 +874,7 @@ A developer-only view (`showView('ai-eval')`) that measures how well AI Analysis - **Same parsing as an import.** `buildDataset()` mirrors `parseCsvFile()` (schema detection, account ids, occurrence suffixes) without touching IndexedDB. If `parseCsvFile()` changes, mirror it here; the sandbox summary must equal what a normal import of `samples/` produces. - **Same request as the AI tab** — `callClaude()` raw mode (§13). Never build a second request path. - **Each run records** `meta.maxTokens` and every call's `usage` (including `output_tokens_details.thinking_tokens`), and EVAL.md reports cut-off answers and thinking per answer. Runs from before v1.10.1 didn't record `maxTokens`, so it is inferred from the size the cut-off answers stopped at. +- **Declined runs** (`stop_reason` `refusal`) have no answer: they are counted in the *Calls* row, like failed calls, and left out of the scoring and token averages. - **Nothing persisted.** Results live in memory only and leave via the download buttons, so §20 needs no entry. Don't add `lsSet()` calls here. The API key is never put in results, exports or logs. - **A summary loaded from a results JSON is never sent.** It can be re-scored locally (`loadResults()` sets `dataset.fromFile`), but Run stays disabled until samples are reloaded and verified. - **The summary format is parsed** by `parseSummary()`: the period line (`S.days`, `S.periodMonths`) and month lines with an optional `(through …)` / `(from …)` note. If `buildFinancialSummary()` changes, update the parser and its derivations *before* the next run. Example: the per-month figures over the period (total ÷ months as stated, or ÷ days/30.44) were added in v1.10.2 together with the dates. diff --git a/README.md b/README.md index b072f51..fef3384 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ > A zero-install, browser-native finance dashboard. Drop in your bank CSVs, get instant clarity on your money. -**Version:** v1.10.2  ·  **Stack:** Vanilla JS, HTML, CSS  ·  **Install:** None +**Version:** v1.10.3  ·  **Stack:** Vanilla JS, HTML, CSS  ·  **Install:** None --- @@ -258,4 +258,4 @@ See [CHANGELOG.md](CHANGELOG.md) for the full release history. --- -*FinLens v1.10.2 — Built for one person's finances, designed to be useful for anyone's.* +*FinLens v1.10.3 — Built for one person's finances, designed to be useful for anyone's.* diff --git a/index.html b/index.html index 94b63f6..6b8666a 100644 --- a/index.html +++ b/index.html @@ -326,7 +326,7 @@
⚙️ Settings
diff --git a/js/views-ai.js b/js/views-ai.js index 16eafcc..b519e00 100644 --- a/js/views-ai.js +++ b/js/views-ai.js @@ -149,11 +149,20 @@ function aiAnswerHtml(text, truncated) { return renderMarkdown(text)+(truncated?'
This answer was cut off. It reached the length limit, so the end is missing. Ask again, or ask a narrower question.
':''); } +// stop_reason "refusal": the API's safety classifiers declined the request. It arrives as a +// normal HTTP 200, not an error, so check stop_reason rather than the content. There is no +// answer to show or keep. stop_details.explanation is display text (it may be null) — show +// it, don't parse it. +function aiRefusalHtml(refusal) { + const why=refusal&&refusal.explanation?`
${escapeHtml(refusal.explanation)}
`:''; + return `
Claude declined to answer this question.${why}
Try asking it a different way.
`; +} + // Every failure path ends in a visible message on the page — never a silent no-op. // // Raw mode — callClaude(question, label, { raw:true, summary }) — sends the same request // but touches no DOM, no busy flag and no AI history. It resolves to -// { text, stopReason, truncated, usage, model } or throws (HTTP errors carry .status and .detail). +// { text, stopReason, truncated, refusal, usage, model } or throws (HTTP errors carry .status and .detail). // The AI Eval view (views-eval.js) uses it so the eval measures exactly what this tab sends. async function callClaude(question, label, opts={}) { if (opts.raw) return send(opts.summary!=null?opts.summary:buildFinancialSummary()); @@ -179,7 +188,8 @@ async function callClaude(question, label, opts={}) { } try { - const { text, truncated }=await send(summary); + const { text, truncated, refusal }=await send(summary); + if (refusal) { respEl.innerHTML=aiRefusalHtml(refusal); return; } // no answer: nothing to copy or keep respEl.innerHTML=aiAnswerHtml(text, truncated); if (aiNoAnswer(text, truncated)) return; // nothing to copy or keep (finally clears aiBusy) // A cut-off answer says so on screen, in copies and saved notes, and in history. @@ -230,7 +240,10 @@ async function callClaude(question, label, opts={}) { // Join all text blocks (don't assume the first content block is text) const text=(data.content||[]).filter(b=>b.type==='text').map(b=>b.text).join('\n\n')||AI_NO_TEXT; const stopReason=data.stop_reason||null; - return { text, stopReason, truncated:AI_CUT_OFF_REASONS.includes(stopReason), usage:data.usage||null, model:data.model||AI_MODEL }; + // A refusal is not an answer: drop any partial text and return why, for display. + const sd=data.stop_details||{}; + const refusal=stopReason==='refusal'?{ category:sd.category??null, explanation:sd.explanation??null }:null; + return { text:refusal?AI_NO_TEXT:text, stopReason, truncated:AI_CUT_OFF_REASONS.includes(stopReason), refusal, usage:data.usage||null, model:data.model||AI_MODEL }; } finally { clearTimeout(timer); } diff --git a/js/views-eval.js b/js/views-eval.js index 8ede5d3..75fb5a2 100644 --- a/js/views-eval.js +++ b/js/views-eval.js @@ -225,6 +225,7 @@ const AIEval = { ...job, text: result ? result.text : null, stopReason: result ? result.stopReason : null, + refusal: result ? result.refusal || null : null, // stop_details category/explanation when declined usage: result ? result.usage : null, model: result ? result.model : null, ms: Math.round(performance.now() - t0), @@ -879,7 +880,9 @@ const AIEval = { score(runs, summary) { const S = this.parseSummary(summary); const forecastIdx = AI_PROMPTS.findIndex(([label]) => this.FORECAST_LABEL.test(label)); - const rows = runs.filter(r => r.text != null).map(r => { + // A declined run (stop_reason "refusal") has no answer, so it isn't scored; it is counted + // separately, like a failed call. + const rows = runs.filter(r => r.text != null && r.stopReason !== 'refusal').map(r => { const { ex, deps, figures } = this.classifyText(r.text, S, r.prompt === forecastIdx); const strict = this.classifyText(r.text, S, r.prompt === forecastIdx, 'strict').figures; figures.forEach((f, i) => { @@ -911,6 +914,7 @@ const AIEval = { overall: this._tally(all), byKind: { '$': this._tally(all.filter(f => f.kind === '$')), '%': this._tally(all.filter(f => f.kind === '%')) }, strict: strictAll, byPrompt, byRep, failures, tags, runsScored: rows.length, runsFailed: runs.filter(r => r.text == null).length, + runsDeclined: runs.filter(r => r.text != null && r.stopReason === 'refusal').length, truncated: runs.filter(r => AI_CUT_OFF_REASONS.includes(r.stopReason)).length, tokens: this._tokenStats(runs), calibration: this._calibrate(rows, S, forecastIdx), @@ -922,7 +926,7 @@ const AIEval = { // isn't shown). maxTokens: the run's own setting, or for runs before v1.10.1 (which didn't // record it) the size every cut-off answer stopped at. _tokenStats(runs) { - const ok = runs.filter(r => r.usage && r.usage.output_tokens != null); + const ok = runs.filter(r => r.usage && r.usage.output_tokens != null && r.stopReason !== 'refusal'); // answers only const think = ok.filter(r => r.usage.output_tokens_details && r.usage.output_tokens_details.thinking_tokens != null); const avg = xs => (xs.length ? Math.round(xs.reduce((a, b) => a + b, 0) / xs.length) : null); const cut = ok.filter(r => AI_CUT_OFF_REASONS.includes(r.stopReason)).map(r => r.usage.output_tokens); @@ -1057,7 +1061,7 @@ const AIEval = { L.push(`| Model | \`${meta.model}\` (\`AI_MODEL\` in \`js/views-ai.js\`) |`); const utc = iso => new Date(iso).toISOString().slice(0, 16).replace('T', ' '); // same text wherever it's re-scored L.push(`| Run | ${utc(meta.startedAt)} – ${utc(meta.finishedAt).slice(11)} UTC, FinLens v${meta.appVersion} |`); - L.push(`| Calls | ${AI_PROMPTS.length} prompts × ${meta.repeats} repeats = ${AI_PROMPTS.length * meta.repeats} (${rep.runsScored} answered${rep.runsFailed ? `, ${rep.runsFailed} failed` : ''}${rep.truncated ? `, ${rep.truncated} cut off at max_tokens` : ''}); \`max_tokens\` ${rep.tokens.maxTokens || 'not recorded'}, default temperature |`); + L.push(`| Calls | ${AI_PROMPTS.length} prompts × ${meta.repeats} repeats = ${AI_PROMPTS.length * meta.repeats} (${rep.runsScored} answered${rep.runsFailed ? `, ${rep.runsFailed} failed` : ''}${rep.runsDeclined ? `, ${rep.runsDeclined} declined (\`stop_reason\` refusal, not scored)` : ''}${rep.truncated ? `, ${rep.truncated} cut off at max_tokens` : ''}); \`max_tokens\` ${rep.tokens.maxTokens || 'not recorded'}, default temperature |`); if (rep.tokens.output != null) L.push(`| Tokens | ${rep.tokens.output} output tokens per answer on average${rep.tokens.thinking != null ? `, of which ${rep.tokens.thinking} thinking (not shown, but counted toward \`max_tokens\`)${rep.tokens.thinkingCut != null ? `; ${rep.tokens.thinkingCut} in the answers that were cut off` : ''}` : ''} |`); L.push(`| Data | \`samples/\` only — synthetic Ally Checking, Ally Savings and Chase CSVs (${ds.txCount} transactions, ${dayjs(ds.minDate).format('MMM D')} – ${dayjs(ds.maxDate).format('MMM D, YYYY')}), date range **All Time** |`); L.push(`| Files | ${(this.samples || []).map(s => `\`${s.name}\` sha256 \`${s.sha256.slice(0, 12)}…\``).join(', ')} |`); @@ -1224,7 +1228,7 @@ registerView('ai-eval', 'AI Eval', (root) => {
Repeats: ${rep.byRep.map(t => `#${t.rep} grounded ${t.pct('grounded').toFixed(1)}% · derived ${t.pct('derived').toFixed(1)}% · unsupported ${t.pct('unsupported').toFixed(1)}%`).join('  |  ')}
Strict ±$1 / ±0.5 pts (no rounding allowance): grounded ${rep.strict.pct('grounded').toFixed(1)}% · unsupported ${rep.strict.pct('unsupported').toFixed(1)}%. - ${rep.runsFailed ? ` · ${rep.runsFailed} calls failed` : ''}${rep.truncated ? ` · ${rep.truncated} answers cut off at max_tokens` : ''}${rep.tokens.thinking != null ? ` · thinking ${rep.tokens.thinking} of ${rep.tokens.output} output tokens per answer` : ''}
`; + ${rep.runsFailed ? ` · ${rep.runsFailed} calls failed` : ''}${rep.runsDeclined ? ` · ${rep.runsDeclined} declined (not scored)` : ''}${rep.truncated ? ` · ${rep.truncated} answers cut off at max_tokens` : ''}${rep.tokens.thinking != null ? ` · thinking ${rep.tokens.thinking} of ${rep.tokens.output} output tokens per answer` : ''}`; root.appendChild(tbl); const fail = document.createElement('div'); fail.className = 'card'; @@ -1239,7 +1243,7 @@ registerView('ai-eval', 'AI Eval', (root) => { const answers = document.createElement('div'); answers.className = 'card'; answers.innerHTML = `

Answers

` + E.runs.map((r, i) => `
- ${escapeHtml(r.label)} — repeat ${r.rep} ${r.error ? `${escapeHtml(r.error)}` : `${(r.ms / 1000).toFixed(1)}s${AI_CUT_OFF_REASONS.includes(r.stopReason) ? ' · cut off' : ''}`} + ${escapeHtml(r.label)} — repeat ${r.rep} ${r.error ? `${escapeHtml(r.error)}` : `${(r.ms / 1000).toFixed(1)}s${AI_CUT_OFF_REASONS.includes(r.stopReason) ? ' · cut off' : ''}${r.stopReason === 'refusal' ? ' · declined' : ''}`}
`).join(''); root.appendChild(answers); diff --git a/js/views-settings.js b/js/views-settings.js index aa9d558..3004b59 100644 --- a/js/views-settings.js +++ b/js/views-settings.js @@ -49,7 +49,7 @@ function _renderGeneralSettings(root) { -
FinLens v1.10.2 ·${State.transactions.length} transactions loaded${State.lastRefresh?' · Refreshed '+dayjs(State.lastRefresh).format('MMM D, YYYY h:mm A'):''}
`; +
FinLens v1.10.3 ·${State.transactions.length} transactions loaded${State.lastRefresh?' · Refreshed '+dayjs(State.lastRefresh).format('MMM D, YYYY h:mm A'):''}
`; root.appendChild(displayCard); /* ---- Anthropic API Key ---- */