Skip to content

v1.10.1: fix AI answers cut off mid-sentence; flag cut-off answers - #12

Merged
gitluis merged 4 commits into
masterfrom
fix-ai-truncated-answers
Oct 26, 2026
Merged

gitluis merged 4 commits into
masterfrom
fix-ai-truncated-answers

Conversation

@gitluis

@gitluis gitluis commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Merge order: this (v1.10.1) → #13 (v1.10.2) → #14 (v1.10.3) → #10 (v1.11.0), per Luis on 2026-10-26. Based on master; #6, which added send(), is merged. Ready for review: the eval re-run is in (fa46ff8).

Summary

AI Analysis answers were being cut off mid-sentence and shown as if complete. This gives answers room to finish, and flags any answer that still gets cut off.

Cause

The v1.10.0 eval (#6) found that 15 of 24 answers stopped at max_tokens (1,500). The same 5 questions were cut off in all 3 repeats; the other 3 never were. Example ending: "Trimming 20% saves about $370 per".

Sonnet 5.5 thinks before answering (default effort high), and thinking counts toward max_tokens even though FinLens only shows the text. The eval's own usage data shows it (usage.output_tokens_details.thinking_tokens):

Answers Thinking Visible answer
Cut off (15) 747 tokens on average 753
Finished (9) 458 819

The prompt already asks for "under 500 words", and the docs say prompt instructions don't reliably reduce Sonnet 5.5's thinking. The fix is room, not wording.

The AI tab also never checked stop_reason, so it couldn't tell a finished answer from a cut-off one.

What changed

  • js/views-ai.js
    • Room to finish: max_tokens is now AI_MAX_TOKENS = 4,096. That's room for thinking plus the answer, and at the eval's speeds it still finishes well inside the 90-second timeout.
    • send() returns truncated when stop_reason is max_tokens or model_context_window_exceeded.
    • Cut-off answers are flagged:
      • the AI tab shows a warning under the answer;
      • history saves truncated: true, so reopening the answer shows the warning;
      • copies and saved notes end with a "Cut off" line.
    • Cut off while thinking: if Claude reaches the limit before writing any text, the tab says no answer was written, instead of "(no response)" plus "the end is missing". Nothing is saved to history or left to copy (aiNoAnswer(), AI_NO_TEXT). Added in review (bf8e09a).
  • js/views-eval.js
    • EVAL.md gains a Tokens row: output tokens per answer, the thinking share, and thinking in the cut-off answers.
    • The Calls row shows the run's real max_tokens. Runs now record meta.maxTokens; older runs infer it from where the cut-off answers stopped.
  • docs/EVAL.md — the eval re-run on this branch (below), with a hand-written Runs table comparing it to v1.10.0. The v1.10.0 report and raw answers moved to docs/eval-runs/v1.10.0/, unchanged except for an "Archived" banner. README's Evaluation numbers are updated, and CLAUDE.md §21 says how to publish a new run.
  • v1.10.1 (patch release; v1.17.0: Settings → Database tab (DB Inspector, DB Schema), rebuilt on the Clarity redesign #10 keeps v1.11.0) in CHANGELOG, index.html, README and views-settings.js. CLAUDE.md §13 and §21 are updated.

Not changed: the effort level and the system prompt, so the re-run measures this fix alone.

How it was tested

Headless Chromium, mocked API, on this branch (#6 + this change):

  • Cut-off answer (mock returns stop_reason: max_tokens plus a thinking block):
    • the warning is shown;
    • history saves truncated: true without changing the answer text;
    • the copy text ends with the cut-off note;
    • reopening it from history shows the warning again.
  • Finished answer: no warning and no flag. Old history entries without the flag still load.
  • Request: sends max_tokens: 4096.
  • Eval on the v1.10.0 JSON: headline unchanged. EVAL.md shows max_tokens 1500 (inferred) and the Tokens row: 1,417 per answer, 639 thinking, 747 in cut-off answers.
  • Mock eval run: records meta.maxTokens = 4096.
  • Regression: classifier 36/36, sandbox = real import path, isolation (no user data or key in requests or exports), 24-call mock run with a 529 retry, AI tab success and 401, file:// picker, tampered/BOM/CRLF samples. All pass.

Re-tested on bf8e09a: 28/28 headless checks pass. Besides the checks above:

  • a response with only a thinking block says no answer was written, saves nothing to history and leaves nothing to copy;
  • the footer, Settings, CHANGELOG, README and CLAUDE.md all say v1.10.1 (renumbered from v1.11.1 in d6701a2, no code change).

Re-tested on fa46ff8: 29/29 pass, including re-scoring both committed runs: v1.10.0 from docs/eval-runs/v1.10.0/ (unchanged) and v1.10.1 (0 cut off, max_tokens 4096). The committed EVAL.md is exactly what the view generates from docs/eval-results.json, plus the hand-written sections.

Eval re-run

Run by Luis on 2026-10-26, 12:00–12:05 UTC, on this branch (v1.10.1). Same model, prompts, samples, summary and matcher as the v1.10.0 run; only max_tokens changed. No errors or retries.

v1.10.0 v1.10.1
max_tokens 1,500 4,096
Answers cut off 15 of 24 0 of 24
Output tokens per answer (thinking) 1,417 (639) 1,560 (669)
Longest answer 1,500 (the cap) 2,054
Words per answer (longest) 343 (414) 401 (486)
Slowest call 17.7 s 19.5 s
Grounded / traceable 48.6% / 68.8% 45.7% / 66.9%
Unsupported 28.4% 29.2%
  • Cut-off answers are gone: all 5 prompts that were cut off in every repeat now finish, using at most half the new limit.
  • Grounding dipped by about the spread between repeats (43.8–48.0% grounded across this run's repeats). The 3 prompts that always finished held steady or improved; the change comes from the endings that used to be missing. About 25 of the 46 unsupported "claims about the data" are monthly figures over a guessed period length (groceries ~$470, ~$515 or ~$516 a month for the same total), which is what v1.10.2: AI summary states its dates; stop the model guessing the period length #13 fixes.

Full results: docs/EVAL.md.

Please check before merging

  • The re-run shows cut-off answers going from 15 toward 0, with the numbers in EVAL.md.
  • Ask the AI tab a long question and check that answers read complete.
  • You're OK with answers using up to 4,096 output tokens. You pay for the tokens actually produced, not the cap: 1,560 per answer in the re-run, longest 2,054 (1,417 at the old 1,500 cap).

Opened in https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq; re-versioned in the session below.


https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR


Generated by Claude Code

In the v1.10.0 eval, 15 of 24 AI Analysis answers stopped at
max_tokens (1,500) and the AI tab showed them as complete. Sonnet 5.5
thinks before answering (default effort high) and thinking counts
toward max_tokens: it took 747 of 1,500 tokens on average in the
cut-off answers (usage.output_tokens_details.thinking_tokens).

- max_tokens is now AI_MAX_TOKENS = 4096
- send() returns truncated for stop_reason max_tokens or
  model_context_window_exceeded; the AI tab shows a warning, history
  keeps truncated: true, copies and saved notes end with a cut-off line
- AI Eval records meta.maxTokens and reports output and thinking
  tokens per answer (inferred max_tokens for older runs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
Luis decided on 2026-10-26 that #12 and #13 merge before #10. Master is
at v1.10.0, so this bug fix is the patch release v1.10.1 (CLAUDE.md
§17); #10 keeps v1.11.0. Renumbered in CHANGELOG, index.html, README,
views-settings.js, CLAUDE.md §1/§13/§21/last-updated, and the
views-eval.js comment on runs that didn't record maxTokens. No code
change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
gitluis pushed a commit that referenced this pull request Oct 26, 2026
…on to v1.10.2

#12 and #13 now merge before #10 (Luis, 2026-10-26), so #12 is v1.10.1
and this fix is v1.10.2; #10 keeps v1.11.0. Conflicts were only the
version strings. Every v1.11.2 is now v1.10.2 and every v1.11.1 is
v1.10.1 (CHANGELOG, index.html, README, views-settings.js, CLAUDE.md,
and comments in views-ai.js and views-eval.js). No code change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
@gitluis gitluis changed the title v1.11.1: fix AI answers cut off mid-sentence; flag cut-off answers v1.10.1: fix AI answers cut off mid-sentence; flag cut-off answers Oct 26, 2026
…thinking

Review follow-up to the cut-off warning. When max_tokens is reached
during thinking, the response has no text block, and the tab showed
"(no response)" with a note that "the end is missing". It now says no
answer was written and keeps nothing in history or for copying.

- AI_NO_TEXT names send()'s placeholder for a response with no text;
  aiNoAnswer() tests for a cut-off answer with no text
- send() and raw mode are unchanged, so the eval measures the same thing
- CHANGELOG 1.10.1 and CLAUDE.md §13 (also: raw mode returns truncated)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
…10.0

Luis ran the AI Eval on this branch (v1.10.1, max_tokens 4096) on
2026-10-26. Same model, prompts, samples, summary and matcher as the
v1.10.0 run; only max_tokens changed.

- Cut-off answers 15 -> 0; longest answer 2,054 of 4,096 tokens;
  output tokens per answer 1,417 -> 1,560; slowest call 19.5 s
- Grounded 48.6% -> 45.7%, traceable 68.8% -> 66.9%: about the spread
  between repeats, and it comes from the endings that used to be cut
  off, mostly monthly figures over a guessed period length (v1.10.2)
- docs/EVAL.md generated by loading the new results JSON in the view,
  plus a hand-written Runs table and findings
- v1.10.0 report and raw answers moved to docs/eval-runs/v1.10.0/
  with an "Archived" banner; README numbers, CLAUDE.md §3 and §21
  (how to publish a new run), CHANGELOG 1.10.1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
gitluis pushed a commit that referenced this pull request Oct 26, 2026
gitluis pushed a commit that referenced this pull request Oct 26, 2026
@gitluis
gitluis marked this pull request as ready for review October 26, 2026 12:12

gitluis commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

Approved by Luis in chat on 2026-10-26. His words:

Awesome! #12 is approved, proceed with merging.

Recorded here before merging, per CLAUDE.md §0. Merging next as a merge commit.


🤖 Generated with Claude Code


Generated by Claude Code

@gitluis
gitluis merged commit 4610a62 into master Oct 26, 2026
@gitluis
gitluis deleted the fix-ai-truncated-answers branch October 26, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants