Repository navigation
v1.10.1: fix AI answers cut off mid-sentence; flag cut-off answers - #12
Merged
Merged
Conversation
In the v1.10.0 eval, 15 of 24 AI Analysis answers stopped at max_tokens (1,500) and the AI tab showed them as complete. Sonnet 5.5 thinks before answering (default effort high) and thinking counts toward max_tokens: it took 747 of 1,500 tokens on average in the cut-off answers (usage.output_tokens_details.thinking_tokens). - max_tokens is now AI_MAX_TOKENS = 4096 - send() returns truncated for stop_reason max_tokens or model_context_window_exceeded; the AI tab shows a warning, history keeps truncated: true, copies and saved notes end with a cut-off line - AI Eval records meta.maxTokens and reports output and thinking tokens per answer (inferred max_tokens for older runs) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq
This was referenced Oct 26, 2026
Luis decided on 2026-10-26 that #12 and #13 merge before #10. Master is at v1.10.0, so this bug fix is the patch release v1.10.1 (CLAUDE.md §17); #10 keeps v1.11.0. Renumbered in CHANGELOG, index.html, README, views-settings.js, CLAUDE.md §1/§13/§21/last-updated, and the views-eval.js comment on runs that didn't record maxTokens. No code change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
gitluis
pushed a commit
that referenced
this pull request
Oct 26, 2026
…on to v1.10.2 #12 and #13 now merge before #10 (Luis, 2026-10-26), so #12 is v1.10.1 and this fix is v1.10.2; #10 keeps v1.11.0. Conflicts were only the version strings. Every v1.11.2 is now v1.10.2 and every v1.11.1 is v1.10.1 (CHANGELOG, index.html, README, views-settings.js, CLAUDE.md, and comments in views-ai.js and views-eval.js). No code change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
…thinking Review follow-up to the cut-off warning. When max_tokens is reached during thinking, the response has no text block, and the tab showed "(no response)" with a note that "the end is missing". It now says no answer was written and keeps nothing in history or for copying. - AI_NO_TEXT names send()'s placeholder for a response with no text; aiNoAnswer() tests for a cut-off answer with no text - send() and raw mode are unchanged, so the eval measures the same thing - CHANGELOG 1.10.1 and CLAUDE.md §13 (also: raw mode returns truncated) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
2 tasks
…10.0 Luis ran the AI Eval on this branch (v1.10.1, max_tokens 4096) on 2026-10-26. Same model, prompts, samples, summary and matcher as the v1.10.0 run; only max_tokens changed. - Cut-off answers 15 -> 0; longest answer 2,054 of 4,096 tokens; output tokens per answer 1,417 -> 1,560; slowest call 19.5 s - Grounded 48.6% -> 45.7%, traceable 68.8% -> 66.9%: about the spread between repeats, and it comes from the endings that used to be cut off, mostly monthly figures over a guessed period length (v1.10.2) - docs/EVAL.md generated by loading the new results JSON in the view, plus a hand-written Runs table and findings - v1.10.0 report and raw answers moved to docs/eval-runs/v1.10.0/ with an "Archived" banner; README numbers, CLAUDE.md §3 and §21 (how to publish a new run), CHANGELOG 1.10.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
gitluis
pushed a commit
that referenced
this pull request
Oct 26, 2026
…ary-date-range # Conflicts: # CLAUDE.md
gitluis
pushed a commit
that referenced
this pull request
Oct 26, 2026
gitluis
marked this pull request as ready for review
October 26, 2026 12:12
Owner
Author
|
Approved by Luis in chat on 2026-10-26. His words:
Recorded here before merging, per CLAUDE.md §0. Merging next as a merge commit. 🤖 Generated with Claude Code Generated by Claude Code |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AI Analysis answers were being cut off mid-sentence and shown as if complete. This gives answers room to finish, and flags any answer that still gets cut off.
Cause
The v1.10.0 eval (#6) found that 15 of 24 answers stopped at
max_tokens(1,500). The same 5 questions were cut off in all 3 repeats; the other 3 never were. Example ending: "Trimming 20% saves about $370 per".Sonnet 5.5 thinks before answering (default effort
high), and thinking counts towardmax_tokenseven though FinLens only shows the text. The eval's own usage data shows it (usage.output_tokens_details.thinking_tokens):The prompt already asks for "under 500 words", and the docs say prompt instructions don't reliably reduce Sonnet 5.5's thinking. The fix is room, not wording.
The AI tab also never checked
stop_reason, so it couldn't tell a finished answer from a cut-off one.What changed
js/views-ai.jsmax_tokensis nowAI_MAX_TOKENS= 4,096. That's room for thinking plus the answer, and at the eval's speeds it still finishes well inside the 90-second timeout.send()returnstruncatedwhenstop_reasonismax_tokensormodel_context_window_exceeded.truncated: true, so reopening the answer shows the warning;aiNoAnswer(),AI_NO_TEXT). Added in review (bf8e09a).js/views-eval.jsmax_tokens. Runs now recordmeta.maxTokens; older runs infer it from where the cut-off answers stopped.docs/EVAL.md— the eval re-run on this branch (below), with a hand-written Runs table comparing it to v1.10.0. The v1.10.0 report and raw answers moved todocs/eval-runs/v1.10.0/, unchanged except for an "Archived" banner. README's Evaluation numbers are updated, and CLAUDE.md §21 says how to publish a new run.index.html, README andviews-settings.js. CLAUDE.md §13 and §21 are updated.Not changed: the effort level and the system prompt, so the re-run measures this fix alone.
How it was tested
Headless Chromium, mocked API, on this branch (#6 + this change):
stop_reason: max_tokensplus a thinking block):truncated: truewithout changing the answer text;max_tokens: 4096.max_tokens 1500(inferred) and the Tokens row: 1,417 per answer, 639 thinking, 747 in cut-off answers.meta.maxTokens = 4096.file://picker, tampered/BOM/CRLF samples. All pass.Re-tested on
bf8e09a: 28/28 headless checks pass. Besides the checks above:d6701a2, no code change).Re-tested on
fa46ff8: 29/29 pass, including re-scoring both committed runs: v1.10.0 fromdocs/eval-runs/v1.10.0/(unchanged) and v1.10.1 (0 cut off,max_tokens4096). The committed EVAL.md is exactly what the view generates fromdocs/eval-results.json, plus the hand-written sections.Eval re-run
Run by Luis on 2026-10-26, 12:00–12:05 UTC, on this branch (v1.10.1). Same model, prompts, samples, summary and matcher as the v1.10.0 run; only
max_tokenschanged. No errors or retries.max_tokensFull results:
docs/EVAL.md.Please check before merging
Opened in https://claude.ai/code/session_01WuftAn7qz6C2y8AQp2EkWq; re-versioned in the session below.
https://claude.ai/code/session_01QiRz5pTW83oqNbk2NC7yMR
Generated by Claude Code