Repository navigation
fix(compaction): recover from near-full-context compaction failures, silent no-ops and stalled requests - #356
Merged
Merged
Conversation
…ream idle watchdog Three provider-level changes behind the near-full-context failures: - Estimate prompt size from UTF-8 bytes instead of JavaScript characters, so a han character costs 0.75 token (the gateway bills about 0.61) instead of 0.25. ASCII text keeps its previous estimate; images keep 1200 tokens. - Classify the gateway's unnamed `The input you provided is invalid` rejection as overflow when the rejected prompt filled at least half the window and 95% of it with the output reservation. The size evidence is measured lazily and only for that wording, so a malformed small request keeps its own error. - Bound a provider stream by event idleness: a stream that keeps sending keep-alives while the model makes no progress no longer holds the request open indefinitely. On expiry the request is cancelled and a transient error lets the retry policy resend it.
… overflow in summaries - Both message estimators, the lightweight projection and the summary request's own sizing now price text by bytes. - When the newest message alone exceeded keepRecentTokens and was a tool result, no cut point qualified, so the whole history was kept and automatic compaction silently did nothing. Fall back to the newest valid cut point, as upstream pi does. - A summary request rejected with the gateway's unnamed invalid-request body is now reduced and retried instead of failing unchanged. - CLM's usage calibration may now fall below 1, because byte pricing is deliberately conservative for CJK and only observed usage can correct it.
…indow Route `invalid_request_error: The input you provided is invalid` through the existing compact-and-retry-once path when the rejected prompt filled at least half the window and 95% of it with the output reservation. Before this, such a run failed outright with no compaction and no retry.
…ounded stalls - `compaction.contextProjection` now defaults to `off` (native summary compaction). CLM stays available through `contextProjection = "clm-v1"`, `--context-projection clm-v1`, or `/clm on`; explicit settings are unchanged. Near a full window CLM's maintenance request carried almost the whole context (about 975K tokens at a 1M window) to save about 12%, so prompts stayed slow: 47–72s against 16–19s for native compaction on a 930K-token session. - Install the stream idle watchdog in the sdk stream function, which serves task, CLM maintenance and summary requests, with `retry.provider.streamIdleTimeoutMs` (default 300000, 0 disables). Verified against the gateway: at 2000ms a 600K-token request was cut at the stall and resent by the retry policy.
Compaction, context-management and settings pages describe native summary compaction as the default with CLM opt-in, the 85% trigger, byte-priced estimates, the unnamed-overflow rule, the cold-prefill bound and the stream idle timeout. CHANGELOG carries the same entries. The test harness under test/repro reproduces the reported request sequences and payload sizes offline: fifteen scenarios across the CLM and native paths, size and local-CPU measurements at 300K/600K/900K, and a stability run.
ZouR-Ma
requested review from
MelodyVAR,
Uking-xxx and
liuedcson
as code owners
November 11, 2026 03:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Users reported that once the context approaches the window limit, every task becomes very slow, sometimes stalling for 20-30 minutes.
Reproduction against the real gateway with the real CLI (40+ requests, about 20M input tokens) isolated three separate problems:
The input you provided is invalid [code=30003]. That wording matches no overflow rule, so the run ended with no compaction and no retry. 5 of 16 real runs failed this way (an earlier batch failed 10 of 13).The reproduction also surfaced an existing bug: when the newest message alone exceeds
keepRecentTokensand is a tool result, no cut point qualifies, so the whole history is kept, nothing is summarized, and automatic compaction silently does nothing. The first acceptance round passed 9/9 only because the corrected output reservation happened to fit, not because compaction ran.Changes
feat(providers)fix(compaction)fix(compaction)feat(compaction)retry.provider.streamIdleTimeoutMsdocs(compaction)Two related value changes: the default
compaction.triggerRatiomoves from 0.9 to 0.85 (worst-case tolerance for estimation error grows from about 53K to 72K tokens), andcompaction.autoClm.coldMaxPrefillTokens(default 262144) skips a cold CLM maintenance request in favour of native compaction.Explicit
clm-v1andlightweight-v1settings keep their behaviour;/clm onand--context-projection clm-v1still enable CLM.Measured on the real gateway (same scripts, model
deepseek-v4.1-flash-api)Verification
tsgo --noEmitand biome pass.settings-manager,regressions/2781,8337,scripts/release-notificationall fail without this branch as well).streamIdleTimeoutMs=2000, a 600K-token request was cut at the stall and resent by the retry policy.Known gaps
test/stream-idle-watchdog.test.tsis skipped: a provider registered through an extension in that harness does not pass through the sdk stream function (noted in the test). The watchdog itself is covered by its own unit tests.