Skip to content

fix(compaction): recover from near-full-context compaction failures, silent no-ops and stalled requests - #356

Merged
ZouR-Ma merged 5 commits into
mainfrom
feat/near-full-context-compaction
Nov 11, 2026
Merged

ZouR-Ma merged 5 commits into
mainfrom
feat/near-full-context-compaction

Conversation

@ZouR-Ma

@ZouR-Ma ZouR-Ma commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Users reported that once the context approaches the window limit, every task becomes very slow, sometimes stalling for 20-30 minutes.

Reproduction against the real gateway with the real CLI (40+ requests, about 20M input tokens) isolated three separate problems:

  1. Slow near the limit. At 930K context, every prompt is preceded by a maintenance request carrying about 975K tokens (roughly 30s), and the task request still needs 26s to its first token. The same command took 7.6s at 60K context and 62s at 930K.
  2. Hard failures. With a large Chinese tool output at the end of the history, prompt size was undercounted by 2.45x, so the host believed it had not reached the threshold and sent the request unchanged; it exceeded the gateway limit and came back as The input you provided is invalid [code=30003]. That wording matches no overflow rule, so the run ended with no compaction and no retry. 5 of 16 real runs failed this way (an earlier batch failed 10 of 13).
  3. Stalls. The client only gives up after 300s with no bytes at all, and keep-alives keep resetting that timer; one request in the logs hung for 19.5 minutes.

The reproduction also surfaced an existing bug: when the newest message alone exceeds keepRecentTokens and is a tool result, no cut point qualifies, so the whole history is kept, nothing is summarized, and automatic compaction silently does nothing. The first acceptance round passed 9/9 only because the corrected output reservation happened to fit, not because compaction ran.

Changes

Commit Scope
feat(providers) Estimate prompt size from UTF-8 bytes; classify the gateway's unnamed 400 as overflow when size evidence supports it; bound provider streams by event idleness
fix(compaction) Apply byte estimates to every estimator and to summary sizing; cut-point fallback (matching upstream pi); reduce and retry a summary rejected with the unnamed 400; allow CLM calibration below 1
fix(compaction) Route the unnamed 400 into the existing compact-and-retry-once path at session level
feat(compaction) Default to native summary compaction with CLM opt-in; install the watchdog and retry.provider.streamIdleTimeoutMs
docs(compaction) Documentation, CHANGELOG, and the offline reproduction harness

Two related value changes: the default compaction.triggerRatio moves from 0.9 to 0.85 (worst-case tolerance for estimation error grows from about 53K to 72K tokens), and compaction.autoClm.coldMaxPrefillTokens (default 262144) skips a cold CLM maintenance request in favour of native compaction.

Explicit clm-v1 and lightweight-v1 settings keep their behaviour; /clm on and --context-projection clm-v1 still enable CLM.

Measured on the real gateway (same scripts, model deepseek-v4.1-flash-api)

Scenario Before After
Chinese tail, default mode 2 of 3 failed 2/2 succeeded, about 21s
Chinese tail, native mode 2 of 3 failed (and never compacted) 2/2 succeeded, 21-26s
880K with no trustworthy usage 1 of 3 failed 2/2 succeeded, about 16s
930K, cold cache 47-72s 16.3s
60K baseline 7.6s 3.6s

Verification

  • tsgo --noEmit and biome pass.
  • providers 531, agent-core 566, coding-agent 638 across the 25 compaction-related files, plus 11 CLI tests.
  • The pre-existing failures are unrelated to this change (6 in settings-manager, regressions/2781, 8337, scripts/release-notification all fail without this branch as well).
  • The watchdog was verified against the gateway: with streamIdleTimeoutMs=2000, a 600K-token request was cut at the stall and resent by the retry policy.

Known gaps

  • The end-to-end watchdog case in test/stream-idle-watchdog.test.ts is skipped: a provider registered through an extension in that harness does not pass through the sdk stream function (noted in the test). The watchdog itself is covered by its own unit tests.
  • Pruning oversized old tool output before compaction (the deepseek-harness pruner approach) is not implemented.
  • The reproduction scripts and evidence bundle (session JSONL, per-run reports) are provided separately and are not part of this PR.

…ream idle watchdog

Three provider-level changes behind the near-full-context failures:

- Estimate prompt size from UTF-8 bytes instead of JavaScript characters, so a
  han character costs 0.75 token (the gateway bills about 0.61) instead of
  0.25. ASCII text keeps its previous estimate; images keep 1200 tokens.
- Classify the gateway's unnamed `The input you provided is invalid` rejection
  as overflow when the rejected prompt filled at least half the window and 95%
  of it with the output reservation. The size evidence is measured lazily and
  only for that wording, so a malformed small request keeps its own error.
- Bound a provider stream by event idleness: a stream that keeps sending
  keep-alives while the model makes no progress no longer holds the request
  open indefinitely. On expiry the request is cancelled and a transient error
  lets the retry policy resend it.
… overflow in summaries

- Both message estimators, the lightweight projection and the summary
  request's own sizing now price text by bytes.
- When the newest message alone exceeded keepRecentTokens and was a tool
  result, no cut point qualified, so the whole history was kept and automatic
  compaction silently did nothing. Fall back to the newest valid cut point, as
  upstream pi does.
- A summary request rejected with the gateway's unnamed invalid-request body is
  now reduced and retried instead of failing unchanged.
- CLM's usage calibration may now fall below 1, because byte pricing is
  deliberately conservative for CJK and only observed usage can correct it.
…indow

Route `invalid_request_error: The input you provided is invalid` through the
existing compact-and-retry-once path when the rejected prompt filled at least
half the window and 95% of it with the output reservation. Before this, such a
run failed outright with no compaction and no retry.
…ounded stalls

- `compaction.contextProjection` now defaults to `off` (native summary
  compaction). CLM stays available through `contextProjection = "clm-v1"`,
  `--context-projection clm-v1`, or `/clm on`; explicit settings are unchanged.
  Near a full window CLM's maintenance request carried almost the whole context
  (about 975K tokens at a 1M window) to save about 12%, so prompts stayed slow:
  47–72s against 16–19s for native compaction on a 930K-token session.
- Install the stream idle watchdog in the sdk stream function, which serves
  task, CLM maintenance and summary requests, with
  `retry.provider.streamIdleTimeoutMs` (default 300000, 0 disables). Verified
  against the gateway: at 2000ms a 600K-token request was cut at the stall and
  resent by the retry policy.
Compaction, context-management and settings pages describe native summary
compaction as the default with CLM opt-in, the 85% trigger, byte-priced
estimates, the unnamed-overflow rule, the cold-prefill bound and the stream
idle timeout. CHANGELOG carries the same entries.

The test harness under test/repro reproduces the reported request sequences and
payload sizes offline: fifteen scenarios across the CLM and native paths, size
and local-CPU measurements at 300K/600K/900K, and a stability run.
@ZouR-Ma ZouR-Ma changed the title fix(compaction): 修复近满上下文下的压缩失败、静默不执行与请求挂起 fix(compaction): recover from near-full-context compaction failures, silent no-ops and stalled requests Nov 11, 2026
@ZouR-Ma
ZouR-Ma merged commit 671305a into main Nov 11, 2026
2 checks passed
@ZouR-Ma
ZouR-Ma deleted the feat/near-full-context-compaction branch November 11, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant