fix(compaction): recover from near-full-context compaction failures, silent no-ops and stalled requests

CHANGELOG.md

@@ -2,8 +2,17 @@
 
 ## [Unreleased]
 
+### Added
+
+- Added `retry.provider.streamIdleTimeoutMs` (default 300000): a provider stream that produces no parsed event for that long is cancelled and ends with a transient error, so the existing retry policy resends it. The HTTP idle timeout only sees bytes, and a gateway sending keep-alives could hold a stalled request open for tens of minutes.
+
 ### Changed
 
+- Changed the default compression mode to native summary compaction: `compaction.contextProjection` now defaults to `off`, and CLM is opt-in through `contextProjection = "clm-v1"`, `--context-projection clm-v1`, or `/clm on`. Near a full window CLM's maintenance request carried almost the whole context (about 975K tokens at a 1M window) to save about 12%, so prompts stayed large and slow; on a 930K-token session CLM took 47–72s per prompt against 16–19s for native compaction. Existing explicit `clm-v1` and `lightweight-v1` settings are unchanged.
+- Changed automatic CLM to skip a maintenance request that cannot reuse the cached actor prefix once it exceeds `compaction.autoClm.coldMaxPrefillTokens` (default 262144) and to use native compaction instead: near a 1M window such a request recomputed about 975K tokens (around 30s) to save roughly 12%, so CLM was slower than native compaction at the point where speed mattered most. Cached-prefix maintenance is unchanged; `0` removes the bound.
+- Changed prompt-size estimation to count UTF-8 bytes, not JavaScript characters: the old `length / 4` heuristic priced a han character at 0.25 token while the Step gateway bills about 0.61 (the byte estimate gives 0.75), so CJK-heavy history could sit above the provider limit while compaction still believed it fit. English prose and code keep the same estimate; images keep their fixed 1200-token estimate. The estimate feeds the compaction threshold, `max_tokens` clamping, CLM's working-context budget, the lightweight projection, and summary request sizing; CLM's usage calibration may now move below 1 to correct the slight CJK overestimate.
+- Changed the default `compaction.triggerRatio` from 0.9 to 0.85, leaving more room for estimation error before a request can exceed the window; explicit ratios and per-model overrides are unchanged, and windows below 131K keep their existing response-headroom thresholds.
+- Changed overflow classification to cover gateways that reject an oversized request without naming a reason: `The input you provided is invalid [code=30003]` now counts as overflow when the rejected prompt fills at least half the window and at least 95% of it once the output reservation is added, so it compacts and retries instead of failing the run. The same wording on a small request stays an ordinary error, and the measurement only runs when the wording matches.
 - The Step system prompt no longer carries the git branch or the uncommitted-file count in its `<env>` block. Any later prompt rebuild (an MCP catalog refresh, a tool registration) re-read git and rewrote the system prompt after a checkout, and because the provider caches the system prompt right after the tool block, one rebuild re-billed the whole conversation as cache writes; between rebuilds the model saw a stale branch. The branch now reaches the model as a hidden runtime-context snapshot appended beside the user's prompt only when it changed, so the system prompt and earlier history stay byte-identical. Within a run, a step that ran tools, or a run started without a user prompt (goal continuation, cron, subagent wake-ups, an approved plan), re-reads the branch and shows a change at the request tail. The uncommitted-file count is dropped rather than moved: it changes with almost every edit the agent itself makes, and the prompt already tells the model to inspect git state before changing files. SDK embedders that use `buildStepSystemPromptAppendix` without `createStepCapabilitiesExtensionInline` can add `createStepRuntimeContextExtension` to keep the branch visible.
 - Parallel subagent batches now run at a fixed default of 8 children instead of a CPU-derived ceiling (`max(1, min(16, cores - 2))`), which squeezed small machines down to one or two children even though a child waits on a remote model, not on a local core. `[subagent] max_concurrency` (StepCode JSON: `subagent.maxConcurrency`) sets it per batch and may be higher or lower than the default; an invalid value is a settings warning and is ignored. `STEP_WORKFLOW_CONCURRENCY_LIMIT` no longer bounds subagent batches — it caps workflow `agent()` fan-out only — so a backend that limits simultaneous requests needs `[subagent] max_concurrency` as well. Batches still take slots from the process-wide admission window shared with workflow nodes, whose capacity is the smallest ceiling among the runs attached, so a concurrent workflow with a lower ceiling also lowers a batch and a 429 seen by either lowers both. See [Step configuration](docs/step-configuration.md#subagents).
 - `AgentLoopConfig.toolCallLeakRetries` is removed. The low-level loop no longer guesses a tool intent from markup in text; it records the structural classification (`classifyIncompleteAssistantTurn()`, now in `@step-harness/providers`) on every finished assistant message as `incomplete`, and hosts decide what to do about it.
@@ -14,6 +23,7 @@
 
 ### Fixed
 
+- Fixed automatic compaction silently doing nothing when the newest message alone exceeded `keepRecentTokens` and was a tool result (for example one large file read near the threshold): no cut point qualified, so the whole history was kept and nothing was summarized. The cut now falls back to the newest valid cut point, matching upstream pi, so older history is summarized and the oversized tool batch is kept.
 - An OpenAI Responses stream that finishes with a tool call whose `output_item.done` never arrived now ends with an error instead of handing that call to the agent loop. Non-compliant servers that reuse or omit `output_index` (observed with OpenAI-compatible gateways) could otherwise produce duplicate or mixed-up tool calls in one assistant message, which ran the same tool twice and then failed every later turn on replay with `Duplicate value for 'tool_call_id'`.
 - Recover a normal assistant response that contains neither visible text nor an executable tool call: keep the original message in history unchanged, append one hidden user-role recovery prompt, and continue. Recovery is bounded by a streak of consecutive empty turns. Only a turn with visible text or an executed tool call resets it, and so does any user input, so a long autonomous run is not ended by one transient empty response and a provider error in between does not refill the budget. Automatic compaction runs first when the response also exceeded the context window.
 - A run that ends without a usable answer reports it on `agent_settled` as `failure` (reason and message), instead of rewriting the assistant message's provider stop reason, so a settled run no longer means success by itself. Subagent lanes and goal continuation act on it, and print/JSON pick their exit code from the same classification, so no consumer decides success from `stopReason` alone. `assistantTerminalFailure()` in `@step-harness/providers` applies the same rule to a message that has not settled yet. The hidden recovery message records the entry it answers in `details.recoversEntryId`.

packages/agent-core/src/harness/compaction/compaction.ts

@@ -3,6 +3,7 @@ import {
 	type AssistantMessage,
 	type Context,
 	contentText,
+	estimateTextTokens,
 	type Model,
 	type Models,
 	type RetryCallbacks,
@@ -245,61 +246,53 @@ export function shouldCompact(contextTokens: number, contextWindow: number, sett
 	return contextTokens > contextWindow - settings.reserveTokens;
 }
 
-const ESTIMATED_IMAGE_CHARS = 4800;
+/** Fixed per-image estimate; images carry no text to measure by bytes. */
+const ESTIMATED_IMAGE_TOKENS = 1200;
 
-function estimateTextAndImageContentChars(content: string | Array<{ type: string; text?: string }>): number {
-	if (typeof content === "string") {
-		return content.length;
-	}
+/** Content blocks mix text (priced by UTF-8 bytes) with images (fixed estimate). */
+function estimateTextAndImageTokens(content: string | Array<{ type: string; text?: string }>): number {
+	if (typeof content === "string") return estimateTextTokens(content);
 
-	let chars = 0;
+	let tokens = 0;
 	for (const block of content) {
-		if (block.type === "text" && block.text) {
-			chars += block.text.length;
-		} else if (block.type === "image") {
-			chars += ESTIMATED_IMAGE_CHARS;
-		}
+		if (block.type === "text" && block.text) tokens += estimateTextTokens(block.text);
+		else if (block.type === "image") tokens += ESTIMATED_IMAGE_TOKENS;
 	}
-	return chars;
+	return tokens;
 }
 
-/** Estimate token count for one message using a conservative character heuristic. */
+/** Estimate token count for one message from its UTF-8 payload size. */
 export function estimateTokens(message: AgentMessage): number {
-	let chars = 0;
-
 	switch (message.role) {
 		case "user": {
-			chars = estimateTextAndImageContentChars(
+			return estimateTextAndImageTokens(
 				(message as { content: string | Array<{ type: string; text?: string }> }).content,
 			);
-			return Math.ceil(chars / 4);
 		}
 		case "assistant": {
 			const assistant = message as AssistantMessage;
+			let tokens = 0;
 			for (const block of assistant.content) {
 				if (block.type === "text") {
-					chars += block.text.length;
+					tokens += estimateTextTokens(block.text);
 				} else if (block.type === "thinking") {
-					chars += block.thinking.length;
+					tokens += estimateTextTokens(block.thinking);
 				} else if (block.type === "toolCall") {
-					chars += block.name.length + safeJsonStringify(block.arguments).length;
+					tokens += estimateTextTokens(block.name) + estimateTextTokens(safeJsonStringify(block.arguments));
 				}
 			}
-			return Math.ceil(chars / 4);
+			return tokens;
 		}
 		case "custom":
 		case "toolResult": {
-			chars = estimateTextAndImageContentChars(message.content);
-			return Math.ceil(chars / 4);
+			return estimateTextAndImageTokens(message.content);
 		}
 		case "bashExecution": {
-			chars = message.command.length + message.output.length;
-			return Math.ceil(chars / 4);
+			return estimateTextTokens(message.command) + estimateTextTokens(message.output);
 		}
 		case "branchSummary":
 		case "compactionSummary": {
-			chars = message.summary.length;
-			return Math.ceil(chars / 4);
+			return estimateTextTokens(message.summary);
 		}
 	}
 
@@ -387,12 +380,10 @@ export function findCutPoint(
 		const messageTokens = estimateTokens(entry.message as AgentMessage);
 		accumulatedTokens += messageTokens;
 		if (accumulatedTokens >= keepRecentTokens) {
-			for (let c = 0; c < cutPoints.length; c++) {
-				if (cutPoints[c] >= i) {
-					cutIndex = cutPoints[c];
-					break;
-				}
-			}
+			// Closest valid cut point at or after this entry. When the newest message alone exceeds
+			// the budget and is a tool result (never a cut point), nothing qualifies; keep from the
+			// newest cut point instead of keeping everything, which left nothing to summarize.
+			cutIndex = cutPoints.find((candidate) => candidate >= i) ?? cutPoints[cutPoints.length - 1];
 			break;
 		}
 	}

packages/agent-core/src/harness/compaction/projection-content.ts

@@ -5,6 +5,7 @@
  */
 
 import type { ImageContent, Message, TextContent, ToolResultMessage, UserMessage } from "@step-harness/providers";
+import { estimateMessageTokens as estimateProviderMessageTokens } from "@step-harness/providers";
 
 // ============================================================================
 // Rewrite markers
@@ -30,7 +31,6 @@ export function containsProjectionMarker(text: string): boolean {
 // Content sizing
 // ============================================================================
 
-const CHARS_PER_TOKEN = 4;
 const ESTIMATED_IMAGE_CHARS = 4800;
 
 function safeJsonStringify(value: unknown): string {
@@ -64,14 +64,16 @@ export function totalContentChars(messages: readonly Message[]): number {
 	return chars;
 }
 
-/** chars/4 estimate, deliberately simple and deterministic. */
+/** UTF-8 bytes/4 estimate shared with the compaction trigger, deliberately simple and deterministic. */
 export function estimateProjectionTokens(messages: readonly Message[]): number {
-	return Math.ceil(totalContentChars(messages) / CHARS_PER_TOKEN);
+	let tokens = 0;
+	for (const message of messages) tokens += estimateMessageTokens(message);
+	return tokens;
 }
 
-/** chars/4 estimate for a single message. */
+/** UTF-8 bytes/4 estimate for a single message; CJK text is priced at its real cost. */
 export function estimateMessageTokens(message: Message): number {
-	return Math.ceil(messageContentChars(message) / CHARS_PER_TOKEN);
+	return estimateProviderMessageTokens(message);
 }
 
 // ============================================================================

packages/agent-core/test/harness/compaction-estimate.test.ts

@@ -0,0 +1,77 @@
+import type { AssistantMessage } from "@step-harness/providers";
+import { describe, expect, it } from "vitest";
+import { estimateTokens } from "../../src/harness/compaction/compaction.ts";
+import { estimateMessageTokens } from "../../src/harness/compaction/projection-content.ts";
+
+const han = "构建失败：解析器在第四十二行抛出断言错误";
+
+function assistant(content: AssistantMessage["content"]): AssistantMessage {
+	return {
+		role: "assistant",
+		content,
+		api: "faux",
+		provider: "faux",
+		model: "faux-1",
+		usage: {
+			input: 0,
+			output: 0,
+			cacheRead: 0,
+			cacheWrite: 0,
+			totalTokens: 0,
+			cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
+		},
+		stopReason: "stop",
+		timestamp: 1,
+	};
+}
+
+describe("harness token estimate", () => {
+	it("keeps ASCII messages at four characters per token", () => {
+		expect(estimateTokens({ role: "user", content: "a".repeat(400), timestamp: 1 })).toBe(100);
+	});
+
+	it("prices CJK text by UTF-8 bytes", () => {
+		const text = han.repeat(10);
+		expect(estimateTokens({ role: "user", content: text, timestamp: 1 })).toBe(Math.ceil((text.length * 3) / 4));
+	});
+
+	it("prices assistant text, thinking and tool arguments by bytes", () => {
+		const message = assistant([
+			{ type: "text", text: han },
+			{ type: "thinking", thinking: han },
+			{ type: "toolCall", id: "1", name: "run", arguments: { command: han } },
+		]);
+		expect(estimateTokens(message)).toBeGreaterThan(Math.ceil((han.length * 3 * 3) / 4) - 1);
+	});
+
+	it("keeps the fixed image estimate", () => {
+		expect(
+			estimateTokens({
+				role: "user",
+				content: [{ type: "image", mimeType: "image/png", data: "x" }],
+				timestamp: 1,
+			}),
+		).toBe(1200);
+	});
+
+	it("prices bash output by bytes", () => {
+		expect(
+			estimateTokens({
+				role: "bashExecution",
+				command: "cat",
+				output: han,
+				exitCode: 0,
+				cancelled: false,
+				truncated: false,
+				timestamp: 1,
+			}),
+		).toBe(Math.ceil(3 / 4) + Math.ceil((han.length * 3) / 4));
+	});
+
+	it("uses the same byte pricing in the lightweight projection", () => {
+		const text = han.repeat(10);
+		expect(estimateMessageTokens({ role: "user", content: text, timestamp: 1 })).toBe(
+			Math.ceil((text.length * 3) / 4),
+		);
+	});
+});