diff --git a/docs/spec/03-runtime/02-agent-runtime.md b/docs/spec/03-runtime/02-agent-runtime.md index 55d5c9924f..533931cad4 100644 --- a/docs/spec/03-runtime/02-agent-runtime.md +++ b/docs/spec/03-runtime/02-agent-runtime.md @@ -481,6 +481,26 @@ boundary falls, not what survives it. The active-user retention limit is 20,000 tokens, capped at half the hard budget so retention alone cannot fill a small window and leave the summary no room. None of these values are configurable. +**Estimate calibration (D606).** Every threshold above is compared against one +number, and that number is corrected against what requests actually cost. pi's +`estimateContextTokens` anchors on the last assistant usage and estimates +everything after it as `chars / 4`: that constant under-counts CJK text, and +with no anchor left it omits the system prompt and the tool schemas, which the +next request still pays for. The two errors are measured and applied +separately — the per-character bias as a scale-free ratio over the guessed +tail, and an unanchored residual as a ratio only for observations taken at a +comparable scale (0.5×–2× of the estimate), otherwise as the observed overhead +capped at 32,000 tokens. + +The correction is asymmetric because this number gates compaction: upward +applies once three observations exist, downward needs three agreeing samples, +is capped at 15 % per step and can never take the value below 85 % of the raw +estimate, so a projection at 1.18× the hard limit (`1 / 0.85`) still compacts. +A report outside 0.5×–3× of what the calibration predicted is treated as a +misreport, and two consecutive misreports freeze the downward direction until a +usable report arrives. + + The provider request layer also caps the concrete output budget before every parent, subagent, and one-shot request. It estimates the serialized input with the pi-ai chars/4 baseline plus a CJK correction, then reserves the larger of diff --git a/docs/spec/06-delivery/04-e2e-test-plan.md b/docs/spec/06-delivery/04-e2e-test-plan.md index c162253cba..8802f17a22 100644 --- a/docs/spec/06-delivery/04-e2e-test-plan.md +++ b/docs/spec/06-delivery/04-e2e-test-plan.md @@ -8002,6 +8002,7 @@ identify the platform validation still needed. | C — Conversation & stream (delegate context budget) | E2E-SUBAGENT-context-overflow-compacts-before-failing, E2E-SUBAGENT-context-overflow-reports-actionable-failure, E2E-SUBAGENT-resume-seeds-within-context-budget | | Quality (delegate context budget) | E2E-SUBAGENT-context-overflow-compacts-before-failing, E2E-SUBAGENT-context-overflow-reports-actionable-failure, E2E-SUBAGENT-resume-seeds-within-context-budget | | M6+ (delegate context budget) | E2E-SUBAGENT-context-overflow-compacts-before-failing, E2E-SUBAGENT-context-overflow-reports-actionable-failure, E2E-SUBAGENT-resume-seeds-within-context-budget | +| C / F / Quality — The context estimate stays safe (calibration) | E2E-CONTEXT-estimate-calibration-stays-safe | | C — Conversation & stream (unique tool-call ids) | E2E-RUNTIME-unique-tool-call-ids-per-request | | Quality (unique tool-call ids) | E2E-RUNTIME-unique-tool-call-ids-per-request | | G — Plugin host lifecycle (crash report) | E2E-PLUGIN-crash-report-names-the-exit-code | @@ -14161,6 +14162,27 @@ the latest destination. These assertions measure work counts, not device FPS. success fixture uses a child-only extra CA; it does not reproduce a specific antivirus installation or claim native macOS/Linux verification. +### E2E-CONTEXT-estimate-calibration-stays-safe + +- **Preconditions:** Deterministic provider fixture whose reported usage can be + scripted; a session whose occupancy is close to the hard boundary; no real + provider credentials. +- **Steps:** Report usage below the prediction repeatedly and confirm the + displayed occupancy never falls below the floor and compaction still fires; + report usage far outside the plausible band and confirm the value does not + move; run a CJK-heavy session and compare the displayed occupancy with the + reported usage. +- **Expected:** The calibrated occupancy stays within 0.85×–6× of the raw + estimate, moves up as soon as there is evidence, moves down only after + agreeing samples, and a projection at 1.18× the hard limit still compacts. + Misreports move it in neither direction. CJK text no longer reads at a + quarter of its measured cost. +- **Specs linked:** `03-runtime/02-agent-runtime.md` §5.1, + `08-meta/decisions-log.md` D606 +- **Acceptance:** C (chat and stream), F (persistence), Quality +- **Status:** Unit-covered (`context-calibration.test.ts`, including the CJK and + boundary cases); desktop E2E pending + ### E2E-RUNTIME-unique-tool-call-ids-per-request - **Preconditions:** A session transcript that carries the same tool call twice diff --git a/docs/spec/08-meta/decisions-log.md b/docs/spec/08-meta/decisions-log.md index eb22d89ebb..cd2031143f 100644 --- a/docs/spec/08-meta/decisions-log.md +++ b/docs/spec/08-meta/decisions-log.md @@ -6476,6 +6476,25 @@ that was sitting at the bottom — including after the turn had finished. state, protocol, persistence, theme schema, or permission change. See `04-ux/08-component-spec.md` and E2E-CHAT-opaque-floating-decision-and-retry-surfaces. +## 2026-09-21 — The context estimate is calibrated conservatively (D606) + +- Every budget threshold reads one number, and that number is corrected against + what requests actually cost. pi's estimator anchors on the last assistant + usage and estimates the rest as `chars / 4`, which under-counts CJK text by + roughly a factor of 2–4 and, with no anchor left, omits the system prompt and + tool schemas entirely. +- The two errors are applied separately: the per-character bias as a + scale-free ratio over the guessed tail, and an unanchored residual as a ratio + only for observations taken at a comparable scale (0.5×–2×), otherwise as the + observed overhead capped at 32,000 tokens. A 100k-scale sample therefore + cannot be applied as a ratio to a 1M projection. +- The correction is asymmetric: upward applies once three observations exist; + downward needs three agreeing samples, is capped at 15 % per step, and can + never take the value below 85 % of the raw estimate, so a projection at + 1.18× the hard limit still compacts. A report outside 0.5×–3× of what the + calibration predicted is a misreport, and two consecutive misreports freeze + the downward direction. See `03-runtime/02-agent-runtime.md` §5.1. + ## 2026-09-21 — A tool-call id is unique in every request (D608, issue #718) - Anthropic-family endpoints, DeepSeek's included, reject a whole turn with diff --git a/docs/zh-CN/spec/03-runtime/02-agent-runtime.md b/docs/zh-CN/spec/03-runtime/02-agent-runtime.md index a5e730765d..9e03f3076a 100644 --- a/docs/zh-CN/spec/03-runtime/02-agent-runtime.md +++ b/docs/zh-CN/spec/03-runtime/02-agent-runtime.md @@ -377,6 +377,16 @@ Headroom 是 16,384 个代币储备底线的最大值,模型最大输出 小窗口,没有留下摘要的空间。这些值都不是 可配置。 +**估算校准(D606)。** 上述每个阈值都对着同一个数字比较,而该数字会按请求的真实开销校正。pi 的 `estimateContextTokens` +以最后一条助手用量为锚,其余一律按 `chars / 4` 估算:该常数会低估中文文本,且在没有锚点时完全不含系统提示与工具结构, +而下一次请求仍要为它们付费。两类误差分开处理——逐字符偏差以「猜测尾部」上的比例表示(与量级无关);无锚点残差只在与之 +量级相当的样本(0.5×–2×)上按比例应用,否则只加上观测到的固定开销(上限 32,000 词元)。 + +校正是不对称的,因为这个数字决定压缩:向上修正只要有三个观测即生效;向下修正需要三个方向一致的样本、每次最多 15 %, +且永远不能把数值压到原始估算的 85 % 以下——因此处在硬限制 1.18×(`1 / 0.85`)的投影仍会触发压缩。偏离校准预测 +0.5×–3× 之外的报告视为误报;连续两次误报会冻结向下修正,直到出现可用报告。 + + 传入的用户提示先于第一个提供商参与预算 请求。自动摘要请求在有界的 pi-ai 重试策略下重试瞬时的提供商失败 (3 次重试,2s/4s/8s 退避,Stop 可取消);配额、鉴权等确定性失败立即返回。 diff --git a/docs/zh-CN/spec/06-delivery/04-e2e-test-plan.md b/docs/zh-CN/spec/06-delivery/04-e2e-test-plan.md index 2b3a9b4f29..b736bd847f 100644 --- a/docs/zh-CN/spec/06-delivery/04-e2e-test-plan.md +++ b/docs/zh-CN/spec/06-delivery/04-e2e-test-plan.md @@ -5288,6 +5288,7 @@ eleven-tool-round desktop paths are verified by | C — 对话和直播(委托上下文预算) | E2E-SUBAGENT-context-overflow-compacts-before-failing、E2E-SUBAGENT-context-overflow-reports-actionable-failure、E2E-SUBAGENT-resume-seeds-within-context-budget | | 品质(委托上下文预算) | E2E-SUBAGENT-context-overflow-compacts-before-failing、E2E-SUBAGENT-context-overflow-reports-actionable-failure、E2E-SUBAGENT-resume-seeds-within-context-budget | | M6+(委托上下文预算) | E2E-SUBAGENT-context-overflow-compacts-before-failing、E2E-SUBAGENT-context-overflow-reports-actionable-failure、E2E-SUBAGENT-resume-seeds-within-context-budget | +| C / F / 品质 —— 上下文估算保持安全(校准) | E2E-CONTEXT-estimate-calibration-stays-safe | | C — 对话与流式(工具调用 id 唯一) | E2E-RUNTIME-unique-tool-call-ids-per-request | | 品质(工具调用 id 唯一) | E2E-RUNTIME-unique-tool-call-ids-per-request | | G — 插件宿主生命周期(崩溃上报) | E2E-PLUGIN-crash-report-names-the-exit-code | @@ -8341,6 +8342,16 @@ the latest destination. These assertions measure work counts, not device FPS. - **规格:** 04-ux/06-settings-ia、04-ux/08-component-spec、 04-ux/09-interaction-patterns;ADR turn-process-and-thinking-display。 +### E2E-CONTEXT-estimate-calibration-stays-safe + +- **先决条件:** 可脚本化上报用量的确定性提供商夹具;占用接近硬边界的会话;不使用真实凭据。 +- **步骤:** 连续上报低于预测的用量,确认显示占用不低于下限且压缩仍然触发;上报远离合理区间的用量,确认数值不动;跑一个 + 中文为主的会话,比较显示占用与上报用量。 +- **预期:** 校准后的占用保持在原始估算的 0.85×–6× 之内;有证据即上调;只有方向一致的样本才下调;处在硬限制 1.18× 的 + 投影仍触发压缩;误报不产生任何方向的移动;中文文本不再只有实测成本的四分之一。 +- **规格:** 03-runtime/02-agent-runtime §5.1、08-meta/decisions-log D606。**验收:** C(对话与流)、F(持久化)、品质。 + **状态:** 单元测试覆盖(含中文与边界用例);桌面 E2E 待补。 + ### E2E-RUNTIME-unique-tool-call-ids-per-request - **先决条件:** 一份把同一次工具调用携带两次的会话转录(重试追加让该调用落在第二个行 id 上),在确定性提供商夹具下加载进 diff --git a/docs/zh-CN/spec/08-meta/decisions-log.md b/docs/zh-CN/spec/08-meta/decisions-log.md index ceaff29b5f..63cffb59f2 100644 --- a/docs/zh-CN/spec/08-meta/decisions-log.md +++ b/docs/zh-CN/spec/08-meta/decisions-log.md @@ -4599,6 +4599,16 @@ that amendment are retired by ADR 0268; the upstream work-panel lifecycle stays. 或权限。见 `04-ux/08-component-spec.md` 与 E2E-CHAT-opaque-floating-decision-and-retry-surfaces。 +## 2026-09-21 —— 上下文估算采用保守校准(D606) + +- 所有预算阈值读取同一个数字,该数字按请求的真实开销校正。pi 的估算器以最后一条助手用量为锚、其余按 `chars / 4` 估算, + 这会低估中文文本约 2–4 倍;没有锚点时还会完全漏掉系统提示与工具结构。 +- 两类误差分开应用:逐字符偏差是按比例(与量级无关)作用于猜测尾部;无锚点残差只在量级相当的样本(0.5×–2×)上按比例 + 应用,否则只加观测到的固定开销(上限 32,000 词元)。因此 100k 量级的样本不会作为比例系数套到 1M 的投影上。 +- 校正不对称:向上修正有三个观测即生效;向下修正需要三个方向一致的样本、每次最多 15 %,且不会低于原始估算的 85 %—— + 处在硬限制 1.18× 的投影仍会压缩。偏离预测 0.5×–3× 的报告视为误报,连续两次误报冻结向下修正。 + 见 `03-runtime/02-agent-runtime §5.1`。 + ## 2026-09-21 —— 每个请求里的工具调用 id 必须唯一(D608,issue #718) - Anthropic 系端点(含 DeepSeek)在请求中同一个调用 id 出现两次时,会以 `tool_use ids must be unique` 拒绝整个回合, @@ -4607,7 +4617,7 @@ that amendment are retired by ADR 0268; the upstream work-panel lifecycle stays. - 因此上线前的最后一个视图对每个 `toolCall` id 只保留第一次出现,丢弃其后重复的调用或结果,使提供商校验的「一调用一结果」 配对保持完整;没有重复的请求原样返回(返回同一对象,而非副本)。磁盘上的内容不会被改写,压缩与保留规则也不变。 - 一旦发生丢弃,会在 `agent` 日志通道上报告一次,带上会话与 id,使下一次同类报障能指向写入方而不只是提供商那句话。 - 见 `03-runtime/02-agent-runtime.md` §5。 + 见 `03-runtime/02-agent-runtime §5`。 - 守卫刻意放在请求边界而不是历史重建处:这样也能覆盖**会话运行期间**产生的重复,而重建期的过滤看不到它。 ## 2026-09-21 —— 插件崩溃上报带上退出码但不复制原始输出(D607,issue #747) diff --git a/packages/agent-runtime/src/context-calibration.test.ts b/packages/agent-runtime/src/context-calibration.test.ts new file mode 100644 index 0000000000..34abca8e48 --- /dev/null +++ b/packages/agent-runtime/src/context-calibration.test.ts @@ -0,0 +1,244 @@ +import { describe, expect, it } from "vitest"; +import { + CONTEXT_CALIBRATION_ANOMALY_FREEZE, + CONTEXT_CALIBRATION_BOUNDARY_GUARD_RATIO, + CONTEXT_CALIBRATION_DOWNWARD_CONSECUTIVE, + CONTEXT_CALIBRATION_FACTOR_MIN, + CONTEXT_CALIBRATION_FACTOR_MAX, + CONTEXT_CALIBRATION_MIN_SAMPLES, + CONTEXT_CALIBRATION_OVERHEAD_CAP, + CONTEXT_CALIBRATION_RATIO_MAX, + CONTEXT_CALIBRATION_WINDOW, + ContextEstimateCalibration, + type ContextCalibration, +} from "./context-calibration.js"; + +/** A projection with no usage left: the estimator guessed at every message. */ +function unanchored(tokens: number): ContextCalibration { + return { + tokens, + usageTokens: 0, + trailingTokens: tokens, + lastUsageIndex: null, + }; +} + +/** A projection anchored on a real usage: only the tail was guessed at. */ +function anchored( + usageTokens: number, + trailingTokens: number, +): ContextCalibration { + return { + tokens: usageTokens + trailingTokens, + usageTokens, + trailingTokens, + lastUsageIndex: 4, + }; +} + +/** One CJK paragraph, so the character count is real text and not a constant. */ +const CJK_PARAGRAPH = + "上下文估计的关键在于尾部:估算器只对最后一条用量之后的消息按字符数除以四来猜," + + "而中文正文的真实开销接近每字一个词元。连续低报时不能把估计值一路压到危险区间," + + "否则压缩会被推迟,请求就会在临界点上失败。无锚点时必须把系统提示与工具结构的固定开销" + + "和按比例的中文字符偏差分开处理,跨量级的样本不能当作比例系数直接套用。"; + +describe("ContextEstimateCalibration", () => { + it("returns the raw estimate until a series has enough observations", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES - 1; i++) { + calibration.recordUnanchored(100_000, 300_000); + calibration.recordAnchored(200_000, 20_000, 260_000); + } + expect(calibration.correct(unanchored(100_000))).toBe(100_000); + expect(calibration.correct(anchored(200_000, 20_000))).toBe(220_000); + }); + + it("scales only the guessed tail when an anchor exists", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + calibration.recordAnchored(200_000, 20_000, 260_000); + } + expect(calibration.trailingRatio()).toBe(3); + expect(calibration.correct(anchored(200_000, 10_000))).toBe(230_000); + expect(calibration.correct(anchored(200_000, 20_000))).toBe(260_000); + }); + + // ① Conservative handling of anomalous usage. + describe("under-reporting", () => { + it("cannot be carried below the safety floor by consecutive low reports", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_WINDOW; i++) { + calibration.recordUnanchored(1_000_000, 600_000); + } + const corrected = calibration.correct(unanchored(1_000_000)); + expect(corrected).toBeGreaterThanOrEqual( + Math.round(1_000_000 * CONTEXT_CALIBRATION_FACTOR_MIN), + ); + expect(corrected).toBeLessThanOrEqual(1_000_000); + }); + + it("does not move down at all before enough agreeing samples", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_DOWNWARD_CONSECUTIVE - 1; i++) { + calibration.recordUnanchored(1_000_000, 600_000); + } + expect(calibration.correct(unanchored(1_000_000))).toBe(1_000_000); + calibration.recordUnanchored(1_000_000, 600_000); + expect(calibration.correct(unanchored(1_000_000))).toBeLessThan(1_000_000); + }); + + it("moves up as soon as there is evidence", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + calibration.recordUnanchored(1_000_000, 1_400_000); + } + expect(calibration.correct(unanchored(1_000_000))).toBe(1_400_000); + }); + + it("freezes the downward direction after consecutive misreports", () => { + const calibration = new ContextEstimateCalibration(); + // A corroborated downward streak first, so the freeze is what stops it. + for (let i = 0; i < CONTEXT_CALIBRATION_DOWNWARD_CONSECUTIVE; i++) { + calibration.recordUnanchored(1_000_000, 700_000); + } + expect(calibration.correct(unanchored(1_000_000))).toBeLessThan(1_000_000); + for (let i = 0; i < CONTEXT_CALIBRATION_ANOMALY_FREEZE; i++) { + calibration.recordUnanchored(1_000_000, 200_000); + } + expect(calibration.frozen()).toBe(true); + expect(calibration.correct(unanchored(1_000_000))).toBe(1_000_000); + // A usable report unfreezes it, and an unusable one never counts as + // evidence in the other direction either. + calibration.recordUnanchored(1_000_000, 1_000_000); + expect(calibration.frozen()).toBe(false); + }); + }); + + // ② Ratio versus fixed offset: the same residual, used where it belongs. + describe("scale", () => { + it("applies a ratio only to an estimate of a comparable scale", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + calibration.recordUnanchored(100_000, 130_000); + } + // A 100k sample says nothing proportional about a 1M projection: only + // the fixed part of the residual is added. + expect(calibration.unanchoredRatio(1_000_000)).toBeNull(); + expect(calibration.overheadTokens()).toBe(30_000); + expect(calibration.correct(unanchored(1_000_000))).toBe(1_030_000); + + // Once the projection is observed at its own scale, the ratio applies. + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + calibration.recordUnanchored(1_000_000, 1_300_000); + } + expect(calibration.unanchoredRatio(1_000_000)).toBeCloseTo(1.3, 5); + expect(calibration.correct(unanchored(1_000_000))).toBe(1_300_000); + }); + + it("caps the fixed part so it stays an overhead", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + // A 40k session with a 300k report is inside the sanity band but its + // absolute residual is far past a system prompt plus schemas. + calibration.recordUnanchored(40_000, 300_000); + } + expect(calibration.overheadTokens()).toBe( + CONTEXT_CALIBRATION_OVERHEAD_CAP, + ); + }); + + it("keeps a small-context sample from under-correcting a large one forever", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + calibration.recordUnanchored(100_000, 130_000); + } + const large = 1_000_000; + const first = calibration.correct(unanchored(large)); + // Observed at scale, the correction becomes proportional instead of + // staying at the fixed 30k it measured on the small session. + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + calibration.recordUnanchored(large, 1_400_000); + } + const second = calibration.correct(unanchored(large)); + expect(first).toBe(1_030_000); + expect(second).toBe(1_400_000); + expect(second).toBeGreaterThan(first); + }); + }); + + // ③ Real CJK text: the chars/4 tail is the error this series exists for. + it("corrects a real CJK tail to the measured cost", () => { + const calibration = new ContextEstimateCalibration(); + const cjkChars = [...CJK_PARAGRAPH].length; + const estimatedTail = Math.ceil(cjkChars / 4); + // Measured cost of CJK text in the tokenizers this app talks to: between + // roughly 0.6 and 1.0 tokens per character. + const measuredTail = Math.round(cjkChars * 0.9); + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + calibration.recordAnchored(10_000, estimatedTail, 10_000 + measuredTail); + } + const ratio = calibration.trailingRatio(); + expect(ratio).toBeGreaterThanOrEqual(3); + expect(ratio).toBeLessThanOrEqual(CONTEXT_CALIBRATION_RATIO_MAX); + const correctedTail = + calibration.correct(anchored(10_000, estimatedTail)) - 10_000; + expect(correctedTail).toBeGreaterThanOrEqual(measuredTail * 0.95); + expect(correctedTail).toBeLessThanOrEqual(measuredTail * 1.05); + }); + + // ④ The compaction boundary: a correction may not carry a session past it. + describe("compaction boundary", () => { + it("still compacts a projection inside the guard ratio of the limit", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_WINDOW; i++) { + // The worst case for the boundary: every report says the estimate + // reads high, at the projection's own scale. + calibration.recordUnanchored(950_000, 600_000); + } + const hardLimit = 950_000; + const guarded = Math.round( + hardLimit * CONTEXT_CALIBRATION_BOUNDARY_GUARD_RATIO, + ); + expect(calibration.correct(unanchored(guarded))).toBeGreaterThanOrEqual( + hardLimit, + ); + }); + + it("bounds how far a correction may defer the boundary", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_WINDOW; i++) { + calibration.recordUnanchored(950_000, 400_000); + } + const hardLimit = 950_000; + const corrected = calibration.correct(unanchored(hardLimit)); + const deficit = hardLimit - corrected; + expect(deficit).toBeGreaterThanOrEqual(0); + expect(deficit).toBeLessThanOrEqual( + Math.round(hardLimit * (1 - CONTEXT_CALIBRATION_FACTOR_MIN)), + ); + }); + + + it("caps the correction at the upper band, and rejects a wild report", () => { + const calibration = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_MIN_SAMPLES; i++) { + calibration.recordUnanchored(1_000_000, 2_900_000); + } + const corrected = calibration.correct(unanchored(1_000_000)); + expect(corrected).toBe(2_900_000); + expect(corrected).toBeLessThanOrEqual( + 1_000_000 * CONTEXT_CALIBRATION_FACTOR_MAX, + ); + + // Four times what the calibration predicts is a misreport, not a + // measurement: it moves the value in neither direction. + const wild = new ContextEstimateCalibration(); + for (let i = 0; i < CONTEXT_CALIBRATION_ANOMALY_FREEZE; i++) { + wild.recordUnanchored(1_000_000, 4_000_000); + } + expect(wild.frozen()).toBe(true); + expect(wild.correct(unanchored(1_000_000))).toBe(1_000_000); + }); + }); +}); diff --git a/packages/agent-runtime/src/context-calibration.ts b/packages/agent-runtime/src/context-calibration.ts new file mode 100644 index 0000000000..d9404c6081 --- /dev/null +++ b/packages/agent-runtime/src/context-calibration.ts @@ -0,0 +1,430 @@ +/** + * Context-estimate calibration. + * + * Every budget decision in this runtime reads one number: the context estimate + * (the automatic compaction check, the idle pre-compaction, and the occupancy + * the desktop shows). That number comes from pi's `estimateContextTokens`, + * whose shape is: + * + * last assistant usage (real, when the transcript still carries one) + * + ceil(chars / 4) over every message after it + * + * Two systematic errors follow from that shape: + * + * 1. `chars / 4` is an English-prose constant. Applied to CJK text it + * under-counts by roughly a factor of two to four, and CJK is the bulk of + * what some sessions send. ASCII fixtures never show it. + * 2. A projection with no assistant usage left — right after a compaction, or + * at the start of a session — is estimated end to end, so the system + * prompt and the tool schemas disappear from the number even though the + * next request still pays for them. + * + * The two errors are different quantities, so they are measured and applied + * separately: + * + * - **trailing ratio** — while an anchor exists the estimator is exact up to + * the messages appended after it. `real - usage` is the true size of those + * messages, and dividing it by the estimator's own trailing number is a + * pure measurement of the per-character bias. It is a ratio, so it is + * scale-free: the same correction applies whether the tail is 2k or 200k + * tokens. + * - **unanchored residual** — with no anchor, `real - estimate` carries the + * missing system/tool overhead *and* the same per-character bias. Those two + * are split here rather than added together: a residual ratio is applied + * only when the observation was taken at a comparable scale + * (`CONTEXT_CALIBRATION_SCALE_BAND_*` of the estimate being corrected), + * and otherwise only the fixed part — the observed overhead, capped by + * `CONTEXT_CALIBRATION_OVERHEAD_CAP` — is added. A sample taken in a 100k + * session therefore cannot be applied as a ratio to a 1M projection. + * + * Safety properties, because this number gates compaction: + * + * - **Asymmetric.** Upward correction applies as soon as there is evidence; + * downward correction needs `CONTEXT_CALIBRATION_DOWNWARD_CONSECUTIVE` + * agreeing samples, is capped at `CONTEXT_CALIBRATION_DOWNWARD_STEP_MAX` + * per step, and can never take the corrected value below + * `CONTEXT_CALIBRATION_FACTOR_MIN` of the raw estimate. A projection that + * reads at or above `CONTEXT_CALIBRATION_BOUNDARY_GUARD_RATIO` of the hard + * limit therefore still reaches it after correction — a session cannot be + * carried past the boundary by its own corrections. + * - **Sanity-checked.** A provider report outside + * `CONTEXT_CALIBRATION_SAMPLE_*_RATIO` of what the calibration predicted is + * treated as a misreport: it is discarded and counted, and + * `CONTEXT_CALIBRATION_ANOMALY_FREEZE` consecutive ones freeze the downward + * direction until a usable report arrives. + * - **Bounded windows and medians.** Each series is the median of its last + * `CONTEXT_CALIBRATION_WINDOW` samples, so one wild report cannot move it. + * + * What this file is not + * --------------------- + * It does not estimate tokens, talk to a provider, or persist anything across + * sessions. The runtime owns the observations; this owns the arithmetic. The + * `view()` snapshot is what a caller can log, so a surprising occupancy can be + * explained after the fact instead of guessed at. + */ + +/** Observations needed before a series is trusted; below this the raw value stands. */ +export const CONTEXT_CALIBRATION_MIN_SAMPLES = 3; + +/** Bounded window per series: the median of these, so one outlier cannot move it. */ +export const CONTEXT_CALIBRATION_WINDOW = 16; + +/** Per-character bias may not be calibrated outside this range. Scale-free. */ +export const CONTEXT_CALIBRATION_RATIO_MIN = 0.75; +export const CONTEXT_CALIBRATION_RATIO_MAX = 4; + +/** + * The corrected value may not leave this band around the raw estimate. The + * lower bound is the dangerous direction — a number that reads low postpones + * compaction — so it is a hard floor that no combination of samples can pass, + * while the upper bound stays generous: an unanchored projection can + * legitimately be missing a whole system prompt's worth of overhead. + */ +export const CONTEXT_CALIBRATION_FACTOR_MIN = 0.85; +export const CONTEXT_CALIBRATION_FACTOR_MAX = 6; + +/** A projection at or above this share of the hard limit still compacts. */ +export const CONTEXT_CALIBRATION_BOUNDARY_GUARD_RATIO = + 1 / CONTEXT_CALIBRATION_FACTOR_MIN; + +/** + * A provider report is only usable as a measurement of the *request*. Outside + * this band around what the calibration predicted, it is a misreport (a cached + * call that never carried the context, a summary request, a gateway + * substituting a number) and it is discarded rather than folded in. + */ +export const CONTEXT_CALIBRATION_SAMPLE_MIN_RATIO = 0.5; +export const CONTEXT_CALIBRATION_SAMPLE_MAX_RATIO = 3; + +/** Consecutive misreports that freeze the downward direction. */ +export const CONTEXT_CALIBRATION_ANOMALY_FREEZE = 2; + +/** Agreeing samples required before the corrected value may move down at all. */ +export const CONTEXT_CALIBRATION_DOWNWARD_CONSECUTIVE = 3; + +/** Most one downward step may take off the raw estimate. */ +export const CONTEXT_CALIBRATION_DOWNWARD_STEP_MAX = 0.15; + +/** How far a sample's scale may sit from the estimate before it stops being a ratio. */ +export const CONTEXT_CALIBRATION_SCALE_BAND_MIN = 0.5; +export const CONTEXT_CALIBRATION_SCALE_BAND_MAX = 2; + +/** The fixed part of an unanchored residual may not exceed this many tokens. */ +export const CONTEXT_CALIBRATION_OVERHEAD_CAP = 32_000; + +/** A report this many times the request is a misreport, never a measurement. */ +const CONTEXT_CALIBRATION_REAL_MAX_RATIO = 20; + +/** The displayed band ignores residuals outside this range. */ +const CONTEXT_CALIBRATION_RESIDUAL_MIN = 0.25; +const CONTEXT_CALIBRATION_RESIDUAL_MAX = 4; + +/** A sample counts as agreeing with a downward move when it reads this much high. */ +const CONTEXT_CALIBRATION_DOWNWARD_EVIDENCE = 0.02; + +export type ContextCalibration = { + /** Real tokens the estimator itself anchors on (the last assistant usage). */ + usageTokens: number; + /** Estimated tokens for the messages after that anchor. */ + trailingTokens: number; + /** Index of the anchor, or null when the projection carries no usage at all. */ + lastUsageIndex: number | null; + /** The estimator's own total: anchor + trailing. Structurally this is the + * object `estimateContextTokens` returns, so callers pass it straight in. */ + tokens: number; +}; + +export type CalibrationView = { + anchoredSamples: number; + trailingRatio: number; + unanchoredSamples: number; + /** Samples whose scale allowed the ratio form to be used. */ + unanchoredScaleSamples: number; + unanchoredRatio: number; + overheadTokens: number; + residualSamples: number; + residualLowRatio: number; + residualHighRatio: number; + /** Consecutive misreports currently seen. */ + anomalies: number; + /** True while misreports keep the downward direction frozen. */ + frozen: boolean; + /** Consecutive samples that said the estimate reads high. */ + downwardStreak: number; + rawTokens: number; + correctedTokens: number; +}; + +/** + * The measured accuracy of the estimate, as the spread of real residuals. A + * ratio of 1 means the prediction was exact; `lowRatio` below 1 means the + * estimate read low (the dangerous direction) as recently as the window shows. + */ +export type ContextCalibrationBand = { + samples: number; + lowRatio: number; + highRatio: number; +}; + +/** Median of a window; the window is never empty when this is called. */ +function median(values: readonly number[]): number { + const sorted = [...values].sort((a, b) => a - b); + const middle = Math.floor(sorted.length / 2); + return sorted.length % 2 === 1 + ? sorted[middle] + : (sorted[middle - 1] + sorted[middle]) / 2; +} + +function usableNumber(value: number): boolean { + return Number.isFinite(value) && value > 0; +} + +function clamp(value: number, min: number, max: number): number { + return Math.min(max, Math.max(min, value)); +} + +type UnanchoredSample = { estimate: number; realTotal: number }; + +/** + * Two rolling series, one per error above. Kept apart on purpose: a session + * that has been compacted has both, and mixing them would apply the system + * overhead twice on an anchored projection. + */ +export class ContextEstimateCalibration { + private readonly trailingRatios: number[] = []; + private readonly unanchored: UnanchoredSample[] = []; + /** `real / predicted` per usable observation; the displayed band. */ + private readonly residuals: number[] = []; + /** Consecutive misreports. */ + private anomalies = 0; + /** Consecutive samples that said the estimate reads high. */ + private downwardStreak = 0; + /** The last value `correct()` returned, for the per-step cap. */ + private lastCorrected: number | null = null; + + /** + * Record one request whose estimate was anchored on a known usage. + * + * `trueTrailing` is derived by the caller as `real − usageTokens`: the anchor + * is the estimator's own number for the prefix, so the difference is the + * actual size of the messages the estimator had to guess at. + */ + recordAnchored( + usageTokens: number, + estimatedTrailing: number, + realTotal: number, + ): void { + if (!usableNumber(realTotal) || !usableNumber(usageTokens)) return; + if (realTotal > usageTokens * CONTEXT_CALIBRATION_REAL_MAX_RATIO) return; + const trueTrailing = realTotal - usageTokens; + // A projection with nothing after the anchor carries no information about + // the per-character bias; a negative one means the anchor over-counted, + // which is not this series' error to absorb. + if (!usableNumber(trueTrailing) || !usableNumber(estimatedTrailing)) return; + const predicted = usageTokens + estimatedTrailing * this.trailingRatio(); + this.recordResidual(predicted, realTotal); + this.recordDownwardEvidence(predicted, realTotal); + this.push( + this.trailingRatios, + clamp( + trueTrailing / estimatedTrailing, + CONTEXT_CALIBRATION_RATIO_MIN, + CONTEXT_CALIBRATION_RATIO_MAX, + ), + ); + } + + /** + * Record one request that had no anchor at all: the estimate covers every + * message, so the measurement carries the missing system/tool overhead. + */ + recordUnanchored(estimate: number, realTotal: number): void { + if (!usableNumber(estimate) || !usableNumber(realTotal)) return; + if (realTotal > estimate * CONTEXT_CALIBRATION_REAL_MAX_RATIO) return; + const predicted = this.correctUnanchored(estimate); + this.recordResidual(predicted, realTotal); + this.recordDownwardEvidence(predicted, realTotal); + this.push(this.unanchored, { estimate, realTotal }); + } + + /** + * Correct one raw estimate. Returns the raw value until its series has enough + * observations, so a fresh session behaves exactly as it did before. + */ + correct(raw: ContextCalibration): number { + const rawTokens = Math.max(0, Math.round(raw.tokens)); + const anchored = + raw.lastUsageIndex !== null && usableNumber(raw.usageTokens); + const corrected = anchored + ? // The anchored part is the provider's own number for the prefix and + // already contains the system/tool overhead of that request, so only + // the guessed tail is scaled — the unanchored residual must not be + // added here. + raw.usageTokens + Math.max(0, raw.trailingTokens) * this.trailingRatio() + : this.correctUnanchored(rawTokens); + const value = this.clampCorrection(corrected, rawTokens); + this.lastCorrected = value; + return value; + } + + /** The per-character bias currently believed, or 1 before there is evidence. */ + trailingRatio(): number { + if (this.trailingRatios.length < CONTEXT_CALIBRATION_MIN_SAMPLES) return 1; + return median(this.trailingRatios); + } + + /** + * The residual ratio currently believed at the estimate's own scale, or + * `null` when no sample was taken at a comparable scale. Scale-free by + * construction: only in-band samples reach it. + */ + unanchoredRatio(atTokens: number): number | null { + const inBand = this.inBandSamples(atTokens); + if (inBand.length < CONTEXT_CALIBRATION_MIN_SAMPLES) return null; + return median(inBand.map((sample) => sample.realTotal / sample.estimate)); + } + + /** + * The fixed part of the residual — the system/tool overhead a projection + * without an anchor is missing — capped so it stays an overhead, or 0 before + * there is evidence. + */ + overheadTokens(): number { + if (this.unanchored.length < CONTEXT_CALIBRATION_MIN_SAMPLES) return 0; + const overheads = this.unanchored.map((sample) => + Math.max(0, sample.realTotal - sample.estimate), + ); + return Math.min( + CONTEXT_CALIBRATION_OVERHEAD_CAP, + median(overheads), + ); + } + + /** What the last `correct()` did, for logging and for tests. */ + view(rawTokens: number, correctedTokens: number): CalibrationView { + const band = this.band(); + return { + anchoredSamples: this.trailingRatios.length, + trailingRatio: this.trailingRatio(), + unanchoredSamples: this.unanchored.length, + unanchoredScaleSamples: this.inBandSamples(rawTokens).length, + unanchoredRatio: this.unanchoredRatio(rawTokens) ?? 0, + overheadTokens: this.overheadTokens(), + residualSamples: band.samples, + residualLowRatio: band.lowRatio, + residualHighRatio: band.highRatio, + anomalies: this.anomalies, + frozen: this.frozen(), + downwardStreak: this.downwardStreak, + rawTokens, + correctedTokens, + }; + } + + /** + * How far the corrected number has actually been from what the provider + * reported, as a ratio (`real / predicted`) over the bounded window. This is + * the band the estimate can honestly be displayed with: the spread of real + * residuals, not a fabricated ±. + */ + band(): ContextCalibrationBand { + const samples = this.residuals.length; + if (samples === 0) return { samples: 0, lowRatio: 1, highRatio: 1 }; + return { + samples, + lowRatio: Math.min(...this.residuals), + highRatio: Math.max(...this.residuals), + }; + } + + /** True while consecutive misreports keep the downward direction frozen. */ + frozen(): boolean { + return this.anomalies >= CONTEXT_CALIBRATION_ANOMALY_FREEZE; + } + + /** An unanchored correction: ratio in band, fixed overhead otherwise. */ + private correctUnanchored(rawTokens: number): number { + if (rawTokens <= 0) return Math.max(0, rawTokens); + const ratio = this.unanchoredRatio(rawTokens); + if (ratio !== null) return rawTokens * ratio; + const overhead = this.overheadTokens(); + return overhead > 0 ? rawTokens + overhead : rawTokens; + } + + /** Samples taken at a scale comparable to `atTokens`. */ + private inBandSamples(atTokens: number): UnanchoredSample[] { + if (!usableNumber(atTokens)) return []; + return this.unanchored.filter( + (sample) => + sample.estimate >= atTokens * CONTEXT_CALIBRATION_SCALE_BAND_MIN && + sample.estimate <= atTokens * CONTEXT_CALIBRATION_SCALE_BAND_MAX, + ); + } + + private recordResidual(predicted: number, realTotal: number): void { + if (!usableNumber(predicted) || !usableNumber(realTotal)) return; + const ratio = realTotal / predicted; + if (!Number.isFinite(ratio)) return; + // A report this far from the prediction is a misreport. Counting it keeps + // the downward direction frozen until a usable report arrives; discarding + // it keeps the series honest. Anomalies never move the value up either: + // an unusable report is not evidence in either direction. + if ( + ratio < CONTEXT_CALIBRATION_SAMPLE_MIN_RATIO || + ratio > CONTEXT_CALIBRATION_SAMPLE_MAX_RATIO + ) { + this.anomalies += 1; + return; + } + this.anomalies = 0; + if ( + ratio < CONTEXT_CALIBRATION_RESIDUAL_MIN || + ratio > CONTEXT_CALIBRATION_RESIDUAL_MAX + ) { + return; + } + this.push(this.residuals, ratio); + } + + /** Does this sample say the estimate reads high enough to justify a step down? */ + private recordDownwardEvidence(predicted: number, realTotal: number): void { + if (!usableNumber(predicted) || !usableNumber(realTotal)) return; + if (realTotal < predicted * (1 - CONTEXT_CALIBRATION_DOWNWARD_EVIDENCE)) { + this.downwardStreak += 1; + return; + } + this.downwardStreak = 0; + } + + private push(series: T[], value: T): void { + series.push(value); + if (series.length > CONTEXT_CALIBRATION_WINDOW) series.shift(); + } + + private clampCorrection(corrected: number, rawTokens: number): number { + if (!Number.isFinite(corrected)) return rawTokens; + // A raw estimate of zero carries no scale to clamp against; the correction + // is then whatever was measured, never negative. + if (rawTokens <= 0) return Math.max(0, Math.round(corrected)); + const value = clamp( + corrected, + rawTokens * CONTEXT_CALIBRATION_FACTOR_MIN, + rawTokens * CONTEXT_CALIBRATION_FACTOR_MAX, + ); + if (value >= rawTokens) return Math.round(value); + // A downward move needs corroboration and may not jump. + if (!this.downwardAllowed()) return rawTokens; + const stepFloor = + this.lastCorrected === null + ? value + : this.lastCorrected * (1 - CONTEXT_CALIBRATION_DOWNWARD_STEP_MAX); + return Math.round(Math.max(value, stepFloor)); + } + + private downwardAllowed(): boolean { + return ( + !this.frozen() && + this.downwardStreak >= CONTEXT_CALIBRATION_DOWNWARD_CONSECUTIVE + ); + } +} diff --git a/packages/agent-runtime/src/runtime.test.ts b/packages/agent-runtime/src/runtime.test.ts index fc87ab1e8e..cc0346e2bb 100644 --- a/packages/agent-runtime/src/runtime.test.ts +++ b/packages/agent-runtime/src/runtime.test.ts @@ -8617,4 +8617,89 @@ describe("DesktopAgentRuntime hosted web search rounds (ADR 0297)", () => { ]); await restored.dispose(); }); + +describe("context estimate calibration", () => { + /** A request-level usage report: the shape `usageFromPi` hands the runtime. */ + const report = (inputTokens: number) => ({ + inputTokens, + outputTokens: 1, + cacheReadTokens: 0, + cacheWriteTokens: 0, + totalTokens: inputTokens + 1, + }); + + const messages = [ + { role: "user" as const, content: "x".repeat(4_000), timestamp: 1 }, + ]; + + it("moves the gate by what past requests actually cost", async () => { + const runtime = createRuntime(); + const raw = (runtime as any).contextBudget(messages).tokens; + expect(raw).toBeGreaterThan(0); + + // Two unanchored reports, each costing three times the estimate: CJK text + // against the estimator's `chars / 4` constant. Below the sample threshold + // the gate has to stay exactly where it was. + for (let i = 0; i < 2; i++) { + (runtime as any).inFlightContextEstimate = { + tokens: raw, + usageTokens: 0, + trailingTokens: raw, + lastUsageIndex: null, + }; + (runtime as any).recordContextCalibration(report(raw * 3), false); + } + expect((runtime as any).contextBudget(messages).tokens).toBe(raw); + + (runtime as any).inFlightContextEstimate = { + tokens: raw, + usageTokens: 0, + trailingTokens: raw, + lastUsageIndex: null, + }; + (runtime as any).recordContextCalibration(report(raw * 3), false); + expect((runtime as any).contextBudget(messages).tokens).toBe(raw * 3); + + await runtime.dispose(); + }); + + it("ignores a failed attempt, a missing report, and a consumed park", async () => { + const runtime = createRuntime(); + const raw = (runtime as any).contextBudget(messages).tokens; + const parked = { + tokens: raw, + usageTokens: 0, + trailingTokens: raw, + lastUsageIndex: null, + }; + const overhead = () => (runtime as any).contextCalibration.overheadTokens(); + + // Two usable observations, then four that must not count. If any of them + // did, the series would already be trusted and the overhead non-zero. + for (let i = 0; i < 2; i++) { + (runtime as any).inFlightContextEstimate = { ...parked }; + (runtime as any).recordContextCalibration(report(raw * 3), false); + } + (runtime as any).inFlightContextEstimate = { ...parked }; + // A failed stream never carried the request... + (runtime as any).recordContextCalibration(report(raw * 3), true); + // ...and it still consumed the park, so this report has nothing to pair + // with rather than reusing the failed attempt's estimate. + (runtime as any).recordContextCalibration(report(raw * 3), false); + // A settled attempt whose response carried no usage at all. + (runtime as any).inFlightContextEstimate = { ...parked }; + (runtime as any).recordContextCalibration(undefined, false); + // A report with nothing parked. + (runtime as any).recordContextCalibration(report(raw * 10), false); + expect(overhead()).toBe(0); + + // The third usable observation crosses the threshold: the two above plus + // this one, each measuring `3 * raw - raw`. + (runtime as any).inFlightContextEstimate = { ...parked }; + (runtime as any).recordContextCalibration(report(raw * 3), false); + expect(overhead()).toBe(raw * 2); + + await runtime.dispose(); + }); +}); }); diff --git a/packages/agent-runtime/src/runtime.ts b/packages/agent-runtime/src/runtime.ts index 9d05eaef44..31decad4fe 100644 --- a/packages/agent-runtime/src/runtime.ts +++ b/packages/agent-runtime/src/runtime.ts @@ -9,6 +9,7 @@ import { BACKGROUND_CONTEXT, compact, convertToLlm, + estimateContextTokens, estimateTokens, prepareCompaction, withAbortSignal, @@ -218,6 +219,11 @@ import { withStreamIdleTimeout, } from "./provider-retry.js"; +import { + ContextEstimateCalibration, + type ContextCalibration, +} from "./context-calibration.js"; + import { rebuildNodeNetworkTransport } from "./node-proxy.js"; import { createProviderTransportHealth, @@ -1565,6 +1571,13 @@ export class DesktopAgentRuntime { */ private providerRequestBytes?: number; private providerRequestMessages?: number; + /** + * The estimate that describes the provider attempt in flight, parked by + * `streamFn` and consumed when that attempt settles with a usage report. + * Without the pair there is nothing to measure the estimator against. + */ + private inFlightContextEstimate?: ContextCalibration; + private readonly contextCalibration = new ContextEstimateCalibration(); /** * Transport cause of the last provider attempt that rejected before any * response arrived, captured where the original Error still exists. pi-ai @@ -1799,6 +1812,12 @@ Delegation rules: this.providerRetryHeaders = undefined; this.providerRequestBytes = undefined; this.providerRequestMessages = context.messages?.length; + // Park what this request is expected to cost, so the usage report that + // settles it can be measured against it (`contextBudget` corrects the + // same shape). + this.inFlightContextEstimate = estimateContextTokens( + context.messages ?? [], + ); // A new model request starts a new transport streak: the evidence that // justified a rebuild does not carry into the next request (issue #234). this.providerFetchFailure = undefined; @@ -5750,7 +5769,56 @@ Delegation rules: } private contextBudget(messages: AgentMessage[]): ContextBudget { - return contextBudgetFor(this.model, messages); + const budget = contextBudgetFor(this.model, messages); + // Correct the raw estimate with what past requests actually cost. The + // `chars / 4` tail is biased on CJK text, and a projection with no usage + // anchor is missing the system/tool overhead; below the sample threshold + // `correct()` returns the raw value unchanged, and the downward direction + // is bounded by `CONTEXT_CALIBRATION_FACTOR_MIN`. + return { + ...budget, + tokens: this.contextCalibration.correct(estimateContextTokens(messages)), + }; + } + + /** + * Fold one provider report into the estimate calibration. + * + * Only a completed, non-aborted response is a measurement: a failed stream + * never carried the request, and counting one would teach the estimator from + * a request the provider rejected. The pair is recorded against the estimate + * parked by `streamFn`, which describes the same context. + * + * The measured value is the request side only (`input + cacheRead + + * cacheWrite`): the response's own output is not in the projection the parked + * estimate described, and it becomes part of the *next* request's anchor. + */ + private recordContextCalibration( + usage: MessageUsage | undefined, + unusable: boolean, + ): void { + const estimate = this.inFlightContextEstimate; + this.inFlightContextEstimate = undefined; + if (!estimate || unusable || !usage) return; + const realRequestTokens = + usage.inputTokens + + (usage.cacheReadTokens ?? 0) + + (usage.cacheWriteTokens ?? 0); + if (realRequestTokens <= 0) return; + const anchored = + estimate.lastUsageIndex !== null && estimate.usageTokens > 0; + if (anchored) { + this.contextCalibration.recordAnchored( + estimate.usageTokens, + estimate.trailingTokens, + realRequestTokens, + ); + } else { + this.contextCalibration.recordUnanchored( + Math.max(0, Math.round(estimate.tokens)), + realRequestTokens, + ); + } } private automaticCompactionNeeded( @@ -6912,6 +6980,11 @@ Delegation rules: ); } const usage = usageFromPi((event.message as any).usage as Usage | undefined); + // Measure the estimator against this request: the parked estimate + // describes the same context, and only a settled, non-aborted + // response actually carried the request. Consumed either way, so a + // failed attempt cannot pair with a later usage report. + this.recordContextCalibration(usage, failed || aborted); const hostedSearch = hostedSearchFromMessage({ content: (event.message as any).content, citations: (event.message as any).hostedSearchCitations,