diff --git "a/docs/\351\203\250\347\275\262.md" "b/docs/\351\203\250\347\275\262.md" index 1be7444e..00a4aa14 100644 --- "a/docs/\351\203\250\347\275\262.md" +++ "b/docs/\351\203\250\347\275\262.md" @@ -293,6 +293,8 @@ docker compose start remote-computer ## 电脑代理的发件箱 -#47 提过发件箱没有大小上限。#91(PR #103)补上了单帧上限:可靠上行进 `ReliableOutbox.append` 之前按 `AGENT_FRAME_MAX_BYTES`(16 MiB,`packages/shared/src/protocol.ts`)截短,截不下的拒收,不再因为一帧超限把整个发件箱卡死(`packages/computer-agent/src/frame-limit.ts`、`reliable-transport.ts`)。 +#47 提过发件箱没有大小上限。#91(PR #103)先补上了单帧上限:可靠上行进 `ReliableOutbox.append` 之前按 `AGENT_FRAME_MAX_BYTES`(16 MiB,`packages/shared/src/protocol.ts`)截短,截不下的拒收,不再因为一帧超限把整个发件箱卡死(`packages/computer-agent/src/frame-limit.ts`、`reliable-transport.ts`)。 -帧数和总字节数仍然没有上限,也没有淘汰策略,总量上限拆到 #115 跟进。控制面只认连续的序号,删掉中间的帧,后面的帧全成缺号(`reliable-transport.ts` 里旧帧截短那段的注释)。发件箱在 `/.agent/outbox/`,控制面确认之后就压缩掉已确认的帧。派活走下行,断线期间不会有新 run 派进来。电脑长期离线时要看的是这个目录的大小,在服务器上就是 `bots` 卷下的 `.agent/outbox/`。 +#115 补上了总量上限:未确认帧按 journal 里那一行的字节数累计,超过 128 MiB 就从最早的帧开始原位收短,收到 96 MiB 以下为止(`OUTBOX_LIMITS`,`reliable-transport.ts`)。只收展示用的输出:`bot_text_done.text` 截到 64 KiB、`agent_error.detail` 截到 4 KiB,末尾写「电脑离线太久,积压的输出太多,后面的已截掉」;`bot_tool_call` 的工具卡按单帧上限同样的规则收短。`run_done`、审批、交接、各种 offer、牌桌这些代表决定和结局的报文一帧不动,超了也照收。超限期间新来的 `work_append` 拒收,代理日志打一行「发件箱积压已有 N 字节,work_append 不进发件箱」,工作记录镜像收到 `invalid` 停流。run 照常收尾,但被拒那一轮的工作记录没上传,下一轮运行器按服务器上的快照重建会话,Bot 丢掉那一轮的会话记忆。停流之后,同一个 run 里剩下的有副作用的工具调用(读和写工作区以外的动作,比如跑命令、写工作区外、发布、发消息)会被道围栏挡下,工具结果是「工作记录没有确认,这一步先不做」。群里会出现一条「工作记录停止上传」的系统消息(文案由 #132 跟进)。收完还超(剩下的都收不动)就照收,代理日志打一行「发件箱收短后仍有 N 字节」。每次收短日志打一行汇总:收了几帧、从多少字节到多少字节。旧版代理留下的超量积压在代理启动时按同一套收短。 + +不删帧,因为控制面只认连续的序号,删掉中间的帧,后面的帧全成缺号;原位换掉不碍去重,已处理过的序号控制面只回确认,没处理过的就处理换上去的那一条。发件箱在 `/.agent/outbox/`,控制面确认之后就压缩掉已确认的帧。派活走下行,断线期间不会有新 run 派进来,结局类报文靠它自己涨满 128 MiB 要几十万个 run,所以实际占用不会远超上限。电脑长期离线时要看的是这个目录的大小,在服务器上就是 `bots` 卷下的 `.agent/outbox/`,日志里出现「发件箱积压超过上限」就说明有输出被收短了。 diff --git a/packages/computer-agent/README.md b/packages/computer-agent/README.md index bcdbf007..649d352f 100644 --- a/packages/computer-agent/README.md +++ b/packages/computer-agent/README.md @@ -2,7 +2,7 @@ 电脑代理以协议 3 完成握手。首次启动生成 UUID,完整落盘到 `/.agent/install-id` 后才发 hello;重连和进程重启复用它。hello 同时携带运行器导出的 Pi `CURRENT_SESSION_VERSION`。每条可靠上行先写入 `/.agent/outbox/` 并 fsync,再通过当前连接发送;welcome 的累计确认和后续 `uplink_ack` 会清理已确认帧。重连或进程重启会按原 stream 和序号补发未确认帧。 -可靠上行进发件箱之前按整帧序列化后的字节数对 `AGENT_FRAME_MAX_BYTES` 检查(`src/frame-limit.ts`),发件箱重开时对未确认帧也按同样规则原位改写,序号不变。控制面收到超限帧以 1009 关连接,代理重连重发同一帧又被关,整台电脑的发件箱就卡死了,所以超限只能在入队和重放前处理:`bot_text_done.text` 和 `agent_error.detail` 按 UTF-8 字节截尾并注明「内容过长」,不切开多字节字符;`bot_tool_call` 只是展示,`view.raw` 换成一句说明字节数的话,命令、路径、预览、参数各截到 4 KiB。`approval_request` 不一样,用户是看着卡决定批不批的,截掉的尾巴照样会执行,所以只换 `view.raw`、截写入预览,命令、说明、路径、工具连接、工具名和参数一个字不动。截完仍超限、带 runId 的帧原位换成同一 runId 的 `agent_error`(`runner_failed`),控制面替这个 run 收尾,代理本地 `cancel` 它,挂着的审批随之按过期结算,工具不执行;审批请求的说明写「审批请求过长,没法完整给你看,这一步没有执行」。`work_append` 由运行器先按字节切批,到这里还超限就回 `invalid` 让镜像停流。`mcp_discovered` 是 bare 报文,schema 已把结果限在 1 MiB,不经这道检查。 +可靠上行进发件箱之前按整帧序列化后的字节数对 `AGENT_FRAME_MAX_BYTES` 检查(`src/frame-limit.ts`),发件箱重开时对未确认帧也按同样规则原位改写,序号不变。控制面收到超限帧以 1009 关连接,代理重连重发同一帧又被关,整台电脑的发件箱就卡死了,所以超限只能在入队和重放前处理:`bot_text_done.text` 和 `agent_error.detail` 按 UTF-8 字节截尾并注明「内容过长」,不切开多字节字符;`bot_tool_call` 只是展示,超过 4 KiB 的 `view.raw` 换成一句说明字节数的话,命令、路径、预览、参数各截到 4 KiB。`approval_request` 不一样,用户是看着卡决定批不批的,截掉的尾巴照样会执行,所以只换超过 4 KiB 的 `view.raw`、截写入预览,命令、说明、路径、工具连接、工具名和参数一个字不动。截完仍超限、带 runId 的帧原位换成同一 runId 的 `agent_error`(`runner_failed`),控制面替这个 run 收尾,代理本地 `cancel` 它,挂着的审批随之按过期结算,工具不执行;审批请求的说明写「审批请求过长,没法完整给你看,这一步没有执行」。`work_append` 由运行器先按字节切批,到这里还超限就回 `invalid` 让镜像停流。`mcp_discovered` 是 bare 报文,schema 已把结果限在 1 MiB,不经这道检查。 `.agent/computer.json` 记录传输状态所属的 `computerId`;换电脑、旧版无归属状态或状态文件损坏时,代理把旧 outbox、下行游标和归属信息移入 `.agent/transport-quarantine-/`,并从新流和游标 0 开始。隔离不移动 `.agent/install-id`,安装标识属于这份 dataDir,不随换电脑重置。服务器 welcome 的确认序号低于本地记录时,也会隔离旧状态并主动重连新流。 @@ -15,3 +15,17 @@ prompt 的 `lane` 原样交给运行器。`cancel{runId}` 停单条运行;`can 本机 MCP 的 `mcp_discover` 不经 Pi 会话或 Bot 串行队列,由运行器独立探测模块处理,`mcp_discovered` 只回固定枚举或操作表。两种报文都不落盘;探测断线或停机时取消,旧连接结果不回到新连接。 附件上传和发布下载在实际请求时读取最近 config 的 hub 和网关 key。config 尚未下发时收到附件发布,返回固定错误「发布出错:附件配置尚未下发」,不连接浏览器;这与实际下载失败的「发布出错:配图从白泽下载失败,请稍后重试」分开。 + +## 发件箱上限 + +电脑长期离线时未确认的上行会一直累积,所以发件箱有总量上限(`OUTBOX_LIMITS`,`src/reliable-transport.ts`)。量的是未确认帧在 journal 里那一行的字节,不按条数:占磁盘和内存的是字节,一条 `bot_text_done` 可能比一千条 `run_done` 还大。累计字节数在入队、确认、收短时增减,不每次重量整队。 + +每次 flush 写盘之前,积压超过 128 MiB 就从最早的未确认帧往后原位收短,收到 96 MiB 以下为止(`trimBacklog`,`src/frame-limit.ts`)。还没落盘的帧直接写收短后的,只有已落盘的帧被换掉才重写 journal:剩下的都收不动时,要是每批新帧都整份重写,每批就要写一遍上限那么大的文件。收哪些: + +- `bot_text_done.text` 截到 64 KiB,`agent_error.detail` 截到 4 KiB,末尾写「…(电脑离线太久,积压的输出太多,后面的已截掉)」。 +- `bot_tool_call.view` 用单帧上限的 `compactView` 收短。不到 4 KiB 的原始参数留着,收过的帧再收不会变小,也就不再换。 +- 其余报文原样:`run_done`、`approval_request`、`approval_abandoned`、`handoff`、`handoff_accept`、各种 offer、`poker_*` 代表决定和结局,超了也照收。审批卡不收,用户是看着卡批的。 + +收不小的帧不换,收过的帧不再收。帧不删:控制面只认连续的序号,删一条后面全成缺号;原位换、序号不变和单帧上限是同一个办法,控制面处理过的序号只回确认,没处理过的处理换上去的那条,所以控制面不需要知道发件箱满过。新来的 `work_append` 会让总量超过上限时直接拒收,走 `OversizedUplink`,代理日志打一行拒收原因,镜像回 `invalid` 停流。停流有代价:run 照常收尾,但被拒那一轮的工作记录没上传,下一轮按服务器快照重建会话,Bot 丢掉那一轮的会话记忆;同一个 run 里剩下的有副作用的工具调用会被 `bot-runner/src/lane-fence.ts` 的 `beforeEffect` 挡下(镜像停流后 `flush` 一直回 `invalid`,见 `work-mirror.ts` 的 `#flush`),读和写工作区不受影响;群里出现一条「工作记录停止上传」的系统消息(文案由 #132 跟进)。单帧超限的 `work_append` 也是同样的后果。收完还超就照收,日志打一行当前字节数。冷启动打开发件箱时同样执行一遍,处理旧版代理留下的超量 journal。 + +重写 journal 只写已落盘的帧(`pending()`),还在 `#writes` 里的帧由下一次 flush 追加,否则同一序号会写两行,重开时被当成损坏隔离。确认之后的压缩原来也有这个问题,一并改了。 diff --git a/packages/computer-agent/src/agent.ts b/packages/computer-agent/src/agent.ts index 04c37371..ca77b17d 100644 --- a/packages/computer-agent/src/agent.ts +++ b/packages/computer-agent/src/agent.ts @@ -35,7 +35,7 @@ import { type AssistantChrome, type AssistantChromeOptions, startAssistantChrome import { createLiveScreen } from "./live-screen.js"; import { checkLogin, createLogins, defaultLoginPages } from "./login.js"; import { createTerminals, ptySpawner } from "./terminal.js"; -import { DownlinkCursor, quarantineTransportState, ReliableOutbox } from "./reliable-transport.js"; +import { DownlinkCursor, type OutboxLimits, quarantineTransportState, ReliableOutbox } from "./reliable-transport.js"; import { OversizedUplink } from "./frame-limit.js"; import { loadInstallId } from "./install-id.js"; import { WorkAppendChannel } from "./work-append.js"; @@ -67,6 +67,7 @@ export interface AgentOptions { readonly loginTimeoutMs?: number; /** 测试用断线注入点:持久化完成后、可靠帧发送前调用。 */ readonly beforeReliableSend?: (frame: UplinkFrame) => void; + readonly outboxLimits?: OutboxLimits; readonly discoverMcp?: typeof discoverMcpTools; } @@ -240,6 +241,7 @@ export function startAgent(options: AgentOptions): AgentHandle { return { seq: stored.seq, body: frame }; }, send: sendStored, + log, cancel: (lane, epoch) => runner?.cancel({ lane: { botId: lane.botId, groupId: lane.groupId }, epoch }), failed: (error) => { log(`工作记录探针落盘失败:${String(error)}`); @@ -386,8 +388,14 @@ export function startAgent(options: AgentOptions): AgentHandle { await previousOutbox.flush(); if (stopping) return; try { - await quarantineTransportState(dataDir, "服务器确认序号回退", options.log); - const fresh = await ReliableOutbox.openForComputer(dataDir, options.computerId, options.log, dropRun); + await quarantineTransportState(dataDir, "服务器确认序号回退", log); + const fresh = await ReliableOutbox.openForComputer( + dataDir, + options.computerId, + log, + dropRun, + options.outboxLimits, + ); outbox = fresh.outbox; downlinkCursor = fresh.downlinkCursor; resettingTransport = false; @@ -644,7 +652,7 @@ export function startAgent(options: AgentOptions): AgentHandle { } void Promise.all([ - ReliableOutbox.openForComputer(dataDir, options.computerId, options.log, dropRun), + ReliableOutbox.openForComputer(dataDir, options.computerId, log, dropRun, options.outboxLimits), loadInstallId(dataDir), ]) .then(([{ outbox: openedOutbox, downlinkCursor: openedCursor }, openedInstallId]) => { diff --git a/packages/computer-agent/src/frame-limit.ts b/packages/computer-agent/src/frame-limit.ts index cc9a42a4..4bd46026 100644 --- a/packages/computer-agent/src/frame-limit.ts +++ b/packages/computer-agent/src/frame-limit.ts @@ -2,6 +2,9 @@ import { AGENT_FRAME_MAX_BYTES, type AgentToControl, type JsonValue, type ToolVi const CLIPPED = "…(内容过长,后面的已截掉)"; const VIEW_TEXT_MAX_BYTES = 4096; +const BACKLOG_CLIPPED = "…(电脑离线太久,积压的输出太多,后面的已截掉)"; +const BACKLOG_TEXT_MAX_BYTES = 64 * 1024; +const BACKLOG_DETAIL_MAX_BYTES = 4 * 1024; export class OversizedUplink extends Error {} @@ -20,10 +23,10 @@ function utf8Prefix(text: string, limit: number): string { return encoded.subarray(0, end).toString("utf8"); } -const short = (text: string): string => - utf8Bytes(text) <= VIEW_TEXT_MAX_BYTES - ? text - : `${utf8Prefix(text, VIEW_TEXT_MAX_BYTES - utf8Bytes(CLIPPED))}${CLIPPED}`; +const clipTo = (text: string, limit: number, tail: string): string => + utf8Bytes(text) <= limit ? text : `${utf8Prefix(text, limit - utf8Bytes(tail))}${tail}`; + +const short = (text: string): string => clipTo(text, VIEW_TEXT_MAX_BYTES, CLIPPED); function shortJson(value: JsonValue): JsonValue { if (typeof value === "string") return short(value); @@ -31,8 +34,11 @@ function shortJson(value: JsonValue): JsonValue { return utf8Bytes(text) <= VIEW_TEXT_MAX_BYTES ? value : short(text); } -const rawNotice = (view: ToolView): { raw?: string } => - view.raw === undefined ? {} : { raw: `原始参数有 ${utf8Bytes(JSON.stringify(view.raw))} 字节,太大没有上传` }; +function rawNotice(view: ToolView): { raw?: JsonValue } { + if (view.raw === undefined) return {}; + const bytes = utf8Bytes(JSON.stringify(view.raw)); + return { raw: bytes <= VIEW_TEXT_MAX_BYTES ? view.raw : `原始参数有 ${bytes} 字节,太大没有上传` }; +} /** 工具卡只是展示,哪个字段都可能装着模型给的大参数,所以整份收短。 */ function compactView(view: ToolView): ToolView { @@ -73,6 +79,19 @@ function approvalView(view: ToolView): ToolView { return view.kind === "write" ? { ...view, preview: view.preview.map(short), ...raw } : { ...view, ...raw }; } +export function trimBacklog(body: AgentToControl): AgentToControl | null { + switch (body.type) { + case "bot_text_done": + return { ...body, text: clipTo(body.text, BACKLOG_TEXT_MAX_BYTES, BACKLOG_CLIPPED) }; + case "agent_error": + return { ...body, detail: clipTo(body.detail, BACKLOG_DETAIL_MAX_BYTES, BACKLOG_CLIPPED) }; + case "bot_tool_call": + return { ...body, view: compactView(body.view) }; + default: + return null; + } +} + function shrink(body: AgentToControl, over: number): AgentToControl | null { // JSON 转义只会让字节变多,按原文字节砍掉 over 就一定够。 const clip = (text: string): string => `${utf8Prefix(text, utf8Bytes(text) - over - utf8Bytes(CLIPPED))}${CLIPPED}`; diff --git a/packages/computer-agent/src/reliable-transport.ts b/packages/computer-agent/src/reliable-transport.ts index 06bfdea3..04fde27f 100644 --- a/packages/computer-agent/src/reliable-transport.ts +++ b/packages/computer-agent/src/reliable-transport.ts @@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto"; import { mkdir, open, readFile, rename, stat } from "node:fs/promises"; import { dirname, join } from "node:path"; import { AgentToControl, UPLINK_DELIVERY, type AgentToControl as AgentMessage, type RunId } from "@baize/shared"; -import { fitUplink, OversizedUplink, type FittedUplink } from "./frame-limit.js"; +import { fitUplink, OversizedUplink, trimBacklog, type FittedUplink } from "./frame-limit.js"; interface StoredUplink { readonly seq: number; @@ -15,8 +15,8 @@ interface OutboxState { } interface PendingWrite { - readonly entry: StoredUplink; - readonly resolve: () => void; + readonly seq: number; + readonly resolve: (written: StoredUplink) => void; readonly reject: (error: unknown) => void; } @@ -26,6 +26,16 @@ export interface OutboxHooks { readonly dropped?: (runId: RunId) => void; } +export interface OutboxLimits { + readonly maxBytes: number; + readonly trimToBytes: number; +} + +const MiB = 1024 * 1024; +export const OUTBOX_LIMITS: OutboxLimits = { maxBytes: 128 * MiB, trimToBytes: 96 * MiB }; + +const lineBytes = (entry: StoredUplink): number => Buffer.byteLength(JSON.stringify(entry), "utf8") + 1; + /** 按线上帧的最大可能长度量:conn 和 seq 取最长的数字。 */ function fitForStream(stream: string, body: AgentMessage): FittedUplink { return fitUplink(body, (candidate) => ({ @@ -115,9 +125,20 @@ export class ReliableOutbox { #timer: NodeJS.Timeout | null = null; #operation: Promise = Promise.resolve(); #hooks: OutboxHooks; - - private constructor(directory: string, state: OutboxState, frames: StoredUplink[], hooks: OutboxHooks) { + #limits: OutboxLimits; + #bytes = 0; + #sizes = new Map(); + #trimmedThrough: number; + + private constructor( + directory: string, + state: OutboxState, + frames: StoredUplink[], + hooks: OutboxHooks, + limits: OutboxLimits, + ) { this.#hooks = hooks; + this.#limits = limits; this.#directory = join(directory, state.stream); this.#statePath = join(directory, "state.json"); this.stream = state.stream; @@ -126,9 +147,15 @@ export class ReliableOutbox { this.#next = Math.max(state.acked, ...frames.map((frame) => frame.seq), 0) + 1; this.#durable = this.#next - 1; this.#journal = join(this.#directory, "frames.jsonl"); + this.#trimmedThrough = state.acked; + for (const frame of frames) this.#track(frame); } - static async open(dataDir: string, hooks: OutboxHooks = {}): Promise { + static async open( + dataDir: string, + hooks: OutboxHooks = {}, + limits: OutboxLimits = OUTBOX_LIMITS, + ): Promise { const directory = join(dataDir, ".agent", "outbox"); await mkdir(directory, { recursive: true }); await syncDirectory(dataDir); @@ -226,8 +253,9 @@ export class ReliableOutbox { ); return { seq: frame.seq, body: fitted.kind === "fit" ? fitted.body : fitted.notice }; }); - const outbox = new ReliableOutbox(directory, state, fittedFrames, hooks); - if (refitted) await outbox.#compact(); + const outbox = new ReliableOutbox(directory, state, fittedFrames, hooks, limits); + const rewrite = outbox.#bytes > limits.maxBytes && outbox.#trim(); + if (refitted || rewrite) await outbox.#compact(); return outbox; } @@ -236,6 +264,7 @@ export class ReliableOutbox { computerId: string, log?: (line: string) => void, dropped?: (runId: RunId) => void, + limits: OutboxLimits = OUTBOX_LIMITS, ): Promise { const hooks: OutboxHooks = { ...(log === undefined ? {} : { log }), ...(dropped === undefined ? {} : { dropped }) }; const agentDirectory = join(dataDir, ".agent"); @@ -272,12 +301,12 @@ export class ReliableOutbox { let outbox: ReliableOutbox; let downlinkCursor: DownlinkCursor; try { - outbox = await ReliableOutbox.open(dataDir, hooks); + outbox = await ReliableOutbox.open(dataDir, hooks, limits); downlinkCursor = await DownlinkCursor.open(dataDir); } catch (error) { if (!(error instanceof TransportStateCorruption)) throw error; await quarantineTransportState(dataDir, "文件损坏", log); - outbox = await ReliableOutbox.open(dataDir, hooks); + outbox = await ReliableOutbox.open(dataDir, hooks, limits); downlinkCursor = await DownlinkCursor.open(dataDir); } await atomicWrite(identityPath, JSON.stringify({ computerId })); @@ -304,14 +333,20 @@ export class ReliableOutbox { this.#hooks.log?.(`${body.type} 截短后仍超过单帧上限,换成一条报错发给控制面`); if (fitted.notice.runId !== undefined) this.#hooks.dropped?.(fitted.notice.runId); } - const entry: StoredUplink = { seq: this.#next++, body: stored }; + const entry: StoredUplink = { seq: this.#next, body: stored }; + if (body.type === "work_append" && this.#bytes + lineBytes(entry) > this.#limits.maxBytes) + return Promise.reject(new OversizedUplink(`发件箱积压已有 ${this.#bytes} 字节,work_append 不进发件箱`)); + this.#next += 1; this.#frames.push(entry); - const persisted = new Promise((resolve, reject) => this.#writes.push({ entry, resolve, reject })); + this.#track(entry); + const persisted = new Promise((resolve, reject) => + this.#writes.push({ seq: entry.seq, resolve, reject }), + ); if (this.#timer === null) { this.#timer = setTimeout(() => void this.#flush(), 100); this.#timer.unref(); } - return persisted.then(() => entry); + return persisted; } async acknowledge(upTo: number): Promise { @@ -322,6 +357,11 @@ export class ReliableOutbox { const state = { stream: this.stream, acked: bounded }; await atomicWrite(this.#statePath, JSON.stringify(state)); this.#acked = bounded; + for (const frame of this.#frames) { + if (frame.seq > bounded) break; + this.#bytes -= this.#sizes.get(frame.seq) ?? 0; + this.#sizes.delete(frame.seq); + } this.#frames = this.#frames.filter((frame) => frame.seq > bounded); await this.#compact(); }); @@ -345,23 +385,63 @@ export class ReliableOutbox { } } + #track(entry: StoredUplink): void { + const bytes = lineBytes(entry); + this.#bytes += bytes - (this.#sizes.get(entry.seq) ?? 0); + this.#sizes.set(entry.seq, bytes); + } + + /** 返回有没有换掉已落盘的帧:换了就得重写日志,只换了还没落盘的帧照常追加就行。 */ + #trim(): boolean { + const { maxBytes, trimToBytes } = this.#limits; + const before = this.#bytes; + let count = 0; + let rewrite = false; + for (const [index, frame] of this.#frames.entries()) { + if (this.#bytes <= trimToBytes) break; + if (frame.seq <= this.#trimmedThrough) continue; + this.#trimmedThrough = frame.seq; + const body = trimBacklog(frame.body); + if (body === null) continue; + const entry: StoredUplink = { seq: frame.seq, body }; + if (lineBytes(entry) >= (this.#sizes.get(frame.seq) ?? 0)) continue; + this.#frames[index] = entry; + this.#track(entry); + count += 1; + if (frame.seq <= this.#durable) rewrite = true; + } + if (count > 0) + this.#hooks.log?.( + `发件箱积压超过上限 ${maxBytes} 字节,收短了最早的 ${count} 帧,从 ${before} 字节到 ${this.#bytes} 字节`, + ); + if (this.#bytes > maxBytes) + this.#hooks.log?.(`发件箱收短后仍有 ${this.#bytes} 字节,超过上限 ${maxBytes} 字节,剩下的报文收不动了,照收`); + return rewrite; + } + async #flushNow(): Promise { if (this.#failed) throw this.#failure; if (this.#timer !== null) clearTimeout(this.#timer); this.#timer = null; const writes = this.#writes.splice(0); if (writes.length === 0) return; + // 写盘前收:还没落盘的帧直接追加收短后的,只有已落盘的帧被换掉才重写日志。收不动的积压下每批都重写,就是每批写一遍上限那么多字节。 + const rewrite = this.#bytes > this.#limits.maxBytes && this.#trim(); + const unwritten = new Map( + this.#frames.filter((frame) => frame.seq > this.#durable).map((frame) => [frame.seq, frame]), + ); try { + if (rewrite) await this.#compact(); const file = await open(this.#journal, "a", 0o600); try { - await file.writeFile(writes.map(({ entry }) => `${JSON.stringify(entry)}\n`).join(""), "utf8"); + await file.writeFile([...unwritten.values()].map((frame) => `${JSON.stringify(frame)}\n`).join(""), "utf8"); await file.sync(); } finally { await file.close(); } await syncDirectory(this.#directory); - this.#durable = writes.at(-1)?.entry.seq ?? this.#durable; - for (const write of writes) write.resolve(); + this.#durable = writes.at(-1)?.seq ?? this.#durable; + for (const write of writes) write.resolve(unwritten.get(write.seq)!); } catch (error) { this.#failed = true; this.#failure = error; @@ -372,7 +452,9 @@ export class ReliableOutbox { } async #compact(): Promise { - const contents = this.#frames.map((frame) => `${JSON.stringify(frame)}\n`).join(""); + const contents = this.pending() + .map((frame) => `${JSON.stringify(frame)}\n`) + .join(""); const temporary = `${this.#journal}.${randomUUID()}.tmp`; const file = await open(temporary, "wx", 0o600); try { diff --git a/packages/computer-agent/src/work-append.ts b/packages/computer-agent/src/work-append.ts index aa3f3df8..9fb3bdf3 100644 --- a/packages/computer-agent/src/work-append.ts +++ b/packages/computer-agent/src/work-append.ts @@ -39,6 +39,7 @@ interface WorkAppendChannelOptions { readonly connected: () => boolean; readonly append: (frame: WorkAppend) => Promise; readonly send: (stored: Stored) => void; + readonly log: (line: string) => void; readonly cancel: (lane: NonNullable, epoch: number) => void; readonly failed: (error: unknown) => void; } @@ -71,6 +72,7 @@ export class WorkAppendChannel { .catch((error: unknown) => { // 运行器已按字节切批,到这里还超限就让镜像停流;回 disconnected 它会拿同一批反复重试。 if (error instanceof OversizedUplink) { + this.#options.log(error.message); this.#settle(waiting, { kind: "invalid" }); return; } diff --git a/packages/computer-agent/test/agent.test.ts b/packages/computer-agent/test/agent.test.ts index 68218df7..5295fd6d 100644 --- a/packages/computer-agent/test/agent.test.ts +++ b/packages/computer-agent/test/agent.test.ts @@ -507,8 +507,12 @@ function spawnAgent( | "log" | "beforeReliableSend" | "computerId" + | "outboxLimits" > - > = {}, + > & { + /** 不传 log,日志走代理自己的默认输出,等同 main.ts 和桌面端拉起代理的用法。 */ + readonly productLog?: true; + } = {}, ): AgentHandle { const calls: RunnerCalls = { approvals: [], @@ -527,15 +531,16 @@ function spawnAgent( runnerCallsLog.push(calls); const dataDir = tuning.dataDir ?? mkdtempSync(join(tmpdir(), "baize-agent-test-")); if (tuning.dataDir === undefined) dataDirs.push(dataDir); + const { productLog, ...options } = tuning; const handle = startAgent({ url, computerId: tuning.computerId ?? ComputerId.parse("computer-1"), token: "dev-token", version: "0.1.0", - log: () => {}, + ...(productLog ? {} : { log: () => {} }), makeRunner: fakeRunner(calls), dataDir, - ...tuning, + ...options, }); agents.push(handle); return handle; @@ -1023,6 +1028,53 @@ describe("电脑代理连控制面", () => { expect(hookCalls).toBe(2); }); + it("不传 log 的代理(main.ts 的用法)冷启动收短发件箱、拒收工作记录,代理日志里都看得到", async () => { + const server = startControlPlane(); + servers.push(server); + const directory = await tempDataDir(); + const previous = await ReliableOutbox.openForComputer(directory, "computer-1"); + await previous.outbox.append({ + type: "bot_text_done", + runId: RunId.parse("run-old"), + botId: BotId.parse("bot-1"), + groupId: GroupId.parse("group-1"), + messageId: MessageId.parse("msg-old"), + text: "旧".repeat(100 * 1024), + }); + const printed: string[] = []; + vi.spyOn(console, "log").mockImplementation((line: unknown) => printed.push(String(line))); + spawnAgent(await server.url(), { + dataDir: directory, + productLog: true, + outboxLimits: { maxBytes: 128, trimToBytes: 64 }, + }); + const socket = await server.waitForSocket(1); + socket.send(JSON.stringify(configMessage)); + await server.waitForMessage("config_applied"); + const result = await initialRunnerConfigs.at(-1)!.workAppend!({ + type: "work_append", + lane: { botId: BotId.parse("bot-1"), groupId: GroupId.parse("group-1"), stream: "chat", epoch: 7 }, + runId: RunId.parse("run-1"), + fromSeq: 0, + lines: ["一行工作记录"], + head: "0".repeat(64), + cause: "turn_end", + }); + + expect(result).toEqual({ kind: "invalid" }); + expect(printed).toEqual( + expect.arrayContaining([ + expect.stringMatching( + /^\[computer-agent\] 发件箱积压超过上限 128 字节,收短了最早的 1 帧,从 \d+ 字节到 \d+ 字节$/, + ), + expect.stringMatching( + /^\[computer-agent\] 发件箱收短后仍有 \d+ 字节,超过上限 128 字节,剩下的报文收不动了,照收$/, + ), + expect.stringMatching(/^\[computer-agent\] 发件箱积压已有 \d+ 字节,work_append 不进发件箱$/), + ]), + ); + }); + it("超过单帧上限的可靠上行截短后送达,截不下的换成同一 run 的报错并停掉它,后面的帧照常到、连接不被 1009 关掉", async () => { const server = startControlPlane(); servers.push(server); diff --git a/packages/computer-agent/test/reliable-transport.test.ts b/packages/computer-agent/test/reliable-transport.test.ts index b36787fd..278d67f6 100644 --- a/packages/computer-agent/test/reliable-transport.test.ts +++ b/packages/computer-agent/test/reliable-transport.test.ts @@ -1,4 +1,4 @@ -import { appendFile, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { appendFile, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, describe, expect, it } from "vitest"; @@ -491,3 +491,182 @@ describe("发件箱入口的单帧上限", () => { expect((await ReliableOutbox.open(directory)).pending()).toEqual([...pending, { seq: 5, body: runDone("run-5") }]); }); }); + +const BACKLOG = "…(电脑离线太久,积压的输出太多,后面的已截掉)"; +const KiB = 1024; +const limits = { maxBytes: 512 * KiB, trimToBytes: 384 * KiB }; +const textDone = (id: string, text: string): AgentToControl => ({ + type: "bot_text_done", + ...lane, + messageId: MessageId.parse(id), + text, +}); +const toolCall = (id: string, command: string): AgentToControl => ({ + type: "bot_tool_call", + ...lane, + callId: id, + tool: "bash", + view: { kind: "bash", command, raw: { command } }, + status: "ok", + durationMs: 5, +}); +const workAppend = (line: string): AgentToControl => ({ + type: "work_append", + lane: { botId: lane.botId, groupId: lane.groupId, stream: "chat", epoch: 1 }, + runId: lane.runId, + fromSeq: 0, + lines: [line], + head: "0".repeat(64), + cause: "turn_end", +}); +const journalBytes = async (outbox: ReliableOutbox, directory: string): Promise => + Buffer.byteLength(await readFile(join(directory, ".agent", "outbox", outbox.stream, "frames.jsonl"), "utf8")); + +describe("发件箱总量上限", () => { + it("积压超过上限时从最早的大帧原位收短到回落线以下,序号连续,结局帧原样,收短的写明原因,盘上也改掉", async () => { + const directory = await dataDir(); + const logs: string[] = []; + const outbox = await ReliableOutbox.open(directory, { log: (line) => logs.push(line) }, limits); + const error: AgentToControl = { + type: "agent_error", + code: "runner_failed", + detail: "e".repeat(3 * KiB), + runId: lane.runId, + }; + const bodies: AgentToControl[] = [ + textDone("m-1", "早".repeat(100 * KiB)), + runDone("run-1"), + toolCall("call-1", "命".repeat(60 * KiB)), + error, + textDone("m-2", "晚".repeat(100 * KiB)), + ]; + for (const body of bodies) await outbox.append(body); + await outbox.flush(); + + const reopened = await ReliableOutbox.open(directory, {}, limits); + const pending = reopened.pending(); + expect(pending.map((frame) => frame.seq)).toEqual([1, 2, 3, 4, 5]); + const [first, done, tool, kept, last] = pending.map((frame) => frame.body); + if (first?.type !== "bot_text_done") throw new Error(`第 1 帧成了 ${first?.type}`); + expect(first.text.endsWith(BACKLOG)).toBe(true); + expect(Buffer.byteLength(first.text)).toBeLessThanOrEqual(64 * KiB); + expect("早".repeat(100 * KiB).startsWith(first.text.slice(0, -BACKLOG.length))).toBe(true); + expect(done).toEqual(runDone("run-1")); + if (tool?.type !== "bot_tool_call" || tool.view.kind !== "bash") throw new Error(`第 3 帧成了 ${tool?.type}`); + expect(Buffer.byteLength(tool.view.command)).toBeLessThanOrEqual(4 * KiB); + expect(tool.view.raw).toMatch(/^原始参数有 \d+ 字节,太大没有上传$/); + expect(kept, "agent_error 的 detail 不到 4 KiB 不收").toEqual(error); + expect(last, "回落到线以下就停,后面的大帧不动").toEqual(bodies[4]); + expect(await journalBytes(reopened, directory)).toBeLessThanOrEqual(limits.trimToBytes); + expect(logs).toEqual([ + expect.stringMatching(/^发件箱积压超过上限 524288 字节,收短了最早的 2 帧,从 \d+ 字节到 \d+ 字节$/), + ]); + }); + + it("旧版代理留下的超量积压在重开时收短并重写日志:agent_error 的 detail 收到 4 KiB,审批请求不收,收过的不再收", async () => { + const directory = await dataDir(); + const outbox = await ReliableOutbox.open(directory); + const approval: AgentToControl = { + type: "approval_request", + ...lane, + callId: "call-1", + tool: "bash", + action: "shell", + target: null, + view: { kind: "bash", command: "批".repeat(200 * KiB) }, + }; + await outbox.append(approval); + await outbox.append({ type: "agent_error", code: "runner_failed", detail: "错".repeat(200 * KiB), ...lane }); + await outbox.append(toolCall("call-2", "命".repeat(60 * KiB))); + await outbox.append(textDone("m-1", "早".repeat(100 * KiB))); + await outbox.flush(); + const once = (await ReliableOutbox.open(directory, {}, limits)).pending(); + expect((await ReliableOutbox.open(directory)).pending(), "日志已重写").toEqual(once); + + expect(once[0]?.body).toEqual(approval); + const error = once[1]?.body; + if (error?.type !== "agent_error") throw new Error(`第 2 帧成了 ${error?.type}`); + expect(error.detail.endsWith(BACKLOG)).toBe(true); + expect(Buffer.byteLength(error.detail)).toBeLessThanOrEqual(4 * KiB); + + const logs: string[] = []; + const twice = await ReliableOutbox.open(directory, { log: (line) => logs.push(line) }, limits); + expect(twice.pending()).toEqual(once); + expect(logs).toEqual([ + expect.stringMatching(/^发件箱收短后仍有 \d+ 字节,超过上限 524288 字节,剩下的报文收不动了,照收$/), + ]); + }); + + it("超过上限时新来的 work_append 拒收不占序号,结局帧照收并在日志里报当前字节数", async () => { + const directory = await dataDir(); + const logs: string[] = []; + const outbox = await ReliableOutbox.open(directory, { log: (line) => logs.push(line) }, limits); + await Promise.all(Array.from({ length: 4000 }, (_, index) => outbox.append(runDone(`run-${index + 1}`)))); + + await expect(outbox.append(workAppend("一行工作记录"))).rejects.toThrow(/work_append 不进发件箱/); + const next = await outbox.append(runDone("run-last")); + await outbox.flush(); + + expect(next.seq).toBe(4001); + expect(outbox.pending().map((frame) => frame.seq)).toEqual(Array.from({ length: 4001 }, (_, index) => index + 1)); + expect(outbox.pending().at(-1)?.body).toEqual(runDone("run-last")); + expect(logs.at(-1)).toMatch(/^发件箱收短后仍有 \d+ 字节,超过上限 524288 字节,剩下的报文收不动了,照收$/); + }); + + it("确认之后积压字节随之减少,work_append 又能进发件箱", async () => { + const outbox = await ReliableOutbox.open(await dataDir(), {}, limits); + await Promise.all(Array.from({ length: 4000 }, (_, index) => outbox.append(runDone(`run-${index + 1}`)))); + await expect(outbox.append(workAppend("被拒"))).rejects.toThrow(/work_append 不进发件箱/); + + await outbox.acknowledge(4000); + + expect((await outbox.append(workAppend("放行"))).seq).toBe(4001); + }); + + it("收不动的积压超过回落线后,新来的工具卡落盘前就收短,日志只追加不整份重写", async () => { + const directory = await dataDir(); + const outbox = await ReliableOutbox.open(directory, {}, limits); + for (let index = 0; index < 6; index += 1) + await outbox.append({ + type: "approval_request", + ...lane, + callId: `approval-${index}`, + tool: "bash", + action: "shell", + target: null, + view: { kind: "bash", command: "批".repeat(40 * KiB) }, + }); + const journalPath = join(directory, ".agent", "outbox", outbox.stream, "frames.jsonl"); + const { ino } = await stat(journalPath); + + for (let index = 0; index < 20; index += 1) { + await outbox.append(toolCall(`call-${index}`, "命".repeat(8 * KiB))); + await outbox.flush(); + expect((await stat(journalPath)).ino, `第 ${index + 1} 条工具卡之后日志还是原来那个文件`).toBe(ino); + } + const onDisk = (await readFile(journalPath, "utf8")) + .split("\n") + .filter((line) => line.length > 0) + .map((line) => JSON.parse(line) as { seq: number; body: AgentToControl }); + expect(onDisk.map((frame) => frame.seq)).toEqual(Array.from({ length: 26 }, (_, index) => index + 1)); + for (const { body } of onDisk.slice(6)) { + if (body.type !== "bot_tool_call" || body.view.kind !== "bash") throw new Error(`工具卡成了 ${body.type}`); + expect(Buffer.byteLength(body.view.command)).toBeLessThanOrEqual(4 * KiB); + expect(body.view.raw).toMatch(/^原始参数有 \d+ 字节,太大没有上传$/); + } + expect(outbox.pending().map((frame) => frame.body)).toEqual(onDisk.map((frame) => frame.body)); + }); + + it("压缩日志时还没落盘的帧不写进去,重开不出重复序号", async () => { + const directory = await dataDir(); + const outbox = await ReliableOutbox.open(directory); + await outbox.append(runDone("run-1")); + const acknowledged = outbox.acknowledge(1); + await new Promise((resolve) => setImmediate(resolve)); + const late = outbox.append(runDone("run-2")); + await acknowledged; + await late; + + expect((await ReliableOutbox.open(directory)).pending()).toEqual([{ seq: 2, body: runDone("run-2") }]); + }); +}); diff --git a/packages/computer-agent/test/work-append.test.ts b/packages/computer-agent/test/work-append.test.ts index 6188fe8c..897a83d0 100644 --- a/packages/computer-agent/test/work-append.test.ts +++ b/packages/computer-agent/test/work-append.test.ts @@ -27,6 +27,7 @@ describe("工作记录探针的确认通路", () => { sent.push(stored.seq); channel.acknowledge({ type: "uplink_ack", upTo: stored.seq, rejected: [] }); }, + log: () => {}, cancel: () => {}, failed: () => {}, }); @@ -47,6 +48,7 @@ describe("工作记录探针的确认通路", () => { send: (stored) => { sent.push(stored.seq); }, + log: () => {}, cancel: (lane, epoch) => { cancelled.push({ lane, epoch }); }, @@ -73,6 +75,7 @@ describe("工作记录探针的确认通路", () => { connected: () => true, append: async (body) => ({ seq: 1, body }), send: () => {}, + log: () => {}, cancel: () => {}, failed: () => {}, }); @@ -103,6 +106,7 @@ describe("工作记录探针的确认通路", () => { connected: () => true, append: async (body) => ({ seq: 1, body }), send: () => {}, + log: () => {}, cancel: () => {}, failed: () => {}, }); @@ -130,6 +134,7 @@ describe("工作记录探针的确认通路", () => { send: (stored) => { sent.push(stored.seq); }, + log: () => {}, cancel: () => {}, failed: (error) => { failures.push(error); diff --git a/packages/e2e/src/stack.ts b/packages/e2e/src/stack.ts index def0731d..31dc4b55 100644 --- a/packages/e2e/src/stack.ts +++ b/packages/e2e/src/stack.ts @@ -63,6 +63,7 @@ export type StackOptions = { readonly cuttableAgentLink?: boolean; /** 可靠上行落盘后、实际发送前的确定注入点。 */ readonly beforeReliableSend?: AgentOptions["beforeReliableSend"]; + readonly outboxLimits?: AgentOptions["outboxLimits"]; /** 发布走的平台接口根地址,指到假平台。不给就是一个连不上的地址,绝不打到真 GitHub 和 X。 */ readonly githubApiBaseUrl?: string; readonly xApiBaseUrl?: string; @@ -236,6 +237,7 @@ export async function startStack(options: StackOptions): Promise { version: AGENT_VERSION, dataDir, ...(options.beforeReliableSend === undefined ? {} : { beforeReliableSend: options.beforeReliableSend }), + ...(options.outboxLimits === undefined ? {} : { outboxLimits: options.outboxLimits }), log: (line: string) => { log += `${line}\n`; }, diff --git a/packages/e2e/test/outbox-cap.test.ts b/packages/e2e/test/outbox-cap.test.ts new file mode 100644 index 00000000..81fc4538 --- /dev/null +++ b/packages/e2e/test/outbox-cap.test.ts @@ -0,0 +1,65 @@ +import { expect, it } from "vitest"; +import { BotListResponse, MessageListResponse } from "@baize/shared"; +import { startStack, type Stack } from "../src/stack.js"; +import { startUpstream } from "../src/upstream.js"; + +const poll = { timeout: 15_000, interval: 25 }; +const KiB = 1024; +const BACKLOG = "…(电脑离线太久,积压的输出太多,后面的已截掉)"; + +it("断线期间发件箱积压超过上限:重连后群里是收短的正文加原因,run 照常收尾记费用,不是报错", async () => { + let started: Stack | undefined; + let cut = false; + const long = "长".repeat(100 * KiB); + const upstream = await startUpstream(() => ({ textChunks: ["开头。", long], usage: { input: 20, output: 5 } })); + try { + const stack = await startStack({ + upstreamBaseUrl: upstream.baseUrl, + priceInputPerM: 3, + priceOutputPerM: 15, + cuttableAgentLink: true, + outboxLimits: { maxBytes: 128 * KiB, trimToBytes: 96 * KiB }, + beforeReliableSend: (frame) => { + if (!cut && frame.body.type === "bot_text_done") { + cut = true; + stack?.cutAgentLink(); + } + }, + }); + started = stack; + const bot = BotListResponse.parse(await stack.get("/api/bots")).items.find((item) => item.bot.name === "小服"); + if (bot === undefined) throw new Error("seed 没建出小服"); + const { groupId } = bot; + await stack.post(`/api/groups/${groupId}/messages`, { text: "说一大段" }); + await expect.poll(() => stack.agentLog(), poll).toMatch(/发件箱积压超过上限 131072 字节,收短了最早的 \d+ 帧/); + // 落盘前就收短,收短的日志比第一次发送早。等掐断真的发生了再接回去,不然先接回去后掐断,就再也连不上。 + await expect.poll(() => cut, poll).toBe(true); + + stack.restoreAgentLink(); + const bodies = async () => + MessageListResponse.parse(await stack.get(`/api/groups/${groupId}/messages`)).messages.map( + (message) => message.body, + ); + await expect.poll(async () => (await bodies()).some((body) => body.kind === "cost"), poll).toBe(true); + + const settled = await bodies(); + const said = settled.flatMap((body) => (body.kind === "bot" ? [body.text] : [])); + expect(said, stack.agentLog()).toHaveLength(1); + const [text = ""] = said; + expect(text.endsWith(BACKLOG)).toBe(true); + expect(`开头。${long}`.startsWith(text.slice(0, -BACKLOG.length))).toBe(true); + expect(Buffer.byteLength(text)).toBeLessThanOrEqual(64 * KiB); + const [run] = await stack.sql<{ input_tokens: number; output_tokens: number }[]>` + SELECT input_tokens, output_tokens FROM runs WHERE group_id = ${groupId} + `; + expect(run).toMatchObject({ input_tokens: 20, output_tokens: 5 }); + const errors = settled.flatMap((body) => (body.kind === "system" ? [body.text] : [])); + expect( + errors.filter((line) => !line.includes("工作记录停止上传")), + "超限只让工作记录停流,正文和 run 都不报错", + ).toEqual([]); + } finally { + await started?.close(); + await upstream.close(); + } +});