Skip to content

feat(panel): capture the fields four host events already carry (#698) - #700

Open
LOKOP10 wants to merge 8 commits into
T-Auto:mainfrom
LOKOP10:work/panel-event-fields
Open

feat(panel): capture the fields four host events already carry (#698)#700
LOKOP10 wants to merge 8 commits into
T-Auto:mainfrom
LOKOP10:work/panel-event-fields

Conversation

@LOKOP10

@LOKOP10 LOKOP10 commented Nov 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #698
Closes #909

What was wrong

The original panel capture dropped fields from four host event types, so job lifecycle records could not be joined to the tool that created them. Real ops_bash jobs also omitted toolCall when registering with the host job registry, even though the host execution context supplied callId and rootCallId; PTC job rows therefore fell back to bash instead of resolving to ops_bash. In long captures, a single-target host result replacement could fill a 20,000-row view before the old row was withdrawn, making both rows disappear.

The change

  • Capture the host fields for PTC dispatches, job lifecycle, inbox splices and todo writes, preserving explicit JSON values and bounded text fields.
  • Pass { callId, rootCallId } to jobs.start only when both identities are supplied by the host. Missing identity is left absent rather than guessed.
  • Keep bounded ordered panel views. A confirmed single-target replacement withdraws the old row from each view before adding the replacement, preserving event counts, error counts, dropped-character accounting and existing truncation flags at the 20,000-row boundary.
  • Keep the upstream shell-file-work classifier, guidance revision 4 and current notice text while integrating PR parsing improvements: UTF-16-safe executable scanning, static top-level command/script extraction, quoted keys, property order and Unicode cases. Dynamic templates and concatenated commands remain conservatively unclassified.

Conflict resolution

The branch was merged with upstream main at d670333 in merge commit 7d890d9. The three content conflicts were resolved explicitly:

File Resolution
docs/CHANGELOG.md Kept upstream Unreleased and release entries, then retained the PR bilingual entries and added the #909 fix entry.
src/ptc/batching-guidance.ts Kept upstream shell-file-work.ts, revision 4 and notice wording; integrated the PR executable-position and static field parsing.
test/batching-trigger.test.mjs Kept upstream pipeline, redirection and Select-String cases, plus PR string, UTF-16, script, quoted-key and property-order cases.
test/shells-host.test.mjs Auto-merged changes were reviewed together with the new call-identity assertions.

Validation

Targeted red-to-green checks were run before and after the fixes:

Before fixes: panel-prune-fold 5 of 14 checks failed at the 20,000-row replacement cases.
Before fixes: panel-ptc-host and shells-host failed the four real call-identity assertions.
After fixes: batching-guidance, batching-trigger, batching-examples, panel-prune-fold, panel-ptc-host and shells-host: 6 suites, 111 checks passed, 0 failed, 0 skipped.
cargo test --workspace --locked --manifest-path native/Cargo.toml: 672 passed, 0 failed, 1 ignored.
npm run typecheck: passed.
npm run check:size: 830 code files checked, all at or under 1000 lines.
node scripts/verify-gate.mjs --skip-install: 255 suites, 1 failed, 4 allowed skips.

The release gate completed all build, Rust, worker, Niubash, Bash IPC, manifest and packaging steps. Its only failed suite was preset-registration.test.mjs; the same two host package resolution assertions fail on the synchronized latest main checkout, so this is an existing local host-layout baseline rather than a failure in this PR change. The allowed skips were linux-attributes, linux-shell, tui-host and tui-native on this Windows checkout.

The new pull_request CI run is Cross-platform validation / 43552423146 (pull_request, head edea55d). It completed successfully; all blocking jobs passed, including CI required.

Environment

Windows x64, Node v24.15.0, npm 11.12.1, DSH 0.2.2-rc.1, local checkout based on upstream d670333. No dependencies, package version, permissions or security settings were changed.

@LOKOP10

LOKOP10 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

CI note: CI required is green. The only red job is the non-blocking validate-macos-intel leg, and it is a timeout rather than an assertion failure:

  FAIL npm pack includes compiled host code and native sources, never host sources or build caches
       "elapsedMs": 30072, "signal": "SIGTERM",
       "error": { "code": "ETIMEDOUT", "message": "spawnSync npm ETIMEDOUT" }

test/package-source.test.mjs:12 runs npm pack --dry-run --json --ignore-scripts with timeout: 30000, and the same suite passed on ubuntu-24.04, macos-latest and windows-latest in this run. This PR touches no packaging input.

I cannot re-run that job from a fork (job 127765475913 cannot be rerun), so it needs a maintainer re-run. 中文：macOS Intel 那一条是 npm pack 30 秒超时，不是断言失败；同一套件在本次 run 的 ubuntu、macOS arm64、Windows 三条 leg 上都过了，我这边没权限重跑。

@LOKOP10
LOKOP10 force-pushed the work/panel-event-fields branch from e92be69 to 474733d Compare November 6, 2026 12:28
@T-Auto

T-Auto commented Nov 11, 2026

Copy link
Copy Markdown
Owner

Review：保留功能，需修真实宿主事件契约

审查 head：474733d；对照 main：8f79848。本轮静态审查，未运行测试；#698 的能力缺口仍存在。

[P2] PTC 对象参数被记录成空串

src/panel/events.ts:103–113 只接受 typeof data.arguments === string。目标宿主 DSH 0.2.2-rc.1 的 PTC executor 把 detached JSON 值 normalized.logged 直接写入 dispatch-start/dispatch，常规对象参数在此会丢成空串。测试先 JSON.stringify fixture，掩盖真实 shape 不匹配。

请按 durable event 的 JSON 值做受控序列化后限长，兼容字符串，避免 [object Object]。补嵌套对象/数组、null、字符串、超长参数和 projector→collector→fold 路径；最好由真实 NodePtc 调用产生事件。

[P2] PTC 子调用的 job 缺少工具名关联

job 的 toolCall.callId 可为子调用 id，但 fold 索引只由顶层 tool/call 填充，新增 tool/ptc-dispatch-start 没加入同一映射。PTC job 仍可能只显示 kind，无法关联实际内层工具。

请将 dispatch-start 的 subCallId/name 纳入有界索引，补子调用→job registered/settled、重复事件、FIFO 淘汰测试。无名称时继续诚实 fallback，不从 kind 捏造。

文档与验证

没有发现新增网络外传或跨 session 抓取，不能将本地持久化误报为外泄。但 inbox splice、todo、内层参数/结果扩大明文保存范围，限长不等于脱敏。请双语说明新增类别及可能包含未最终接纳/取消输入，补 capture disabled 不写入、超长截断、非文字 block、HTML 不执行断言。

当前与 main 冲突，旧 CI 绿不能覆盖整合。同步 main 后按真实宿主 event shape 补回归并重跑 CI。

结论：请求修改，不建议直接合并。

@LOKOP10
feat(panel): capture the fields four host events already carry (T-Aut…
c4b9109 
…o#698)
@LOKOP10
fix(panel): serialise PTC arguments and resolve sub-call job names
d221a2e 
- Read the dispatched value as the host writes it (detached JSON) instead of
  dropping every non-string argument to an empty string: a string stays
  verbatim, an object/array keeps its JSON text, null records empty text, and
  textLimit still clamps the result with its full length countable.
- Index a PTC dispatch's sub call id and name in the same bounded FIFO the
  top-level tool/call records feed, so a job dispatched from a run_code
  program resolves the inner tool. A dispatch without a name indexes nothing
  and the job keeps its honest kind fallback.
- Assert nested objects, arrays, null, string and over-limit arguments, the
  sub-call to job join, a repeated start and FIFO eviction; state in both
  changelogs that these categories hold plain text up to textLimit and that
  clamping is not redaction.
@LOKOP10
LOKOP10 force-pushed the work/panel-event-fields branch from 474733d to d221a2e Compare November 11, 2026 05:08
@LOKOP10

LOKOP10 commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

CI 更新（head edea55d）

pull_request 运行 43552423146 已完成，CI required 及所有阻塞 job 均为 success。

更新：已解决上游冲突并修复 #909（head edea55d）

本次更新已将 PR 分支同步到上游 main d670333bff106c144fc3628a038fd7f197cf497e，通过合并提交 7d890d9 解决了三处冲突；自动合并的 test/shells-host.test.mjs 也已复核。

冲突文件 解决方式
docs/CHANGELOG.md 保留上游 [Unreleased]、0.5.5 发布记录，并追加 PR 原有中英文条目；新增 #909 的中英文修复说明。
src/ptc/batching-guidance.ts 保留上游 shell-file-work.ts 分类器、revision 4 和最新 notice；移植 PR 的 UTF-16 可执行调用扫描、引号键、属性顺序及 command/script 顶层静态字面量提取。动态模板和拼接按上游保守策略返回 false，避免重新误报 `git diff
test/batching-trigger.test.mjs 保留上游管道、重定向和 Select-String 正反例，同时保留 PR 的字符串伪调用、emoji/UTF-16、嵌套模板、script、引号键和属性顺序测试。

#909 修复

  • src/shells/jobs.ts 只有在 callId 与 rootCallId 都是非空字符串时，才向宿主 jobs.start 传递原始 toolCall；缺任一值时省略整个字段，不猜造身份。
  • src/panel/fold.ts 的 trace 与 conversation 使用有界有序集合；处理明确的单目标 surfaceOp 替换时先从两个视图撤回旧行，再接纳替换行，因此达到 20,000 行时不会同时丢失新旧行，计数和截断标志保持原口径。
  • 新增真实 LocalJobs + Subprocess + NodePtc + ops_bash + collector 回归，覆盖根调用、PTC 子调用、WAL 配对与折叠名称；新增 19,999/20,000 容量、替换链、失败计数和 droppedChars 回归。
  • 关联新 issue：#909。issue 已提交完整复现记录并由我在评论中认领；GitHub 当前账号没有给该 issue 应用 bug 标签的权限，正文和认领不受影响。

本机验证

npm run build                                              通过
node test/run.mjs batching-guidance batching-trigger       52 项 batching 断言通过
node test/run.mjs panel-prune-fold panel-ptc-host shells-host batching-examples  6 套件、111 项检查通过
cargo test --workspace --locked --manifest-path native/Cargo.toml  672 passed, 0 failed, 1 ignored
node scripts/verify-gate.mjs --skip-install                 255 套件：1 失败、4 个允许跳过

release gate 唯一失败为 preset-registration.test.mjs 的 2 个宿主安装树解析断言；同一套件在同步后的最新 main 对照运行也失败，未触及本 PR 改动。4 个跳过为 Windows/外部 checkout 的允许项。相关 PR 套件和真实新增回归均通过。

原评论中的历史复现、旧 head CI 和范围说明如下保留。

更新（head 1951d5c）：在原四项之外补了第 5 节——宿主裁剪替换导致同一调用出现两行，已修并加回归。上面第 1～4 节仍对应当前 head。

Review 修订与本次扩展范围

本次 head 为 4e9b300，已合并上游 77cd34f（v0.5.4）。采用 merge 保留已经推送的提交历史，不再 rebase/force-push。原 CHANGELOG 冲突已解决，PR 条目保留在 [Unreleased]，上游发布记录保留。

这次除原 #700/#698 的事件采集修复外，还一并纳入下述可观测性修复，单列在提交 4e9b300；原 review 的完善为 93e2f22。这些扩展涉及原 issue 之外的文件，特此说明，不将它们冒充原 review 已批准的范围；是否接受一并合入请维护者评审。

1. 原 review 的落实

  • PTC 参数按宿主 durable JSON 值序列化后限长，字符串保持兼容。显式 null 现在记录为 null，仅缺失的 undefined 记空；之前评论中“null 记空”的说明已不适用。结果内容仅取 type: text 块。
  • tool/call 与 PTC dispatch 共用 2048 项 FIFO 的 callId→工具名索引。补了子调用 job 注册/结束、重复旧 key 不刷新 FIFO 年龄、不额外占容量、淘汰和无名 fallback 断言。
  • panel-ptc-host 现在实际启动宿主 NodePtc 执行工具，采集 executor 产生的 start/dispatch 事件，经过真实 collector 写 WAL、生成 JSON 并 fold。覆盖对象、嵌套数组、null、字符串、超长参数与明确拒绝 null 的工具失败路径。job 事件使用测试工具内产生的生命周期夹具，但 callId 取自真实执行上下文，不把它称为真实后台进程测试。
  • 另在真实宿主下对新增类别发送事件，验证 capture:false 不落盘；产品 DOM 测试执行实际 src/panel/panel.html 的脚本，使用真实投影/fold产生的六类 trace，验证恶意文本不生成危险节点或执行，并有 HTML 事件属性确实可执行的正对照。不是用 bench 的另一套模板替代验证。
  • CHANGELOG 与双语配置文档说明新增明文类别、可能包含撤回/取消/未最终接纳输入、限长不等于脱敏。

2. 本次一并纳入的可观测性修复

问题 处理
成功命令的 stderr 关键词被误判为 sandbox 启动失败 不再将用户输出作为“命令尚未执行”的证据；按真实 confinement 准备失败来源及宿主 runner 规则分类，保留权限变化和取消语义
追加写失败后报告仍显示完整 持久化 capture-failed 标记并降级报告；WAL 标记持续拒写而 JSON 可写时，从明确的 JSON 状态跨重启恢复缺口
无换行残尾吞掉恢复后的首条事件 隔离旧残尾，保留 malformed 证据，不吞掉后续正常事件
disk status 文本隐藏截断和跳过 显示 partial、跳过数量/原因/路径；不可测量的零计数不再冒充 missing
面板旧请求覆盖用户最新选择 代际检查覆盖 fetch、响应体、失败与 FileReader；迟到索引不覆盖手动文件
字符串中的工具示例被当成真实调用 只计可执行位置，保留括号调用和模板表达式；修正 Unicode 索引偏移
shell description 等文案触发文件绕过误报 只检查实际 shell 调用顶层 command 的静态字面片段，动态变量不猜测

边界：若 WAL 和 JSON 都持续不可写且进程退出，没有持久化通路可保存零写缺口；不宣称能解决这一物理限制。未新增运行时依赖、未改权限策略；相对合入后的 main，本 PR 不改版本/依赖声明。

3. 对之前验证表述的更正

  • 原先称作“全链路”的纯 fixture 实际没有经过 collector，现已修正命名并增加上面的真实落盘测试。
  • 历史日志取证只证明实际采样事件的形态，不等于在测试里调用 NodePtc；本次才加入真实调用回归。
  • 单独重跑通过不能证明并发失败与改动无关，不再用“与 panel 无关”或简单 grep 无 AssertionError 来宣称全量无回归。下面以最终 head 的实际套件结果为准。

4. Validation

本机：Windows x64，Node v24.15.0；本地源码构建，worker 0.5.4 使用本机 Rust 1.90.0 构建（不是 CI 固定的 1.97.1），已 stage 并校验；Niubash 1.5.0 使用固定 archive。不会把此验证替代跨平台发布门禁。

本次推送已核对：远端 head=4e9b300a2d849a3c10ad252ce552b80a577e07e2，GitHub 显示 MERGEABLE（无冲突）。

$ node scripts/build.mjs
build: compiled src/ to lib/ and copied 60 assets
$ node node_modules/typescript/bin/tsc -p . --noEmit
# exit 0
$ node scripts/check-file-size.mjs
check-file-size: 819 code file(s) checked, all at or under 1000 lines
$ node scripts/validate-manifest.mjs
validate-manifest: all 92 checks passed
$ node test/run.mjs panel disk-status disk-observability batching-trigger shells-host
dsh-ops test: 16 suite(s) passed in 2s

本机全量在相同 head 上执行，设置 DSH_OPS_TEST_JOBS=4、DSH_OPS_SUITE_TIMEOUT_MS=300000、DSH_OPS_REQUIRE_RUNTIME=1，提供固定 Niubash archive 和 niu.exe 路径，仅允许 Linux 与外部 TUI 四个具名跳过：

dsh-ops test: 247 suites, 4 worker(s); exclusive suites run afterwards
dsh-ops test: 1 of 247 suite(s) failed, 4 skipped in 216s
# 242 passed + 1 failed + 4 skipped = 247; exit 1

唯一失败为 niubash-consumer：Ambiguous packed Niubash member: winuxcmd/usr/bin/gawk.exe（真实 tar Link 条目与仅接受 File 的校验冲突）。该打包失败在此次扩展前已观察到，相关打包实现本次未改，但本机全量仍是失败，不将其称为全绿或简单归为环境问题。跳过项为 linux-attributes、linux-shell、tui-host、tui-native；disk-status 既有 Windows symlink/权限夹具分支不可用亦未算作实际覆盖。

行为级 red→green：旧实现的 null 保真断言及真实 NodePtc 持久化测试失败，修复后通过；采集故障、磁盘 partial、请求乱序、stderr 误报、批量提醒均有旧 lib 失败后对应新实现通过的回归。真实 Niubash 安全套件本次全量 11/11 通过。

新 head 的运行：Cross-platform validation / 43015803373，event=pull_request，head=4e9b300。查询时 in_progress，不是已通过；以这次运行的 CI required 最终结论为准。旧 head d221a2e 的 CI success 不替代本次验证。

5. 新发现：宿主裁剪替换让同一调用出现两行（已修，commit 1951d5c）

这一条不在原 issue 的四个事件范围内，是我在做真实采集验证时发现的独立缺陷，一并修掉。

现象：长会话里同一个 callId 在面板 trace 上出现两条并列结果，toolErrors/droppedChars 也各算两次。

根因（两层，都不是采集器的原始意图）：

  • 宿主 @deepseek-ai/dsh-compaction-tool-result-pruner 在压缩时会把超预算（默认 8192 字符）的旧工具结果替换成一条新事件：session.append("tool/result", …, { surfaceOp: { op: "replace", startSeq: N, endSeq: N }, sourceEventSeqs: [N] })。原始那条仍留在日志里，新的一条才是模型看到的裁剪版。pruneSession 的唯一调用方是 dsh-compaction-basic，所以只有发生压缩的长会话才会出现。本机采样：compaction/prune 41 条 ↔ surfaceOp: replace 41 条 ↔ 面板同 callId 两行 41 对，逐一对应。
  • 采集器在 projectEvent 里只投影了 callId/isError/payload，把事件顶层的 surfaceOp/sourceEventSeqs 丢掉了；折叠时无从判断"第二条是替换"，于是照原样画两行。
  • 之所以肉眼看不出来：textLimit 只保留前缀，裁剪标记 [... tool result middle pruned ...] 位于内容中部（实测真实内容 5443 字符），被截掉后两行前缀完全相同。

修法（src/panel/events.ts、src/panel/fold.ts、src/panel/types.ts）：

  • 只保留单目标替换元数据（op === "replace" 且 startSeq === endSeq）；区间替换（压缩摘要改写一段）维持原行为，op 结构非法则记为 malformed，不信任。
  • 折叠时撤回被替换的行，给替换行标 pruned: true 与 supersedes: N，并在面板上显示"已裁剪（替换 #N）"；同时回退原行的 toolErrors/droppedChars 贡献。
  • 计数口径保持诚实：events.types 仍是宿主实际发出的每条事件（2 条就记 2 条），metrics.eventCount 不变，只有结果行视角合并；limits.pruneReplacements 记录撤回次数。
  • 结果行索引有界（20 000），超出后更早的结果不再撤回——宁可多画一行，也不无界增长。
  • 对该元数据出现之前写下的采集保持两行：只凭重复的 callId 不能证明发生过替换，因此不做推断性删除，旧日志由使用者自行清理（删 JSON 与 WAL 两份，只删 JSON 会在下次 turn/end 被重折回来）。

验证（本机真实会话，同一份宿主事件，只切换是否带 surfaceOp）：

指标 无元数据（修复前） 有元数据（修复后）
结果行 612 571
唯一 callId 571 571
重复行 41 0
toolErrors 24 23
limits.pruneReplacements 0 41

新增 test/panel-prune-fold.test.mjs（8 项断言：只画一行并撤回原行、计数口径、失败替换不双计、继承 name/duration、区间替换不动、旧采集保留两行不猜、只投影单目标 replace 且非法值算 malformed、真实宿主记录形状）。

本机门禁：build / tsc --noEmit 通过、check-file-size 819 文件、validate-manifest 92/92、node test/run.mjs panel 13 套件全绿、全量 247/248 通过、1 失败、4 跳过。唯一失败是 niubash-consumer 的 Ambiguous packed Niubash member: gawk.exe——本机 payload 为硬链接布局（awk/gawk/winuxcmd 共享 inode）导致 npm pack 写出 tar 硬链接条目，而校验器只接受普通文件条目；与本次改动无关（该套件不引用 panel 代码），CI 上 validate (win32-x64) 为 green。

CI：新 head 1951d5c 的 pull_request 运行已触发：https://github.com/T-Auto/dsh-ops/actions/runs/43032869348

@LOKOP10
merge: integrate upstream v0.5.4 into panel event capture
692ec9f
@LOKOP10
fix(panel): preserve null arguments and verify real capture paths
93e2f22 
Preserve explicit JSON null and retain only text result blocks. Add real NodePtc-to-collector persistence, capture-off, FIFO and production viewer DOM safety regressions. Document plaintext retention and withdrawn input in English and Chinese.
@LOKOP10
fix(observability): preserve capture gaps and correct diagnostic repo…
4e9b300 
…rting
@LOKOP10
fix(panel): draw a host-replaced tool result once
1951d5c 
Keep the host's single-target `surfaceOp` replacement in the captured record and
withdraw the row it superseded while folding, so a pruned tool result is read once
instead of twice. `toolErrors` and `droppedChars` stop counting the same result
twice, `events.types` still counts every event the host emitted, and
`limits.pruneReplacements` names how many withdrawals happened.

A capture written before that metadata existed keeps both rows: a repeated call id
alone is not proof that a replacement happened.
@LOKOP10 LOKOP10 mentioned this pull request Nov 13, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[BUG] ops_bash 作业缺少调用关联且 PR #700 满额替换丢行 [BUG] panel capture drops the fields four host events already carry

2 participants

@LOKOP10 @T-Auto