Repository navigation
fix(benchmark): preserve nonfinite results across local JSON boundaries - #7771
Conversation
Signed-off-by: song <liusongstep@gmail.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 46edc08a2e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
|
Integrated main at The post-merge publication/report suite passed 100 tests, and native offline coverage passed 29 tests. The optional FastAPI import guard remains present; the relevant modules were run with their actual provider dependencies. The merge was conflict-free. The changed production/test blobs from the prior PR head remain identical; where needed, the documentation merge was inspected separately. RFC bilingual and diff hygiene checks passed. The API-published tree matches the locally tested tree. Earlier browser CI failures remain recorded as failures; these focused checks do not turn them into passes or establish a new CI result. Required checks and independent review must qualify this new head. No merge was performed. |
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max
APPROVE — 未发现阻塞问题。 本次完整审查 head 1109cb5d971bbcc6cbacb8096416e9fe2ab760a5,未沿用旧 head 的结论。
动机 / Motivation
离线回填结果或保存可用评测机会的维护者会遇到该问题。例如原生评测返回 Infinity 时,旧离线写文件和 HTTP 响应均失败;新版本保存并读回原值。同一配对探针确认六个非有限数失败转为成功,有限结果文件及读取结果保持一致。本 PR 不改变原生评分、选择策略、机会预算或恢复执行授权。 修复覆盖原来的命令、保存文件、重启读取和机会捕获,不要求操作者增加配置或确认步骤。
改动思路 / Approach
两个失效边界共享一个 provider-local codec,既不借用原生 checkpoint authority,也不引入第二个评分或状态 owner。本次边界是离线文件与 opt-in 机会读取直到惰性 artifact 回读。 把数值改成零、删除字段或直接允许非标准 JSON 常量都会破坏原值或 HTTP 边界;现有原生 checkpoint 私有 codec 又绑定另一份 manifest 和授权协议。因此,在研究适配层保留一个小型公共实现是合理的。原生 selector、admission/history locks 和现有 typed artifact owner 继续负责各自规则。
| 输入 | baseline → 本 head 的实际观察 |
|---|---|
| 有限 JSON | 离线结果文件逐字节相同,HTTP 客户端逻辑结果相同 |
| ±Inf / NaN | 3 个离线失败、3 个实际 HTTP 失败均变成成功,原生状态和源归档不变 |
| 非法路径 | 重复、缺失、非 null 目标及 authority/clock/counter 编码均拒绝 |
具体改动 / Changes
验收依据是改动前 benchmark/README.md; benchmark/edgebench/README.md,spec_revision 为完整提交 2f2c4efadf0c3ac21c51f7fc09f4985c8447642e;新写的说明不充当独立规范。逐项结论:offline-provenance、native-selection、released-read-only、bound-incumbent、inert-artifact 均 implemented,分别由离线来源/重启、原生多策略选择、只读 HTTP、归档绑定和实际 CLI/SQLite 惰性物化用例支持。
关键代码讲解
sforge_json.encode_document/decode_document:有限数据保持旧表示;非有限数用 null 与明确的 path/kind 表表达。根 marker 会转义且只解码一次,嵌套同形 metrics 保持普通数据。严格读取同时拒绝重复 JSON key、未知 kind、bool/负索引及未标记的溢出常量。offline_scoring.score_captures:现有来源和 evaluator digest 检查仍在;保存与重启共用 codec,随后交给原生select_best。NaN 处于第一条时的原生比较行为也保留;损坏缓存不会重新评分或覆盖既有汇总。checkpoint_judge.snapshot_locked/sforge_checkpoint.capture_opportunity:只允许released_history、released_results编码数值。规范化编码进入原绑定摘要;NaN 重读和 object key 换序维持绑定,真实结果变化则拒绝旧归档请求。客户端先严格解码,再走原有下载和截止时间校验。checkpoint.capture_after_wake:保存参与文件时也使用同一 codec,再由原 artifact owner 捕获;实际 SQLite 回读保留 Infinity、-Infinity 和 NaN,物化仍不授予执行权限。
本 diff 为 5 个生产文件、4 个聚焦测试文件和 2 份说明;119 行 codec 服务所有现有调用者,没有新增评分器、策略或协调状态。
对主干的风险 / Risks
最重要的反例是 marker 与任务自有 metrics 冲突,或结果编码越过边界改变时钟、轮次或授权。除 literal/collision、重启和绑定篡改测试外,我另经实际 HTTP 向这三个根字段注入 NaN 编码:均在下载前拒绝,归档请求为零。机会入口继续默认关闭;native route parity 与 hook 关闭/失败时零 IO 都执行通过。离线传输修复本身是普通路径的显式行为修复,没有宣称整份 PR 都默认关闭。
PASS:原生 provider/FastAPI 下 93 项;实际 CLI/SQLite 捕获与关闭路径 8 项;默认 test-extra 28 项通过、15 项可选依赖跳过;Ruff、编译及全部 11 文件公开边界扫描通过。首次物化验证因归档目录缺少 Git 身份在原有 workspace 校验前置拒绝;同一 source 补齐真实 HEAD/index 后 8 项通过,未改生产字节、测试或门禁。首次配对 HTTP 探针的临时路径未规范化,两臂同样被拒绝;只修正 harness 路径后重跑。原失败均保留。
语义与 CI 对齐
新 kind/marker 是 provider-local 序列化词汇,未扩展通用控制面协议。按当前 review policy 25 使用仓库原生本地验证,未查询、等待或触发 CI。实际评测、容器运行、隐藏任务、模型行为和已安装 App 旅程均未宣称验收;本地原生 provider 固定在 c77e71b8b58fff97c176ab96e55faac5a87d2d93。作者在新 head 的补充说明已给出新树验证;建议把 PR 正文仍指旧树的段落一起更新,属于非阻塞说明整理。
我的整体评价 / Verdict
该修复完成了两个本地传输缺陷对应的可用路径,规模与影响相称。long_horizon improved:重启直接读回原结果,不靠重评推进;user_experience improved:原命令继续可用,无新增人工准备或确认回合。有限数据 parity、无权限变化和历史读者保留已有证据。本次仅改变 JSON 传输与私有文件回读,未改变渲染页面。无受影响的渲染表面。保留 decoder 是持久产物的读取义务;没有发现需要继续拆分或增加框架的理由。批准结论不代替独立 merge readiness,也不授予合并权限。
English verdict: APPROVE 1109cb5d971bbcc6cbacb8096416e9fe2ab760a5. Both local nonfinite transport defects are repaired; finite parity, real HTTP and SQLite artifact readback are verified. No blocking finding; native scoring and authority remain unchanged.
Goal And Delivered Outcome
Native SForge reports can contain
Infinity,-InfinityorNaNin a score or nested metric. Two local adapters could not preserve those legitimate native results: offline scoring failed while writingscore.json/result.json, and the opt-in opportunity endpoint failed during FastAPI response serialization. The former stranded a completed scoring pass; the latter prevented the actual checkpoint client from preserving an otherwise reconciled registration.This PR carries those values through both complete local paths without changing native grading or selection. The same synthetic baseline/head probe preserves both finite observations and changes all six nonfinite boundary cases from failure to success.
loopx-project/loopx:main, starting at46f1175fbbb67bf2d0d358f62efc8ade0652bbbc.huangruiteng/EdgeBench@e278cda3f4d2da3cf06aabf2df4daa99b9749501.Author Declaration
Implemented against
Specification:
benchmark/README.mdandbenchmark/edgebench/README.mdat the base revision above. Their native-owner and checkpoint contracts preserve released results, native selection, consumed opportunities, immutable source evidence and original deadlines. The reproduced boundary defects are the basis for the representation repair.offline_scoring.score_capturescheckpoint_judge,sforge_checkpointruntime.checkpoint.capture_after_wakeImplementation And Scope
The local
benchmark.runtime.sforge_jsoncodec keeps ordinary finite JSON unchanged. Documents with nonfinite numbers carry an explicitsforge_nonfinite_json: 1envelope: thevalueobject has null placeholders, and a separate path table records each numeric kind. Literal tag-shaped metric objects remain data; a literal root transport marker is escaped and decoded exactly once. Strict readback rejects duplicate keys, unmarked constants and overflow, malformed paths, duplicate targets and non-null targets.Offline score files, the final result and CLI summary use that codec. Restart decodes each saved report before the unchanged native selector receives it. A nonfinite report remains a completed native report, including the native first-entry NaN comparison behavior.
The opportunity response, HTTP client and saved checkpoint participant use the same transport. Only
released_historyandreleased_resultsmay contain encoded numbers; clocks, round counters and authority fields are outside that allowance. The archive binding covers canonical encoded state. Repeated NaN reads and equivalent object-key order retain the same binding; changed released data invalidates the previous archive request. The client restores logical numeric values, and the checkpoint writer saves them with the same codec before immutable artifact capture.This completes the two local transport repairs. Source-default selection and the native provider compatibility patch are separate owners and are unchanged. There is no new scoring policy, native manifest format, model behavior, admission operation, budget or deadline setting. The shared codec is a bounded refactor across active Python provider-transport callers; it adds no control-plane decision owner. The existing worker payload already includes
benchmark/runtime.Validation
fb6dd729ec22c6394e6e0b0baf7f414073b90a4b; the focused suites and repeated paired probe exercised those exact bytes. Final published commit1493936d080d5d5299a70d93f23bf11f1a5db65fhas treeab9c2fbccf5ea80a1203075c972924c647b98f50. Its only follow-up difference is the optional FastAPI test guard described below; both missing-dependency and complete native HTTP runs were repeated on that final tree.python -m pytest -q benchmark/tests/test_sforge_json.py benchmark/tests/test_edgebench_online.py -k 'sforge_json or offline': 55 cases, including CLI summary, restart and malformed-cache behaviorpython -m pytest -q benchmark/tests/test_checkpoint_judge.py: 17 cases, including actual loopback HTTP through the native FastAPI owner tocapture_opportunity; grading inputs are synthetic, with no Docker jobspython -m pytest -q tests/control_plane/test_research_checkpoint_capture.py -k real_cli_settlement_capture_verification_and_inert_materialization: 7 cases using the public CLI and isolated SQLite/TypeScript artifact owner; capture preserves the encoded participant through inert materializationModuleNotFoundError: fastapiin the default test-extra environment.pytest.importorskip("fastapi")now matches the existing controller tests: this file has 2 passed / 15 skipped without the optional native stack and 17 passed with the complete stack. The guard changes no runtime behavior or dependency declaration.git diff --check, bilingual RFC diff check, and the repository public-boundary scanner over all 11 proposed filesCoverage limits: the semantic-inventory advisory supports
loopx/**and rejects explicitly suppliedbenchmark/**paths, so it cannot certify this local transport vocabulary. The numeric kinds and envelope were reviewed manually as a provider-local serialization contract. The public-boundary scan reported an absent local registry, while its candidate-file scan was clean. FastAPI emitted its existing TestClient deprecation warning; the real HTTP cases also passed.The fixtures contain arbitrary synthetic candidates and nested values, including literal tags, missing values and invalid encodings. There is no benchmark/task-name special case, hidden evaluator content, threshold change, zero substitution or dropped result. Preserving a native nonfinite result does not establish scorer integrity, statistical validity or a research benefit.
Frontend / Visual Evidence
Type Of Change
LoopX Area
Shared-authority RFC Fixture Impact
N/A. No shared-authority schema, provider routing or core state projection changes.
Boundary Checklist
Signed-off-bytrailer.