Skip to content

fix(benchmark): preserve nonfinite results across local JSON boundaries - #7771

Merged
loopx-agent merged 3 commits into
loopx-project:mainfrom
songoow:codex/edgebench-nonfinite-json
Nov 1, 2026
Merged

loopx-agent merged 3 commits into
loopx-project:mainfrom
songoow:codex/edgebench-nonfinite-json

Conversation

@songoow

@songoow songoow commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

Native SForge reports can contain Infinity, -Infinity or NaN in a score or nested metric. Two local adapters could not preserve those legitimate native results: offline scoring failed while writing score.json/result.json, and the opt-in opportunity endpoint failed during FastAPI response serialization. The former stranded a completed scoring pass; the latter prevented the actual checkpoint client from preserving an otherwise reconciled registration.

This PR carries those values through both complete local paths without changing native grading or selection. The same synthetic baseline/head probe preserves both finite observations and changes all six nonfinite boundary cases from failure to success.

  • Outcome basis: reproduced serialization defects in the shipped research adapters.
  • Intended base: loopx-project/loopx:main, starting at 46f1175fbbb67bf2d0d358f62efc8ade0652bbbc.
  • Native provider used for local validation: huangruiteng/EdgeBench@e278cda3f4d2da3cf06aabf2df4daa99b9749501.

Author Declaration

  • Written by: model_agent, GPT-6 Astra Pro / OpenAI.

Implemented against

Specification: benchmark/README.md and benchmark/edgebench/README.md at the base revision above. Their native-owner and checkpoint contracts preserve released results, native selection, consumed opportunities, immutable source evidence and original deadlines. The reproduced boundary defects are the basis for the representation repair.

Criterion Disposition Owner / path Evidence
Preserve native task, grading and selection semantics implemented offline_scoring.score_captures Original selector runs on decoded numbers; all three policies, both directions and finite/±Inf/NaN cases
Resume saved offline results without silently regrading implemented Offline score-file readback Repeated scoring reads saved reports; malformed cached transport rejects without regrading or rewriting the previous summary
Preserve complete released opportunity state and bound incumbent implemented checkpoint_judge, sforge_checkpoint Actual HTTP server/client round trips, stable repeated bindings, changed-result rejection and unchanged archive bytes
Keep capture inert and preserve source authority implemented runtime.checkpoint.capture_after_wake Public CLI, SQLite artifact capture, verification and inert materialization

Implementation And Scope

The local benchmark.runtime.sforge_json codec keeps ordinary finite JSON unchanged. Documents with nonfinite numbers carry an explicit sforge_nonfinite_json: 1 envelope: the value object has null placeholders, and a separate path table records each numeric kind. Literal tag-shaped metric objects remain data; a literal root transport marker is escaped and decoded exactly once. Strict readback rejects duplicate keys, unmarked constants and overflow, malformed paths, duplicate targets and non-null targets.

Offline score files, the final result and CLI summary use that codec. Restart decodes each saved report before the unchanged native selector receives it. A nonfinite report remains a completed native report, including the native first-entry NaN comparison behavior.

The opportunity response, HTTP client and saved checkpoint participant use the same transport. Only released_history and released_results may contain encoded numbers; clocks, round counters and authority fields are outside that allowance. The archive binding covers canonical encoded state. Repeated NaN reads and equivalent object-key order retain the same binding; changed released data invalidates the previous archive request. The client restores logical numeric values, and the checkpoint writer saves them with the same codec before immutable artifact capture.

This completes the two local transport repairs. Source-default selection and the native provider compatibility patch are separate owners and are unchanged. There is no new scoring policy, native manifest format, model behavior, admission operation, budget or deadline setting. The shared codec is a bounded refactor across active Python provider-transport callers; it adds no control-plane decision owner. The existing worker payload already includes benchmark/runtime.

Validation

  • Primary validation tree: fb6dd729ec22c6394e6e0b0baf7f414073b90a4b; the focused suites and repeated paired probe exercised those exact bytes. Final published commit 1493936d080d5d5299a70d93f23bf11f1a5db65f has tree ab9c2fbccf5ea80a1203075c972924c647b98f50. Its only follow-up difference is the optional FastAPI test guard described below; both missing-dependency and complete native HTTP runs were repeated on that final tree.
  • Run state: finished.
  • Input classes: synthetic.
Check kind Result Public-safe evidence / limitation
unit / integration passed python -m pytest -q benchmark/tests/test_sforge_json.py benchmark/tests/test_edgebench_online.py -k 'sforge_json or offline': 55 cases, including CLI summary, restart and malformed-cache behavior
real_entrypoint / integration passed python -m pytest -q benchmark/tests/test_checkpoint_judge.py: 17 cases, including actual loopback HTTP through the native FastAPI owner to capture_opportunity; grading inputs are synthetic, with no Docker jobs
real_entrypoint / real_backend passed python -m pytest -q tests/control_plane/test_research_checkpoint_capture.py -k real_cli_settlement_capture_verification_and_inert_materialization: 7 cases using the public CLI and isolated SQLite/TypeScript artifact owner; capture preserves the encoded participant through inert materialization
regression_parity passed One identical synthetic harness at base and head: the finite offline result file is byte-identical and finite HTTP-client observations match; ±Inf/NaN fail at each baseline boundary and pass at head, with unchanged registration authority, deadlines, rounds and original source bytes
optional_dependency / regression passed The standalone malformed-response test reproduced ModuleNotFoundError: fastapi in the default test-extra environment. pytest.importorskip("fastapi") now matches the existing controller tests: this file has 2 passed / 15 skipped without the optional native stack and 17 passed with the complete stack. The guard changes no runtime behavior or dependency declaration.
static passed Ruff on the nine changed Python files, git diff --check, bilingual RFC diff check, and the repository public-boundary scanner over all 11 proposed files
manual passed Full caller and consumer trace, finite compatibility, codec collision/rejection cases, source-preserving bindings and overfitting assessment
integration not_run Containerized grading, paid models, hidden tasks, historical cohort replay and the complete repository CI suite were not run locally; this PR makes no benchmark outcome or performance claim

Coverage limits: the semantic-inventory advisory supports loopx/** and rejects explicitly supplied benchmark/** paths, so it cannot certify this local transport vocabulary. The numeric kinds and envelope were reviewed manually as a provider-local serialization contract. The public-boundary scan reported an absent local registry, while its candidate-file scan was clean. FastAPI emitted its existing TestClient deprecation warning; the real HTTP cases also passed.

The fixtures contain arbitrary synthetic candidates and nested values, including literal tags, missing values and invalid encodings. There is no benchmark/task-name special case, hidden evaluator content, threshold change, zero substitution or dropped result. Preserving a native nonfinite result does not establish scorer integrity, statistical validity or a research benefit.

Frontend / Visual Evidence

  • UI impact: none. This changes research-adapter JSON and private artifact readback, with no visual surface or first-screen change.

Type Of Change

  • Bug fix
  • Documentation update
  • Test update

LoopX Area

  • Benchmark boundary (adapters, runners, verifiers, scoring, evidence)

Shared-authority RFC Fixture Impact

N/A. No shared-authority schema, provider routing or core state projection changes.

Boundary Checklist

  • The diff and PR description contain no private state, credentials, raw task data, trajectories, verifier output, internal links or local machine paths.
  • Validation used public-safe synthetic data and did not run maintainer-owned benchmark tasks.
  • The change is scoped to the reproduced local transport defects.
  • No paired RFC changed; the bilingual diff check passes.
  • UI impact is none.
  • The commit includes a DCO Signed-off-by trailer.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T02:09:01.709278Z 46edc08 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 46edc08a2e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>

songoow commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

Integrated main at 2f2c4efadf0c3ac21c51f7fc09f4985c8447642e with a signed merge that preserves the existing PR history. Current head: 1109cb5d971bbcc6cbacb8096416e9fe2ab760a5; tested tree: 597d31574352908152e6500afa49ff63d50510f8.

The post-merge publication/report suite passed 100 tests, and native offline coverage passed 29 tests. The optional FastAPI import guard remains present; the relevant modules were run with their actual provider dependencies.

The merge was conflict-free. The changed production/test blobs from the prior PR head remain identical; where needed, the documentation merge was inspected separately. RFC bilingual and diff hygiene checks passed. The API-published tree matches the locally tested tree.

Earlier browser CI failures remain recorded as failures; these focused checks do not turn them into passes or establish a new CI result. Required checks and independent review must qualify this new head. No merge was performed.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max

APPROVE — 未发现阻塞问题。 本次完整审查 head 1109cb5d971bbcc6cbacb8096416e9fe2ab760a5,未沿用旧 head 的结论。

动机 / Motivation

离线回填结果或保存可用评测机会的维护者会遇到该问题。例如原生评测返回 Infinity 时,旧离线写文件和 HTTP 响应均失败;新版本保存并读回原值。同一配对探针确认六个非有限数失败转为成功,有限结果文件及读取结果保持一致。本 PR 不改变原生评分、选择策略、机会预算或恢复执行授权。 修复覆盖原来的命令、保存文件、重启读取和机会捕获,不要求操作者增加配置或确认步骤。

改动思路 / Approach

两个失效边界共享一个 provider-local codec,既不借用原生 checkpoint authority,也不引入第二个评分或状态 owner。本次边界是离线文件与 opt-in 机会读取直到惰性 artifact 回读。 把数值改成零、删除字段或直接允许非标准 JSON 常量都会破坏原值或 HTTP 边界;现有原生 checkpoint 私有 codec 又绑定另一份 manifest 和授权协议。因此,在研究适配层保留一个小型公共实现是合理的。原生 selector、admission/history locks 和现有 typed artifact owner 继续负责各自规则。

输入 baseline → 本 head 的实际观察
有限 JSON 离线结果文件逐字节相同,HTTP 客户端逻辑结果相同
±Inf / NaN 3 个离线失败、3 个实际 HTTP 失败均变成成功,原生状态和源归档不变
非法路径 重复、缺失、非 null 目标及 authority/clock/counter 编码均拒绝

具体改动 / Changes

验收依据是改动前 benchmark/README.md; benchmark/edgebench/README.md,spec_revision 为完整提交 2f2c4efadf0c3ac21c51f7fc09f4985c8447642e;新写的说明不充当独立规范。逐项结论:offline-provenance、native-selection、released-read-only、bound-incumbent、inert-artifact 均 implemented,分别由离线来源/重启、原生多策略选择、只读 HTTP、归档绑定和实际 CLI/SQLite 惰性物化用例支持。

关键代码讲解

  • sforge_json.encode_document / decode_document:有限数据保持旧表示;非有限数用 null 与明确的 path/kind 表表达。根 marker 会转义且只解码一次,嵌套同形 metrics 保持普通数据。严格读取同时拒绝重复 JSON key、未知 kind、bool/负索引及未标记的溢出常量。
  • offline_scoring.score_captures:现有来源和 evaluator digest 检查仍在;保存与重启共用 codec,随后交给原生 select_best。NaN 处于第一条时的原生比较行为也保留;损坏缓存不会重新评分或覆盖既有汇总。
  • checkpoint_judge.snapshot_locked / sforge_checkpoint.capture_opportunity:只允许 released_history、released_results 编码数值。规范化编码进入原绑定摘要;NaN 重读和 object key 换序维持绑定,真实结果变化则拒绝旧归档请求。客户端先严格解码,再走原有下载和截止时间校验。
  • checkpoint.capture_after_wake:保存参与文件时也使用同一 codec,再由原 artifact owner 捕获;实际 SQLite 回读保留 Infinity、-Infinity 和 NaN,物化仍不授予执行权限。

本 diff 为 5 个生产文件、4 个聚焦测试文件和 2 份说明;119 行 codec 服务所有现有调用者,没有新增评分器、策略或协调状态。

对主干的风险 / Risks

最重要的反例是 marker 与任务自有 metrics 冲突,或结果编码越过边界改变时钟、轮次或授权。除 literal/collision、重启和绑定篡改测试外,我另经实际 HTTP 向这三个根字段注入 NaN 编码:均在下载前拒绝,归档请求为零。机会入口继续默认关闭;native route parity 与 hook 关闭/失败时零 IO 都执行通过。离线传输修复本身是普通路径的显式行为修复,没有宣称整份 PR 都默认关闭。

PASS:原生 provider/FastAPI 下 93 项;实际 CLI/SQLite 捕获与关闭路径 8 项;默认 test-extra 28 项通过、15 项可选依赖跳过;Ruff、编译及全部 11 文件公开边界扫描通过。首次物化验证因归档目录缺少 Git 身份在原有 workspace 校验前置拒绝;同一 source 补齐真实 HEAD/index 后 8 项通过,未改生产字节、测试或门禁。首次配对 HTTP 探针的临时路径未规范化,两臂同样被拒绝;只修正 harness 路径后重跑。原失败均保留。

语义与 CI 对齐

新 kind/marker 是 provider-local 序列化词汇,未扩展通用控制面协议。按当前 review policy 25 使用仓库原生本地验证,未查询、等待或触发 CI。实际评测、容器运行、隐藏任务、模型行为和已安装 App 旅程均未宣称验收;本地原生 provider 固定在 c77e71b8b58fff97c176ab96e55faac5a87d2d93。作者在新 head 的补充说明已给出新树验证;建议把 PR 正文仍指旧树的段落一起更新,属于非阻塞说明整理。

我的整体评价 / Verdict

该修复完成了两个本地传输缺陷对应的可用路径,规模与影响相称。long_horizon improved:重启直接读回原结果,不靠重评推进;user_experience improved:原命令继续可用,无新增人工准备或确认回合。有限数据 parity、无权限变化和历史读者保留已有证据。本次仅改变 JSON 传输与私有文件回读,未改变渲染页面。无受影响的渲染表面。保留 decoder 是持久产物的读取义务;没有发现需要继续拆分或增加框架的理由。批准结论不代替独立 merge readiness,也不授予合并权限。

English verdict: APPROVE 1109cb5d971bbcc6cbacb8096416e9fe2ab760a5. Both local nonfinite transport defects are repaired; finite parity, real HTTP and SQLite artifact readback are verified. No blocking finding; native scoring and authority remain unchanged.

@loopx-agent
loopx-agent merged commit 32f4deb into loopx-project:main Nov 1, 2026
15 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants