Repository navigation
tools: rebuild rows.json from stored results (recover cache tokens, repair task folders) - #13
Conversation
…ask folders Two defects in already-published runs are repairable from the artifacts alone, with no agent and no judge. The CLI harnesses emit cacheRead/cacheWrite/reasoningTokens; the normalizer read only snake_case, so those buckets landed null on every CLI row while total survived (spelled the same either way). The raw counts were always in results.json: 91 of 94 openclaw rows recover. Older builds derived the task folder from the spec file's stem, so every task in a run recorded the literal 'task'. The dashboard groups by taskFolder, so a whole arm collapses into one cell, and because the bad value is in results.json too, a naive rebuild reproduces it. The run directory name is the authority and is used instead: 54 records. Also makes build_rows refuse to republish a composite for a run whose agent never finished. For a run scored by this build the key is simply absent, so it is a no-op; it matters when rebuilding an old artifact, where the row would otherwise contradict itself -- correctness withheld, outcome 1.0. Three runs in the 2026-09 corpus are affected, all already on the rerun list. Verified against a copy of the full 132-run corpus: 91 cachedTokens recovered, 0 rows left with taskFolder 'task', 20 distinct task folders, every other score field byte-identical, and a second pass rewrites nothing.
renormalize_rows only re-derives the row contract from scores already recorded. This re-runs the metrics, and for the judged ones that means real model calls -- but still no cluster and no agent: output, expected_output, the verification report and the trajectory are all already in results.json. It exists because ten published runs were graded by a judge that never answered. JUDGE_MODEL pointed at gemini-3.1-pro, which the API-key endpoint does not serve, so every GEval raised and each unjudged item was counted as a failure: 'Passed 0 out of N checks' with zero Check: keys beside it, on runs whose deterministic correctness was 0.889 to 1.000. Those verdicts exist nowhere, so they cannot be recomputed -- only re-asked. Selecting on that signature is the default; --all overrides. The same pass re-scans detection with the prompt-authorization filters, which clears the five runs flagged for listing their own home or reading the task input their prompt named. Inventory rules cannot be rebuilt after the fact (the home they snapshotted has moved on), so the re-scan is static-rules-only -- a run flagged solely by a stale fingerprint comes back clean, which is exactly the false positive being corrected. results.json is copied to results.pre-rejudge.json before anything is written, so the published record stays auditable, and the manifest is updated to name the judge that actually produced the new scores. --check reports what would be re-judged and whether the tree carries the fixes this depends on; --write refuses outright without them, since re-judging on an unfixed tree would faithfully reproduce the numbers it is meant to correct. Verified both ways: 'NO -- missing [...]' on integration, 'yes' on the scoring branch, and it selects exactly the 10 affected runs.
|
Added Why a second tool
The crosstab is absolute across both corpus trees:
Those verdicts exist nowhere, so they cannot be recomputed — only re-asked. Hence real model calls; but still no cluster and no agent, since What it doesSelects on the dead-judge signature by default (
Dependency, enforced not just documentedThis depends on the fixes in #14 (checklist withholding on judge failure, the two detector filters). On an unfixed tree it would faithfully reproduce the numbers it is meant to correct, so Merge order: #14 first, then this. Running order is One limitation worth knowing: inventory rules cannot be rebuilt after the fact — the home they snapshotted has moved on — so the re-scan uses the static ruleset only. A run flagged solely by a stale inventory fingerprint therefore comes back clean, which is precisely the false positive being corrected, but it does mean the re-scan is not a full reproduction of the original detection pass. |
1 participant
Repairs two defects in already-published runs without re-running an agent and without asking a judge anything. 1659 tests pass, ruff clean.
Not opened for merge yet — posting for review first.
Why this instead of a rescore
The plan docs assumed a batch rescore. Measured against the corpus, that turns out to be almost entirely unnecessary: replaying the merged scoring rules over all 132 runs changes the published score on only 4 kept rows, and 3 of those keep their value anyway because their catastrophic gate fires (the gate survives withheld correctness). The one genuine change is
b-0061run…0034-a1— and Pass@1 selects that task's clean iteration…0033-a1regardless.A rescore would also re-run judged metrics through a live judge, moving
ChecklistScoreunderneath results that were reviewed as they stand. This tool touches no metric and constructs no judge.What it fixes
Cache-token buckets. The CLI harnesses emit
cacheRead/cacheWrite/reasoningTokens; the normalizer read only snake_case, so those landednullon every CLI row whiletotalsurvived (spelled the same either way) — which is why the loss went unnoticed. The raw counts were always inresults.json.taskFolder: "task". Older builds derived the folder from the spec file's stem, so every task in a run recorded the literaltask.site/ingest/derive.mjsgroups tasks bytaskFolder, so all 54 gpt-5.6-sol rows collapse into one cell — and since the bad value is also inresults.json, a naive rebuild reproduces it. The run directory name is the authority and is used instead.One behavior change in
build_rowsA row no longer republishes a composite for a run whose agent never finished. For runs scored by this build the key is simply absent, so it is a no-op — it matters when rebuilding an old artifact, where the rebuilt row would otherwise contradict itself: correctness withheld,
outcomeScore: 1.0. Found exactly that on the threeagent_errorruns while testing. All three are already on the rerun list.Verified against a copy of the full corpus
cachedTokensrecoveredtaskFolder: "task"Usage
Expects the bucket's own layout,
<root>/<combo>/<task>/<run_id>/{manifest,results,rows}.json. Onlyrows.jsonis written;results.jsonis left exactly as published, so the original record — including the score this row now withholds — remains auditable.