claude-opus-5 on nine splits: the annapolis displacement does not generalise (#139) by jonfroehlich · Pull Request #146 · ProjectSidewalk/RampNet

jonfroehlich · 2026-08-19T13:22:27Z

Closes #139.

claude-opus-5 at effort=low ran the eight remaining city splits (984 panoramas). This
publishes the detections, scores them, and answers the question the issue was opened to
settle.

The annapolis displacement does not generalise

	annapolis only	pooled, 7 city splits
`claude-opus-5` (low)	0.588	0.588
`gemini-3.1-pro-preview`	0.567	0.608
delta	+0.021	−0.021

The one split that made this worth running was the one split that flattered it. annapolis is
the survey-grade Trimble MX7 city — the least representative rig in the benchmark — and the
issue pre-registered that as the risk. gemini-3.1-pro remains the best challenger and
RampNet's lead is unchanged at +0.219 F1.

The shape survives even though the ranking doesn't, and it is the more useful result

Opus trades −0.083 precision for +0.041 recall, which makes it the highest-recall chat
VLM on the board (0.614, above every Gemini leg). It loses on the aggregate and wins on the
axis operating_point.md says to optimise. The scoreboard's own third finding is "precision
is not the differentiator; recall is" — so quoting the ranking alone gets the direction right
and the reason wrong. It is written up as finding 4 under the headline table rather than as a
footnote, for that reason.

budapest replicates as Opus's collapse split (0.378, worst by a wide margin), consistent with
every other chat VLM. paterson is its only precision-strong split (0.752).

`standing` stays `False`, deliberately

roster.py raises at import for any pinned leg that is standing — a scored entry has to be
what a bare --models spec reproduces, and both efforts of claude:claude-opus-5 share one
spec. It is also unnecessary: standing gates the roster tables (fp_taxonomy, null_recall,
the frozen WITNESS_POOL_46), not the scoreboard, which keys off coverage. Opus entered the
headline table on regeneration by itself. gemini-3.7-flash and the three YOLO pano arms sit in
exactly the same position. Recorded in the doc's caveat list so the next reader doesn't try the
flip and hit the assertion.

Cost: $70.41, and layer 1 didn't record it

The run wrote no row to analysis_out/usage_log.jsonl. Recovered from Cloud Monitoring the
next day (11,988,993 in / 418,503 out) and written up in docs/model_comparison.md § "Cost
accounting". Reported on #143, which had assumed the paid-API half needed no further work — the
--usage-log path derives from REPO_ROOT, so a leg run from a scratch worktree writes its
ledger inside a directory that then gets deleted.

Two findings that generalise, both in the doc:

Estimate input from geometry; never estimate output. Opus is exactly 12,186 input
tokens/pano, so 984 panos predicts 11,991,024 against 11,988,993 billed — 0.02% out,
re-derivable from the committed detections with no cloud access. The same extrapolation for
output was 62% high.
Recovery is per-model per-day, so per-split attribution is permanently gone even after a
successful pull. Retention is ~6 weeks.

Verification

export_model_cache.py --verify — 9/9 pairs score IDENTICALLY to the cache
scoreboard.py --check — green (it went red the moment the detections landed, mid-branch)
pytest -q — 1,303 passed, 1 skipped

Three tests failed on this change and all three were right to. Two encoded "Opus low is a
single-split leg" and now assert the opposite, with a note that a reappearance means lost
coverage rather than a test needing a nudge. The third is the published-corpus ledger in
replication.md, which exists because that count silently drifted four times: 114 → 122 files.

Not in scope

manual_gold was not run, and that is a decision rather than a pending task — gemini-3.1-pro
has no manual_gold row either, so a Claude-only run there would have no peer to compare
against. Stated in full in #144.

🤖 Generated with Claude Code (claude-opus-5[1m])

…ts (#139) 984 panoramas across bend, budapest_district5, clovis, gainesville, morgantown, paterson, richmond and sao_paulo, run 2026-08-18. With the annapolis leg already on main this takes `claude-opus-5-effort-low` to nine of the ten benchmark splits. Committed on its own, ahead of the scoring and doc work, because these are paid artifacts that existed only as untracked files in a worktree. The sibling worktree that produced #141 was deleted the same day; its commits survived only because they had been pushed. These had no such protection. `manual_gold` was NOT run and is not merely pending -- see the split's own issue for why a Claude-only run there cannot settle the question it appears to address. KNOWN GAP, and it is not recoverable in-repo: the API spend for these 984 panoramas was never written to analysis_out/usage_log.jsonl. The ledger holds $0.34 -- the sonnet annapolis leg and two richmond smoke tests -- and nothing for this run. Per compare.py's own note, the detection cache stores points only, so a re-run makes zero API calls and has no usage to report; the token counts cannot be back-filled from anything in this repository. The measured smoke rate was $0.058/pano, which puts the true figure near $57, but that is an inference and is recorded as one. Only Vertex billing / Cloud Monitoring can give the real number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…that says The eight-split claude-opus-5 run wrote no row to analysis_out/usage_log.jsonl. `vertex_usage.py --days 3` recovered it the next day: 11,988,993 input / 418,503 output = $70.41. Layer 1 failed, layer 3 did its job. Written up in docs/model_comparison.md next to the Gemini history it parallels, and reported on #143, which had assumed the paid-API half needed no further work. This CORRECTS the previous commit's claim that the spend was "not recoverable" and its inferred figure of ~$57. That inference divided a smoke test's cost by `panos_scored` (4) when only 3 panos had actually been billed -- the fourth was cached. The real per-pano rate is $0.077, not $0.058. Two results worth keeping, both of which fall out of the recovery: - Estimate input from geometry; never estimate output. An Opus pano is exactly 12,186 input tokens (6 views x 2,031), identical across both smoke records, so 984 panos predicts 11,991,024 against 11,988,993 billed -- 0.02% out, a single call, re-derivable from the committed detections with no cloud access. The same extrapolation applied to output was 62% high ($16.92 against $10.46), because output is thinking plus box count and the two smoke panos were unusually verbose. Input is checkable; output is not. - Recovery is per-model per-day, so per-split attribution is permanently gone even on a successful pull. That is tolerable here only because the eight splits were one contiguous run of one model. And the deadline, which is the part that matters for next time: Cloud Monitoring retention is ~6 weeks. Had this gone unnoticed until October the number would have been unrecoverable at any price. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…neralise (#139) claude-opus-5 at low effort now has 7/7 pooled coverage, so it enters the headline and by-split tables and leaves the partial one. scoreboard.py --check is green again; it went red the moment the detections landed in e2f66fc. The result the issue was opened to get: on annapolis Opus led gemini-3.1-pro by +0.021 F1, the only time anything had displaced the top challenger. Pooled over seven city splits it TRAILS by -0.021 (0.588 vs 0.608). The one split that made the result look real was the one split that flattered it -- annapolis is the survey-grade Trimble MX7 city, and the issue pre-registered that as the risk. gemini-3.1-pro remains best challenger; RampNet's lead is unchanged at 0.219. What survives is the shape, not the ranking, and it matters more than the F1: Opus trades -0.083 precision for +0.041 recall, which makes it the highest-recall chat VLM on the board (0.614, above every Gemini leg). It loses on the aggregate and wins on the axis operating_point.md says to optimise. The scoreboard's own third finding is "precision is not the differentiator; recall is", so quoting the ranking alone gets the direction right and the reason wrong. Added as finding 4 rather than a footnote for that reason. standing stays False, and that is not an oversight. roster.py raises at import for any pinned leg that is standing -- a scored entry has to be what a bare --models spec reproduces, and both efforts of claude:claude-opus-5 share one spec. It is also unnecessary: standing gates the roster tables (fp_taxonomy, null_recall, the frozen WITNESS_POOL_46), not the scoreboard, which keys off coverage. gemini-3.7-flash and the three YOLO pano arms sit in the same position. Documented in the doc's own caveat list so the next reader does not try the flip and hit the assertion. export_model_cache.py --verify: 9/9 pairs score IDENTICALLY to the cache. Three tests failed on this change and all three were right to. Two encoded "Opus low is a single-split leg"; they now assert the opposite, with a note saying that a reappearance means lost coverage rather than a test needing a nudge. The third is the published-corpus ledger in replication.md, which exists because that count silently drifted four times: 114 -> 122 files, 23.1 -> 23.2 MB. manual_gold remains deliberately unrun (#144). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

) Review of #146. Eight findings, and one of them had a clock on it. **The "unrecoverable" cost claims were wrong, and are now recovered.** Both `replication.md` and `model_comparison.md` said the four #122 legs' token counts "cannot be recovered", giving a reason that only rules out re-running. This branch had just demonstrated the actual path, so it was applied to the older gap: `vertex_usage.py --days 7` returns claude-opus-5 $21.47 and claude-sonnet-5 $7.79 for 2026-08-15, $29.26 against the console's $28.82 -- the published figures were right to 1.5%. The per-leg split stays permanently lost (monitoring is per model per day and both efforts ran the same day), so the console table stands as the only per-leg record. The method validated itself on the one leg that did log: the 2026-08-18 Sonnet re-run appears in monitoring as 12,594/480, token-for-token identical to its `usage_log.jsonl` record. **The ledger breakdown summed to 114 under a heading saying 122.** Only the total was updated when the eight files landed; the per-leg rows were not. Fixed, and `test_the_ledger_count_matches_the_directory` now asserts the decomposition sums to the directory, not just the total -- the total alone is not a ledger. **The coverage matrix still said "annapolis only" and "the other nine splits have not been run".** That is the device this repo uses to keep an omission distinguishable from a withheld result, so a stale entry there does real damage. Restated per leg, with the $57 estimate reconciled against the measured $70.41 (~$8.80/split for Opus). **The cost row double-counted the smoke panos.** Read literally, "eight splits + 4 smoke panos" implies 988 panos, which would put the input-geometry check 0.42% out rather than the 0.02% the finding rests on. Both smoke records carry `bundle: richmond` and the export covers all 124 richmond panos, so those four are inside the 984 and were billed once; the ledger's $0.31 is a subset of the $70.41, not a line to add. Also states the residual precisely -- billed is 2,031 x 5,903, one view short of the 5,904 the geometry demands, unexplained -- and adds an independent confirmation: the annapolis leg's $8.94/125 panos scales to $70.4. **"Does not generalise" was doing more work than the data supports.** Per split the two models trade wins: Opus takes three of the seven pooled splits (four of nine overall) and the deficit is carried by gainesville and richmond. The annapolis margin was smaller than four of the seven per-split gaps, so the transferable result is that a single-split margin under ~0.07 F1 is unresolved until pooled -- stronger than the ranking correction, and stated as the headline now. Also: the nine-split `--verify` command is written down in both docs (it was claimed but not reproducible); "highest-recall chat VLM on the board" is qualified, since opus/high reaches 0.656 on annapolis four paragraphs below; and `test_the_annapolis_displacement_does_not_survive_pooling` pins the whole result, which was asserted nowhere -- the doc-currency tests only catch a forgotten regeneration, not a silently regenerated wrong number. Corroborating #143 while recovering the spend: `REPO_ROOT` derives from `__file__`, so a worktree breaks the read side too -- `vertex_usage.py` cannot find the repo-root `.env` and exits asking for `--project`. Same root cause, both directions. pytest 1,304 passed / 1 skipped; `scoreboard.py --check` green; `export_model_cache.py --verify` 9/9 identical, re-run independently against `.model_cache`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…#139, #143) The previous commit said the per-leg split was "unrecoverable, and always will be", because Cloud Monitoring reports per model per day and both efforts ran the same day. That was wrong for the same reason the original "cannot be recovered" was: it named a limit of the default query, not of the data. Daily alignment is a query parameter. Re-query at 60s and the legs separate. There is no `effort` label -- effort is a request parameter and never reaches the metric -- so the split rests on two facts the repo already holds. Input is deterministic (12,186 tokens per Opus pano, 6 views x 2,031), so total input pins the pano count exactly: the 08-15 Opus day is 251.00 panos, two 125-pano legs plus the one re-run pano, and the input half divides by geometry with no inference. Effort bills as output, so the high leg has both a higher output ratio and lower throughput. One unknown remains, and the minute series shows the predicted shape: the legs ran CONCURRENTLY, holding ~5 panos/min until 18:32 UTC, then dropping 2.54x to ~1.7 as the output ratio doubles (0.0675 -> 0.1203) -- low finishing and leaving high alone. anchor low high sum tail ratio = pure high (0.1203) $9.21 $12.20 $21.41 low ratio = 0.0349, from the #139 leg $8.95 $12.47 $21.41 console output, recorded at run time $8.94 $12.46 $21.40 The rate anchor reproduces the run-time record to 0.1% from a wholly independent source. The anchor came from the #139 leg's measured output rate before either number was compared, so that is a check and not a fit. The docs still quote $8.94/$12.46 -- the run-time record -- with this as corroboration. **Sonnet does not separate, and the script refuses rather than guessing.** Its ratio is flat across the whole run: throughput drops only 1.63x and the ratio moves the WRONG WAY (0.0365 -> 0.0281), because its high leg spent 17,820 thinking tokens against Opus's 127,227. A mixture solver handed that series returns "high effort cost less than low", which is false and entirely plausible-looking, so `MIN_RATIO_LIFT` makes it print NOT SEPARABLE instead. That is the transferable result: a per-effort split is recoverable exactly when effort changed the model enough to be worth splitting, and the telemetry is not the limiting factor. New: `scripts/analysis/vertex_effort_split.py` (read-only, same ADC as `vertex_usage.py`) and `tests/test_vertex_effort_split.py`, which pins the changepoint detector and the refusal on synthetic series whose answer is known by construction -- the cloud half needs credentials, the decision half does not. Also corrected: the guards paragraph no longer implies a warning could not have helped. Everything here was reconstructed five days late and worked only because nobody waited six weeks; which case you are in is unknowable until after the money is spent, so layer 1 is still the only layer that always works. pytest 1,309 passed / 1 skipped; scoreboard --check green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jonfroehlich · 2026-09-03T13:13:58Z

Deep review

Re-ran everything from the branch head (ed9f34f) in a clean detached worktree. The result is right, the artifacts are what the scripts produce, and the verification claims all reproduce. What follows is one replication gap with a deadline, and a handful of small corrections.

What was re-run and what it gave:

pytest -q: 1309 passed, 1 skipped (the body's 1,303 predates the last commit, which added six tests).
scoreboard.py --check: both analysis_out/scoreboard.json and docs/model_scoreboard.md current. A full regenerate, figures included, is byte-identical — git status empty afterwards, all five PNGs too.
export_model_cache.py --verify --models claude:claude-opus-5 --claude-effort low --publish-as claude-opus-5-effort-low, pointed at the main checkout's .model_cache: 9 pairs, published detections score IDENTICALLY to the cache, plus the expected manual_gold: no published export to check. This is a read-only comparison (verify() only calls cache.get), so it spent nothing.
Every number in the body and both docs against scoreboard.json: pooled P/R/F1 0.5726 / 0.6141 / 0.5876 vs Gemini 0.6558 / 0.5728 / 0.6081 ⇒ −0.083 P, +0.041 R, −0.021 F1; annapolis +0.021; RampNet lead over Gemini 0.219; budapest 0.378; paterson P 0.752; f1 range 0.479–0.649; 1.13 FP/pano. Per split, Opus takes clovis +0.037, morgantown +0.006, annapolis +0.022 and loses richmond −0.066, bend −0.033, paterson −0.039, gainesville −0.069, budapest −0.004, wins sao_paulo +0.015 — 3 of 7 pooled, 4 of 9 overall, as stated. Gainesville + richmond carry −0.135 of the −0.142 summed deficit.
The detection files: 122 in the directory, 12 Claude, all nine opus/low files have the same seven top-level keys as the Gemini exports, signature.model_id = claude-opus-5 (= the roster label), published_as = claude-opus-5-effort-low, n_uncached = 0, six views each, LF. 1,109 panos, 2,897 boxes ⇒ 2.61 boxes/pano (2.56 on annapolis), 984 for the eight new splits.
Cost arithmetic, all exact: 2,031 × 5,903 = 11,988,993; 12,186 × 984 = 11,991,024 (0.02% out); $59.94 input + $10.46 output = $70.41; 425 output tokens/pano actual vs 688 extrapolated ⇒ 62% high; 418,503 / 11,988,993 = 0.034907; the 08-15 Opus day 3,058,702 / 12,186 = 251.00 panos and $21.47; Sonnet at the $2/$10 launch rate = $7.79; $29.26 vs $28.82 = 1.5%; $70.41 / 8 = $8.80 per split; $8.94 / 125 × 984 = $70.4.
docs/replication.md's breakdown: 68 + 10 + 30 + 9 + 3 + 2 = 122, and the directory is 23.23 MB.
git merge-tree against origin/main and against feat/cost-and-time-accounting-143 (Record time as well as money for every non-free run (#143) #147): both clean.
No ToS or licensing language anywhere in the diff.

1. Medium — every recovered cost number rests on telemetry that expires in about three weeks, and none of the raw series is committed

The effort split ($8.95 / $12.47), the 251.00-pano count, the 18:32 UTC changepoint, the 2.54× throughput drop, the 0.0675 → 0.1203 ratios, the Sonnet NOT SEPARABLE verdict, and the daily totals behind $21.47 / $7.79 / $70.41 all come from Cloud Monitoring, which the doc itself says retains about six weeks. The 2026-08-15 series ages out around 2026-09-26 and the 2026-08-18 day around 2026-09-29. After that nobody — Jon included — can re-derive a single number in "Splitting a two-leg day by effort" or the #139 recovery table, and docs/model_comparison.md:2150 already concedes "only its output is replicable from this repo".

The series is small: fetch_minute_series returns one (ts, input, output) row per active minute — a few hundred rows for the 4.5-hour window. Committing it turns a one-machine, one-month result into a permanent one, and lets the tests exercise the real shape instead of synthetic series.

Fix: add --save-series <path> to vertex_effort_split.py (write the rows as JSON, LF-pinned) and --from-series <path> that skips the cloud query; commit the 08-15 Opus and Sonnet series under docs/data/vertex_minute_series/, and the vertex_usage.py daily rows for 08-15 through 08-19 beside them; point the doc at the files. This needs ADC on the billing project — if the fixer does not have it, write the two expiry dates into the doc next to the numbers so the gap is dated rather than implied.

2. Low — "four of the seven per-split gaps" is five

docs/model_scoreboard.md:83: "that lead was smaller in magnitude than four of the seven per-split gaps." Against the annapolis margin of 0.022, the gaps that exceed it are richmond 0.066, bend 0.033, clovis 0.037, paterson 0.039 and gainesville 0.069 — five; only morgantown (0.006) is smaller, and annapolis is the lead itself. The sentence is stronger with the correct count.

Fix: "smaller in magnitude than five of the other six per-split gaps."

3. Low — `vertex_effort_split.py` cannot reach its own "no verified price" message

report() (line 203 onward) formats estimate_cost(...) with :7.2f, and estimate_cost returns None for any id not in PRICING. So for an unpriced model the script raises TypeError inside report("tail anchor", …) before it reaches the if price_for(args.model) is None: line at 228 that was written for exactly that case. Harmless today (both Claude ids are priced) but the guard is dead code.

Fix: check price_for(args.model) before calling report, and print token counts without dollar figures when it is None.

4. Low — the pricing test pins the rate card by hand

tests/test_vertex_effort_split.py::test_rate_anchor_and_tail_anchor_bracket_the_published_opus_split computes cost as per_leg_in * 5 / 1e6 + out * 25 / 1e6. If the claude-opus-5 row in pricing.py ever moves, the script's output moves and this test keeps passing on numbers it no longer produces.

Fix: use pricing.estimate_cost("claude-opus-5", per_leg_in, out) and let the assertion follow the rate card.

5. Low — the #139 leg records money and not time

The doc gives $70.41 and per-token detail for the 984-pano run, and nothing about how long it took. The minute series has it (count of active minutes; the doc already quotes ~5 panos/min for the 08-15 leg), so it is recoverable now under the same deadline as finding 1. #147 is the branch that makes "time as well as money" a repo rule, so this is stated as a gap rather than a rule violation on this branch.

Fix: add the leg's active minutes and wall-clock span next to $70.41, from the same query, or state that the wall-clock was not recorded.

Decisions for Jon, not fixes

The PR body is behind the branch. It quotes 1,303 tests (now 1,309) and says nothing about the last commit: the effort-split tool, the recovery of the Make adding a challenger cheap: one registry, and stop it moving the #46 human-pass item list #122 legs' spend from Cloud Monitoring ($29.26 against the console's $28.82), or the new per-leg breakdown check in test_export_model_cache.py. Worth a short "What else landed" section in the body before merge.
Keep vertex_effort_split.py? It is 240 lines, used once, and its result is a 0.1% corroboration of numbers already recorded at run time. The transferable part is the NOT SEPARABLE refusal. I would keep it, but only with finding 1 done — without the series it is a script whose inputs vanish this month.

What holds up

The central claim — the annapolis displacement is one split's noise and the pooled ranking does not change — is correct on the committed JSON, and test_the_annapolis_displacement_does_not_survive_pooling pins the claim itself (3 wins of 7, spread > 3× the lead, highest pooled-VLM recall), not just the artifact.
standing=False is the right call and the reasoning is accurate: rampnet/roster.py:354 raises at import for any standing leg with pins, and scripts/analysis/scoreboard.py only records standing (line 387) — the pooled tables key off coverage, which is why Opus entered the headline table on regeneration alone.
The "estimate input from geometry, never output" rule is real: input lands within one view of the exact prediction; the output extrapolation misses by 62%.
The NOT SEPARABLE guard in vertex_effort_split.py is the correct shape — the decision logic is what can hand back a plausible wrong number, and the tests pin it on synthetic series whose answer is known by construction.
The replication ledger is now checked at the row level, not only the total, and the check would have caught the drift it was written for.

Fix list

[F1] Medium — Add --save-series/--from-series to scripts/analysis/vertex_effort_split.py; commit the 2026-08-15 Opus and Sonnet minute series and the vertex_usage.py daily rows for 2026-08-15..19 under docs/data/vertex_minute_series/ (LF, rounded), and reference them from docs/model_comparison.md §"Splitting a two-leg day by effort" and §"The claude-opus-5 nine-split leg". If ADC on the billing project is not available, instead write the expiry dates (~2026-09-26 for the 08-15 series, ~2026-09-29 for the 08-18 day) beside the numbers in both sections and in docs/replication.md. — files: scripts/analysis/vertex_effort_split.py, docs/data/vertex_minute_series/*, docs/model_comparison.md, docs/replication.md, tests/test_vertex_effort_split.py (add a replay test over the committed series if it lands)
[F2] Low — docs/model_scoreboard.md:83: "four of the seven per-split gaps" → "five of the other six per-split gaps". — files: docs/model_scoreboard.md
[F3] Low — In vertex_effort_split.py, check price_for(args.model) before report(); when None, print token splits without dollar figures and the existing "no verified price" line, so the script no longer raises TypeError for an unpriced id. Add a test. — files: scripts/analysis/vertex_effort_split.py, tests/test_vertex_effort_split.py
[F4] Low — tests/test_vertex_effort_split.py::test_rate_anchor_and_tail_anchor_bracket_the_published_opus_split: compute costs with pricing.estimate_cost("claude-opus-5", …) instead of hand-coded 5/25 rates. — files: tests/test_vertex_effort_split.py
[F5] Low — Record the Run claude-opus-5 (effort low) on the nine remaining splits: the only model that ever beat the top challenger has 1/10 coverage #139 leg's wall-clock (active minutes and span from the minute series, or "not recorded") next to $70.41 in docs/model_comparison.md §"The claude-opus-5 nine-split leg". — files: docs/model_comparison.md

🤖 Generated with Claude Code (claude-fable-5-1)

…live retention (#139) Review findings F1 and F3. `vertex_effort_split.py` gains --save-series/--from-series and `vertex_usage.py` gains --save-rows; the four snapshots behind the #122 effort split and the #139 leg are committed under docs/data/vertex_minute_series/, fetched 2026-09-03. Replaying them needs no credentials and reproduces every published figure exactly, which takes the numbers off the ~6-week metric clock. Also decides `price_for` once before report(), so an unpriced model prints its token split instead of raising TypeError on the None that estimate_cost returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ed rate card (#139) Review findings F1, F3 and F4. Seven tests replay the three committed minute series and assert the figures the doc publishes -- 251 panos, the 18:32 changepoint, both anchors around $8.94/$12.46, Sonnet still NOT SEPARABLE, and the #139 leg's 83 active minutes and $70.41 -- plus a save/load round-trip that holds the artifacts' shape and LF endings. The opus bracket test now prices through pricing.estimate_cost, so a change to the rate card moves the expectation with the script instead of leaving the test passing on stale numbers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…139) Review findings F1, F2 and F5. Both cost sections and docs/replication.md now name the retention expiry dates (~2026-09-26 for the 08-15 series, ~2026-09-29 for the 08-18 day) and the committed snapshots that replace them, with the --from-series commands that re-derive the split offline. The #139 leg gains the wall-clock it was missing: 2026-08-18 23:29 to 2026-08-19 01:15 UTC, 106 minutes, 83 of them active. Corrects the scoreboard's count of per-split gaps larger than the annapolis lead -- five of the other six, not four of seven. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

jonfroehlich · 2026-09-03T13:45:47Z

Review fixes

All five applied. ADC on the billing project turned out to be live, so F1 took the full path rather than the expiry-date fallback — the minute series is committed and the numbers are off the retention clock.

[F1] fixed — 4dbc11f (script + artifacts), c7cee65 (tests), 247df90 (docs). vertex_effort_split.py gains --save-series / --from-series, and vertex_usage.py gains --save-rows. Four snapshots are committed under docs/data/vertex_minute_series/, fetched 2026-09-03, all still inside retention and all reproducing the published figures exactly: claude-opus-5_2026-08-15.json (76 active minutes, 3,058,702 / 247,222 — 251.00 panos, the 18:32 changepoint, /2.54, 0.0675 → 0.1203, $8.95 / $12.47), claude-sonnet-5_2026-08-15.json (55 minutes, 3,300,368 / 118,470 — /1.63, 0.0365 → 0.0281, still NOT SEPARABLE), claude-opus-5_2026-08-18.json (83 minutes, 11,988,993 / 418,503 = $70.41) and the daily-row snapshot. --from-series replays them with no credentials, no project and no network. The expiry dates (~2026-09-26 and ~2026-09-29) are written into both model_comparison.md sections and replication.md anyway, so the window is dated rather than implied. Two notes for the record: the Sonnet window has to be wider than the Opus one (that leg started before 17:00Z; the narrow window clips it to 3,157,769 input and moves the head ratio to 0.0374), and Sonnet's output is one token under the daily row's 118,471 — a 60 s alignment boundary artifact, not a discrepancy in the day. Only fetched_utc moves between regenerations; verified byte-stable across two runs.
[F2] fixed — 247df90. docs/model_scoreboard.md: "smaller in magnitude than five of the other six per-split gaps". Confirmed against scoreboard.json — richmond 0.066, gainesville 0.069, paterson 0.039, clovis 0.037 and bend 0.033 all exceed the 0.022 annapolis lead; only morgantown (0.006) does not.
[F3] fixed — 4dbc11f. price_for is now decided once before report(), which prints token splits without dollar figures for an unpriced id. Confirmed the finding first: estimate_cost returns None off the rate card and :7.2f raised TypeError before reaching the guard at the old line 228.
[F4] fixed — c7cee65. test_rate_anchor_and_tail_anchor_bracket_the_published_opus_split now prices through pricing.estimate_cost("claude-opus-5", …), so a change to the rate card moves the expectation with the script.
[F5] fixed — 247df90. Wall-clock recovered from the same 60 s series rather than left as "not recorded": 2026-08-18 23:29 to 2026-08-19 01:15 UTC — a 106-minute span, 83 of those minutes active, 9.3 panos/min, beside the $70.41. Re-derivable from the committed series alone; tests/test_vertex_effort_split.py pins it.

Seven new tests replay the committed series and assert the published answers, including a save/load round-trip that holds the artifacts' shape and LF endings, and the two SystemExit guards on the new flags.

pytest -q: 1316 passed, 1 skipped (from 1309/1). scoreboard.py --check: both analysis_out/scoreboard.json and docs/model_scoreboard.md current; no committed number changed, so no artifact needed regenerating.

🤖 Generated with Claude Code (claude-opus-5)

jonfroehlich and others added 5 commits August 19, 2026 06:02

jonfroehlich mentioned this pull request Sep 3, 2026

Record time as well as money for every non-free run (#143) #147

Open

This was referenced Sep 3, 2026

Vistas at resolution parity: the handicap was real, and it was recall (#126) #137

Open

The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153

Merged

jonfroehlich and others added 3 commits September 3, 2026 06:38

Navigation Menu

Uh oh!

claude-opus-5 on nine splits: the annapolis displacement does not generalise (#139) - #146

claude-opus-5 on nine splits: the annapolis displacement does not generalise (#139)#146
jonfroehlich wants to merge 8 commits into
mainfrom
data/claude-opus-5-nine-splits-139

jonfroehlich commented Aug 19, 2026

Uh oh!

jonfroehlich commented Sep 3, 2026

Uh oh!

jonfroehlich commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jonfroehlich commented Aug 19, 2026

The annapolis displacement does not generalise

The shape survives even though the ranking doesn't, and it is the more useful result

standing stays False, deliberately

Cost: $70.41, and layer 1 didn't record it

Verification

Not in scope

Uh oh!

jonfroehlich commented Sep 3, 2026

Deep review

1. Medium — every recovered cost number rests on telemetry that expires in about three weeks, and none of the raw series is committed

2. Low — "four of the seven per-split gaps" is five

3. Low — vertex_effort_split.py cannot reach its own "no verified price" message

4. Low — the pricing test pins the rate card by hand

5. Low — the #139 leg records money and not time

Decisions for Jon, not fixes

What holds up

Fix list

Uh oh!

jonfroehlich commented Sep 3, 2026

Review fixes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`standing` stays `False`, deliberately

3. Low — `vertex_effort_split.py` cannot reach its own "no verified price" message