The RampNet-vs-YOLO gap at matched operating points is 0.039, not 0.252 — plus the label rescue and the equirect control (#51)

[jonfroehlich]

Three connected things for #51, in the order they had to happen: rescue the training data a purge had deleted, then answer the standing equirect objection by scoring the tiles arms that objection exists for, then fix the comparison itself — which turned out to have been reading the two models at operating points chosen by different procedures.

Net effect on the headline: the RampNet-vs-YOLO residual is 0.039 F1, not 0.252 and not 0.160.

1. The comparison was not at parity, and that is most of the gap

model_scoreboard.md and yolo_geometry_51.md compared rows whose operating points were chosen differently: RampNet at 0.55 (its shipped deployment threshold), every YOLO leg at 0.25 — the Ultralytics predict() default that nobody selected. So the published gap mixed which model is better with whose default happened to suit F1 on this benchmark.

Give each model one uniform threshold, picked the same way, on a dev split the headline is never reported over (sao_paulo), macro-meaned over the seven pooled US splits:

model	sel thr	P	R	F1	ΔF1 vs RampNet	at published op	gain from parity
RampNet	0.30	0.896	0.797	0.843	—	0.824	+0.018
y11x_tiles (ep44)	0.10	0.875	0.749	0.804	−0.039	0.667	+0.137
y11x_pano (ep38)	0.10	0.787	0.726	0.752	−0.091	0.623	+0.129
y11x_pano_h200 (ep60)	0.10	0.839	0.683	0.751	−0.092	0.575	+0.176

The asymmetry in the last column is the whole finding. 0.55 was nearly right for RampNet; 0.25 was 0.137 F1 away from YOLO's own optimum. The gap holds at 0.032–0.039 across all three candidate dev splits, so the choice of dev split is not carrying it. Selection never touches a reported split, and test_no_candidate_dev_split_is_ever_reported_over asserts that rather than trusting review.

Two things reviewers should see before merging

RampNet loses manual_gold at parity — 0.902 against the tiles arm's 0.911 — and loses morgantown (0.845 vs 0.862) and ties bend. manual_gold is the only split whose ground truth was labelled independently of RampNet's own outputs, the least circular comparison in the benchmark, and the supervised baseline takes it. This is the first split any non-RampNet model has won outright at a defensible operating point.

AP is unmoved and still favours RampNet, 0.849 vs 0.773. A correction travels with that: an earlier reading of this analysis claimed the AP columns were asymmetrically floored, and that was wrong — op_cache meta.score_floor and YoloDetector.score_threshold are both 0.05, so that column was always like-for-like. The honest summary is that RampNet has the better curve, while at each model's own best point the F1s are much closer than published.

2. The equirect objection, measured

#51's claim is that the architecture is what RampNet contributes. The standing objection is that the YOLO arms are handicapped — fed 2048×4096 equirectangular panoramas, where straight edges bow and a far ramp is a few pixels. The tiles arms exist to answer exactly that, and no tiles checkpoint had ever been scored, so the control for the headline was unmeasured.

At the pre-registered conf 0.25, pooled macro-mean F1:

leg	epoch	geometry	F1	vs RampNet
y11x_tiles	44	perspective, imgsz 1024	0.667	−0.160
y11x_pano	38	whole-pano, imgsz 1280	0.623	−0.204
y11x_pano_h200	60	whole-pano, imgsz 1280	0.575	−0.252 (the published arm)

The raw +0.092 tiles-minus-published difference would overclaim, because ep44-vs-ep60 mixes geometry with budget. Split on the pano lineage, where only budget moves:

• over-training (pano ep60 → ep38): +0.048, same geometry
• geometry (pano ep38 → tiles): +0.044, roughly matched budget

So the handicap is real and worth about 0.044 F1 — half the recoverable difference, not all of it. As with the Vistas parity arm, the geometry half is recall (0.474 → 0.520) at flat precision.

The control reproduced exactly. y11x_pano_h200 was published 2026-08-14 against a repo predating the #132 seam fix, which changed how the matcher wraps the 360° seam. Re-scored here under the same commit as the two new legs it returns P 0.969 / R 0.416 / F1 0.575 / AP 0.730 — identical to three decimals. That validates the comparison and means the published #51 table needs no seam amendment. The assertion is in the script, so the day it stops holding it fails loudly.

A result nobody was looking for: more training made the pano arm worse out of distribution. ep38 beats ep60 on 6 of 7 city splits while the ordering reverses in-distribution (manual_gold 0.840 vs 0.851). That lines up with the already-recorded finding that the 1-epoch RampNet release beats all three 60-epoch YOLO arms, and it is the main reason the open ~400 GPU-hour ep60 run still does not look worth it.

The budget term is confounded — y11x_pano and y11x_pano_h200 are divergent continuations of one lineage on different hardware, not one run read at two epochs. Stated next to the number, in the doc.

3. The data rescue that had to come first

Every label directory under /gscratch/scrubbed/jfroehli/yolo/ was empty — all five dataset variants, train and val — while every image directory was intact. That is why y11x_tiles (38612069) and its two retries died at dataset init with No labels found in .../tiles/labels/train.cache.

The mechanism is worth naming because it will recur: scrubbed purges by access time, and the .cache files are precisely what stopped anything from reading the individual label .txt files. Their atime froze while the images kept being read every epoch, so the purge took the labels and left the images. The cache that made training fast is what got the labels deleted — same shape as the partially-purged conda package cache earlier in #51. A cache is not a backup, and its presence is not evidence its source still exists.

Recovery needed no GPU and no re-derivation: Ultralytics' cache stores parsed labels (cls and normalized-xywh bboxes), which is the on-disk .txt format itself — 968,227 boxes over 557,413 train records. The writer is lifted from prepare_yolo_dataset.py::_write_pair so rebuilt files are byte-identical for anything surviving the float32 round trip, including the zero-byte file a background tile gets — load-bearing, since Ultralytics counts a missing label as nm and an empty one as nf, and the error above fires when nf == 0.

Then the same failure was closed off for the arms actually under discussion: the pano label caches had no durable copy at all, and the checkpoint snapshot was 14 days stale on every tiles arm (y11x_tiles ep44 vs ep32 saved, y11x_pano ep38 vs ep30, y26_tiles ep12 vs ep9, y11l_tiles ep11 vs ep9) — so the checkpoints an evaluation would score existed at their current epoch in exactly one place, and it was the volume with the demonstrated purge behaviour. The rescue is a committed script rather than hand-assembly, because a hand-assembled backup cannot be re-run by anyone else.

The epoch-curve follow-up is retracted

The geometry write-up closed by recommending the cheap follow-up — score the tiles checkpoints that already exist at several epochs — to turn the confounded budget term into a measured one. Those checkpoints never existed. Every arm trained save_period: -1, so Ultralytics kept only best.pt and last.pt; find over both the durable snapshot and /gscratch/scrubbed/jfroehli/yolo_runs returns zero epoch*.pt, and best is 0–4 epochs from last on every arm. y11x_pano_h200 was a resume, so it inherited save_period=-1 and Tillicum's SAVE_PERIOD=5 never applied. retarget_yolo_checkpoint.py already documented exactly this, so the recommendation contradicted a committed docstring in this repo. The budget term cannot be de-confounded cheaply: full retrain or nothing.

Gaps, stated

• One seed per cell. This is now the binding limitation, not a footnote. At a 0.160 gap seed variance was irrelevant; at 0.039 it is the entire question, and Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51's own rule is that differences under ~0.02 should not be read. Three seeds of y11x_tiles is the experiment that would settle whether the architecture claim survives — and it should come before the ~400 GPU-h ep60 run.
• The tiles arm's parity F1 is a lower bound. With budapest_district5 as dev its selected threshold is 0.05, the cache floor — the edge of what was ever measured. Closing this means re-dumping both models at a lower floor; doing one would reintroduce the asymmetry this PR removes.
• y11x_tiles is at ep44 of a 60-epoch schedule, but the pano lineage suggests more epochs would hurt it out of distribution, so this is not simply "the undertrained arm".
• Only the x architecture is swept; y11l_* and y26_* have no committed sweep.
• The VLM and pointing challengers cannot be included at all — no calibrated confidence, so one operating point rather than a curve. Their rows in model_comparison.md are unaffected in either direction.

Reproducing

CPU only, no GPU, no network — everything it reads is committed:

python scripts/analysis/operating_point_parity_51.py --sensitivity
python scripts/analysis/operating_point_parity_51.py --check   # fails if the artifact drifted
pytest tests/test_operating_point_parity_51.py tests/test_yolo_geometry_51.py -q

The control worth keeping. RampNet at 0.55 re-scores to 0.824 from op_cache against the bundle-derived published 0.827. That −0.0025 is peak extraction, not a floor effect (the bundle floor is 0.5501, i.e. above 0.55): peak_local_max at a 0.05 floor finds a different peak set than at 0.55, because newly admitted low peaks change which maxima survive min_distance. The script asserts the two stay within 0.005 and fails loudly if that stops being true. Never assert equality between the two sources.

Related: #151 found that the RampNet–YOLO ordering flips between imagery rigs over the same ground, which qualifies this result further.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Deep review

Re-ran everything from the branch head (92bc1d6) in a clean detached worktree on Windows, CPU only. The measurement is sound and reproduces exactly, and the headline — the published RampNet-vs-YOLO gap is mostly an operating-point artifact — survives. What does not survive is the explanation the PR gives for its own control, and the branch will go red the moment it meets #152/#153.

• operating_point_parity_51.py --sensitivity and yolo_geometry_51.py both rewrite their JSON byte-for-byte identical to the committed copies (sha256 9da31321…, 678ae904…); both --check commands exit 0.
• pytest tests/test_operating_point_parity_51.py tests/test_yolo_geometry_51.py: 37 passed. Full suite: 1359 passed, 1 skipped in 121 s.
• Every number in the PR body and both docs that I could trace matches the artifacts: the parity table (0.30/0.10; 0.843/0.804/0.752/0.751; −0.039/−0.091/−0.092; 0.824/0.667/0.623/0.575; +0.018/+0.137/+0.129/+0.176), the sensitivity band (0.032 / 0.039 / 0.038 for the three dev splits), the per-split table, the geometry table (0.667/0.623/0.575, AP 0.773/0.752/0.730), the decomposition (+0.048 / +0.044 / +0.092), the recall term (0.4743 → 0.5197), "ep38 beats ep60 on 6 of 7" (paterson is a tie at 0.635), and the control leg (0.969 / 0.416 / 0.575 / 0.730, and the test really does fail when a split is perturbed).
• Merge-tree is clean against main and every other open branch. The head did not move during the review.

---

1. High — the scripts and tests read the live US_SPLITS, so this PR and #152/#153 cannot both be green on main

Both scripts import ALL_SPLITS / US_SPLITS from analysis.low_floor_sweep (operating_point_parity_51.py:93, yolo_geometry_51.py:44) and macro_at / pooled return None unless every pooled split has a sweep. #152 adds laurens to US_SPLITS; #153 renames it laurens_mapillary and adds laurens_gsv to CITY_SPLITS. No YOLO report exists for either.

I built a scratch merge of this branch with fix/laurens-challenger-coverage (no textual conflicts — git merge-tree is clean) and ran the two test files on it: 12 of 37 fail, and both --check commands fail. Worse, the parity script does not just fail — it re-pools RampNet over eight splits (laurens has an op_cache), drops all three YOLO legs silently, and prints RampNet at F1 0.820 against a "published" 0.789. The control catches it (test_control_agrees_on_the_committed_data fails), which is the right outcome, but whichever of these PRs merges second turns main red.

This is exactly the trap #152's body describes for farfield_forensics.py and silent_activation.py ("a frozen study reading the live US_SPLITS"), and it applied the fix there: an explicit split tuple in the script with the reason. Same fix here.

Fix: pin POOLED_SPLITS = ("richmond", …, "gainesville") and ALL_SPLITS_AS_RUN (the ten in env.txt) as module constants in both scripts with a comment saying they are the population the 2026-08-30 run covered and are not to follow the registry; derive NON_POOLED from those; make test_pooled_uses_exactly_the_seven_us_splits and test_every_split_has_all_three_legs assert against the pinned tuples rather than mod.US_SPLITS / mod.ALL_SPLITS.

2. High — the control's stated mechanism is wrong, and docs/operating_point.md already says what the real one is

The PR body, operating_point_parity_51.md:153–158, the script docstring (:51–70) and test_control_delta_is_the_known_extraction_difference all say the −0.0025 between RampNet's op_cache re-score and its bundle at 0.55 "is peak extraction: peak_local_max at a 0.05 floor finds a different peak set than at 0.55, because newly admitted low peaks change which maxima survive min_distance."

That cannot be the mechanism. Lowering threshold_abs only adds candidates; a pixel that is the maximum of its min_distance neighbourhood at 0.55 is still that maximum at 0.05. A lower floor can never move or remove a ≥ 0.55 peak.

Measured per split, bundle (≥ 0.55) against op_cache (≥ 0.55), matching detections within 0.004 normalized:

split	bundle	op_cache	in bundle, not in cache	at the seam	in cache, not in bundle	ΔF1 @0.55
richmond, clovis, morgantown, annapolis, budapest_district5	—	—	0	0	0	0.0000
bend	265	257	35	0	27	+0.0034
paterson	284	281	36	0	33	−0.0048
gainesville	205	197	33	1	25	−0.0163
sao_paulo (the dev split)	251	228	53	1	30	−0.0189
manual_gold	3610	3487	495	26	372	−0.0095

Five splits agree detection-for-detection (tp/fp/fn identical). The other five differ by 13–14 % of their detections, mostly at different locations (bend: 8 of 35 are the same point at a different confidence; the rest are elsewhere). That is two different heatmaps, not one heatmap read at two floors. And the repo has already measured why — docs/operating_point.md:80–90: manual_gold's bundle was exported with horizontal-flip TTA ("3610 vs 3487 at ≥ 0.55", the exact counts above), while op_cache is the no-TTA deployment path; and for the GSV splits (bend, paterson, gainesville, and sao_paulo) the production bundle came from a different resample of the panorama than the native-res cache, with detections displaced up to 0.44 R. The five that agree are the Mapillary splits, which #152 records as bit-exact between the two paths.

Consequences for this PR:

• The 0.005 tolerance passes on the pooled macro only by cancellation (+0.003, −0.005, −0.016 across the three GSV splits in the pool). A per-split control would have failed on gainesville.
• The dev split has the largest discrepancy of the ten (−0.019). RampNet's threshold was selected on the split where its op_cache curve is least like its shipped detections. (The selection is probably robust — the sensitivity table lands on 0.30–0.35 either way — but that is an argument the doc has to make, not one it can skip.)
• On manual_gold the TTA-vs-no-TTA delta (0.0095) is the same size as the margin by which the tiles arm "wins" it (0.009). See finding 3.

Fix: replace the peak-extraction paragraph in the doc, the script docstring, and the test docstring with the documented causes and a pointer to operating_point.md's parity gate; add the per-split Δ table above to the artifact (control.per_split) and print it; keep the macro assertion but add a per-split one with its own tolerance, including the dev split and manual_gold, so cancellation cannot hide a split-level divergence again.

3. Medium — the per-split "wins" are read below the doc's own noise rule and inside the source discrepancy

operating_point_parity_51.md "Gaps" says differences under ~0.02 F1 should not be read. Then §2 reads 0.009 (manual_gold, 0.911 vs 0.902) as "loses the split that matters most" and "the first result in this benchmark where a non-RampNet model takes a split outright", 0.017 (morgantown) as a win, and the status line counts "3 of the 10 splits". The PR body promotes manual_gold to one of the two things reviewers must see before merging.

The direction is not fragile to RampNet's threshold — over the whole 0.05–0.95 grid RampNet's best on manual_gold is 0.904 @0.35 and on morgantown 0.851 @0.40, both still below the tiles arm — but the margin is inside the TTA delta on that split (finding 2): RampNet's committed manual_gold configuration (TTA, detections_meta.json) at 0.55 scores 0.9085 against the no-TTA 0.8990, so at 0.30 with TTA it would plausibly tie 0.911. A paired test would settle it and needs no GPU (both models' per-pano detections exist), but the tiles arm's detections are not committed (see the decision below), so it cannot be run from a clean clone.

Fix: qualify §2, the status line and the "3 of 10" count with the 0.02 rule and the TTA caveat ("within the noise rule; the manual_gold margin is the size of the TTA delta on that split; a paired test is blocked on publishing the tiles detections"). The PR body's reviewer note is Jon's to soften.

4. Medium — the scoreboard and the YOLO README now contradict this PR and do not point to it

• docs/model_scoreboard.md:362: "The YOLO tiles arms are absent — still training." The tiles arm is scored on all ten splits in this PR.
• scripts/model_comparison/yolo_baseline/README.md:88: "the tiles arms remain unevaluated."
• The scoreboard's headline table and its manual_gold column (RampNet 0.908 vs YOLO11x 0.851) are the published comparison; a reader of that page has no way to find that at matched operating points the residual is 0.039 and the best YOLO leg is within noise of RampNet on three splits. This PR touches neither file.

Fix: a short caveat block in model_scoreboard.md beside the headline table (or under "Choosing an operating point") linking operating_point_parity_51.md and yolo_geometry_51.md, stating that the YOLO rows are at the un-selected 0.25 default and what parity does to them; replace the two stale sentences. docs/model_comparison.md's YOLO section should get the same one-line pointer.

5. Medium — provenance is recorded, but four gaps are not stated beside the numbers

• env.txt: "repo dirty : 10 paths" at d964d5d. The doc says "at repo d964d5d" only. git status --porcelain | wc -l counts untracked outputs and modified code alike, so a reader cannot tell which; the driver should log the list, not the count.
• The control leg was not re-inferred. All ten *_pano.txt reports say [y11x_pano_h200] all N panos already cached; model load skipped. The control re-scores detections cached on makelab2 from the 2026-08-14 publishing run — which is the right test for a matcher change and is what "re-scored" means, but run_yolo_geometry_eval.sh:19 says "re-run", and the doc should say the control isolates the matcher and does not test inference reproducibility.
• Where the checkpoints live. env.txt records sha256s of copies under makelab2 yolo_ckpts/. Nothing in the PR says which durable path on klone (/gscratch/makelab/jonf/rampnet_yolo_baseline_51/<arm>/weights/best.pt per run_durable_snapshot.slurm) carries those hashes, or that the checkpoints are unpublished — so "reproducible by someone with cluster access" is true but undiscoverable, and "not reproducible from a clean clone" is unstated.
• The rescue run's outcome lives only in the PR body. rebuild_yolo_labels_from_cache.py and check_yolo_dataset_loads.py carry the expected counts (557,413 / 968,227 / 59,923 background) in docstrings; the job ids, dates, --verify PASS and the acceptance-test output are not in the repo.

Fix: a "Provenance and cost" block in yolo_geometry_51.md covering the four points; run_yolo_geometry_eval.sh logs git status --porcelain in full; commit the rebuild/snapshot job logs (or a dated section in yolo_baseline/README.md with job ids and the PASS lines).

6. Medium-low — cost is not recorded

driver.log gives the eval as 17:10:09 → 18:17:48 on one A40: 67 min 39 s wall, ≈ 1.13 GPU-h, $0 (makelab2). The klone rescue and snapshot jobs (CPU, ckpt-all) have no job ids anywhere in the repo. The time-and-money rule is a standing one even though #147 has not merged.

Fix: one line in each doc; a compute_log.jsonl row once #147 lands (state "pending #147" until then).

7. Low

• rebuild_yolo_labels_from_cache.py overclaims "byte-identical". Ultralytics verify_image_label (read in 8.4.104 here; 8.4.120 at eval) drops duplicate rows and, when it does, reorders the survivors (lb = lb[i] from np.unique(..., return_index=True)). A file that had a duplicated box is rebuilt deduplicated and sorted — training-equivalent, not byte-identical. write_text(body) also takes the platform newline; add newline="\n". The float32 arithmetic itself is right (≤ 6e-8 error against 5e-7 rounding), and the 1e-6 verify tolerance is correct.
• test_no_candidate_dev_split_is_ever_reported_over cannot fail. NON_POOLED is defined as the complement of US_SPLITS (:106), so isdisjoint is a tautology. The invariant is real — argparse choices and build() enforce it — but the doc cites this test as the guarantee. Assert it on build(dev) for every candidate instead (dev not in result["pool"] and the pool is exactly the pinned seven).
• PUBLISHED_RAMPNET_F1 = 0.827 and 0.55 are duplicated across the two scripts; the parity script already imports from yolo_geometry_51. Import them.
• The AP table's RampNet 0.849 is not in either artifact. It is the scoreboard's macro AP (model_scoreboard.md:32, scoreboard.json); say so in the caption or compute it from op_cache in the script.
• Slang in committed text: "load-bearing" (yolo_geometry_51.md:40, yolo_geometry_51.py:19, both test docstrings), "split-brain" (operating_point_parity_51.md:156, .py:64). Plain wording.
• scripts/analysis/README.md indexes the analysis scripts and does not list the two new ones.

---

Decisions for Jon, not fixes

• Register y11x_tiles and y11x_pano as roster legs and publish their detections. Neither is in rampnet/roster.py and benchmark/model_detections/ has no files for them, so the scoreboard cannot show the parity result and the paired test in finding 3 cannot be run from a clean clone. export_model_cache.py from the makelab2 cache is the path; it is the same "unpublished parity detections" gap Vistas at resolution parity: the handicap was real, and it was recall (#126) #137 carries for Vistas.
• Merge order with Add Laurens, IA — two imagery arms over one rural footprint, and the deficit is the rig, not the town (#149, #151) #152/The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153. After finding 1 there is none. Before it, whichever merges second is red.
• Three seeds of y11x_tiles — the doc's own recommendation and a GPU spend.
• The GSV-split op_cache vs production-resample discrepancy (finding 2) is a pre-existing GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55/Assess location precision for the unassessed candidate cities — the gate every route to 500k depends on #96 matter, not this PR's; whether to regenerate those caches is separate.

What holds up

• Both artifacts regenerate byte-identically on Windows; LF-only, floats rounded to 4 dp; --check on both is a real guard (the JSON comparison is of the full payload, sensitivity included).
• The methodology is the fair one: a single uniform threshold per model, selected on a split the headline never reports over, with sensitivity over all three candidates; macro-mean matches the scoreboard's convention (scoreboard.py:33, 356–398); both floors really are 0.05 (meta.score_floor on all ten caches, YoloDetector.score_threshold = 0.05 at detectors.py:1290), so the AP comparison was always like-for-like as the correction says.
• RampNet's selected threshold on sao_paulo is 0.30 — Lower the deployment operating point (peak threshold 0.55 → ~0.25–0.35): a free, recall-first lever #54's recommendation, reached by an independent route.
• The control leg reproduces its published row to every digit and the test proves the check can fail.
• The decomposition arithmetic, the "6 of 7" claim (paterson a tie at 3 dp), the in-distribution reversal, and the recall-not-precision mechanism all check against the JSON.
• The retraction is kept beside the recommendation it retracts (yolo_geometry_51.md:152–167), and the "Superseded" block in §2 points forward — nothing was silently deleted.
• The rescue tooling has no destructive step: .tmp + rename after a hash match, a second source hash to catch a torn copy, refusal to overwrite a differing durable copy without FORCE=1, stale .cache files moved aside not deleted. Zero-byte background files are the correct Ultralytics semantics (nf, not nm), and the acceptance test drives YOLODataset itself.
• env.txt pins repo HEAD, ultralytics version, GPU and three sha256s, and a test asserts that stays true.
• No ToS or licensing text anywhere in the diff.

Fix list

• [F1] High — Pin the split population in both scripts as explicit tuples with a reason (seven pooled + the ten as run), derive NON_POOLED from them, and make the tests assert against the pinned tuples, not the live US_SPLITS/ALL_SPLITS — files: scripts/analysis/operating_point_parity_51.py, scripts/analysis/yolo_geometry_51.py, tests/test_operating_point_parity_51.py, tests/test_yolo_geometry_51.py
• [F2] High — Replace the "peak extraction / min_distance" explanation with the documented causes (flip-TTA on manual_gold; production-vs-native resample on the GSV splits; Mapillary splits identical), cite docs/operating_point.md's parity gate, add a per-split bundle-vs-op_cache Δ table to the artifact and the doc, and add a per-split control assertion (dev split and manual_gold included) beside the macro one — files: scripts/analysis/operating_point_parity_51.py, docs/operating_point_parity_51.md, tests/test_operating_point_parity_51.py, docs/data/operating_point_parity_51.json
• [F3] Medium — Qualify the manual_gold / morgantown / "3 of 10" reads with the doc's own 0.02 rule and the TTA caveat; state that a paired test is blocked on publishing the tiles detections — files: docs/operating_point_parity_51.md
• [F4] Medium — Add a caveat block to model_scoreboard.md linking both new docs and stating the YOLO rows are at the un-selected 0.25 default; replace the stale "tiles arms are absent / remain unevaluated" sentences; one-line pointer in docs/model_comparison.md's YOLO section — files: docs/model_scoreboard.md, scripts/model_comparison/yolo_baseline/README.md, docs/model_comparison.md
• [F5] Medium — Add a "Provenance and cost" block: the 10 dirty paths caveat, the control re-scored from cache not re-inferred, the durable checkpoint path + unpublished status, and the rescue job ids/dates/PASS output; make the driver log git status --porcelain in full — files: docs/yolo_geometry_51.md, scripts/model_comparison/yolo_baseline/run_yolo_geometry_eval.sh, scripts/model_comparison/yolo_baseline/README.md
• [F6] Medium-low — Record the eval cost (67 min 39 s on one A40 ≈ 1.13 GPU-h, $0) and the klone rescue jobs' ids in the docs — files: docs/yolo_geometry_51.md, docs/operating_point_parity_51.md
• [F7] Low — Docstring caveat on dedup/reorder (not byte-identical for files with duplicate boxes); write_text(body, newline="\n") — files: scripts/model_comparison/rebuild_yolo_labels_from_cache.py
• [F8] Low — Make the dev-split test assert on build(dev) output for every candidate instead of the tautological isdisjoint — files: tests/test_operating_point_parity_51.py
• [F9] Low — Import PUBLISHED_RAMPNET_F1 (and the 0.55 deployed threshold) from yolo_geometry_51 instead of redefining — files: scripts/analysis/operating_point_parity_51.py
• [F10] Low — State the source of the AP table's RampNet 0.849 (scoreboard macro AP) or compute it in the script — files: docs/operating_point_parity_51.md, scripts/analysis/operating_point_parity_51.py
• [F11] Low — Replace "load-bearing" and "split-brain" with plain wording — files: docs/yolo_geometry_51.md, docs/operating_point_parity_51.md, scripts/analysis/yolo_geometry_51.py, scripts/analysis/operating_point_parity_51.py, tests/test_operating_point_parity_51.py, tests/test_yolo_geometry_51.py
• [F12] Low — Add the two new scripts to the analysis index — files: scripts/analysis/README.md

🤖 Generated with Claude Code (claude-fable-5-1)

[jonfroehlich]

Second review — verdicts on F1–F12, plus three the first pass missed

Re-ran the branch head (92bc1d6) in a clean detached worktree on Windows, CPU only. The first review holds up: eleven of its twelve findings are real, and only one is wrong. The measurement itself still reproduces — both artifacts rewrite byte-identically, both --check commands exit 0, 37/37 of the two new test files pass, full suite 1359 passed / 1 skipped.

One thing changed since that review was written: #152 and #153 both merged on 2026-09-04. F1 is therefore no longer a merge-order question — origin/main already carries laurens_mapillary in US_SPLITS and laurens_gsv in CITY_SPLITS, so this branch as it stands turns main red the moment it merges. Measured, not predicted: swapping scripts/analysis/low_floor_sweep.py for origin/main's copy fails 12 of 37 tests and both --check commands, and operating_point_parity_51.py does not even fail cleanly — it raises TypeError: unsupported format string passed to NoneType.__format__ at :309 because macro_at returned None.

I also verified F2's mechanism claim from first principles rather than taking it on trust, and it is correct.

---

Findings

High

H1 — scripts/analysis/operating_point_parity_51.py:93, scripts/analysis/yolo_geometry_51.py:44: a frozen study reads the live split registry, and main has already moved.
Both scripts import ALL_SPLITS / US_SPLITS from analysis.low_floor_sweep, and pooled() / macro_at() return None unless every pooled split has a report.
Failure scenario, measured: with origin/main's low_floor_sweep.py in place (8-split US_SPLITS), docs/data/yolo_geometry_51/laurens_mapillary_*.txt does not exist, so all three YOLO legs pool to None — while analysis_out/op_cache/laurens_mapillary.json does exist on main, so RampNet would silently re-pool over a different population than the models it is being compared against. In this worktree it crashes instead; after a real merge it produces a number. 12 tests fail either way.

H2 — scripts/analysis/operating_point_parity_51.py:61, docs/operating_point_parity_51.md:154, tests/test_operating_point_parity_51.py:172: the control's stated mechanism is impossible, and the real one is already measured in this repo.
The claim is that the −0.0025 between RampNet's op_cache re-score and its bundle at 0.55 is "peak extraction: peak_local_max at a 0.05 floor finds a different peak set than at 0.55, because newly admitted low peaks change which maxima survive min_distance."
Verified false by construction: peak_local_max suppresses on a maximum filter, so a pixel that is the maximum of its min_distance neighbourhood at 0.55 is still that maximum at 0.05. Lowering threshold_abs can only add. I ran 200 random 200x400 heatmaps through skimage 0.26 comparing threshold_abs=0.55 against the >= 0.55 subset of threshold_abs=0.05: 0 trials differ.
What it actually is, computed per split from the committed op_cache against model_scoreboard.md's by-split row:

split	op_cache @0.55	bundle	delta
richmond / clovis / morgantown / annapolis / budapest_district5	—	—	all within 0.0005
bend	0.8532	0.850	+0.0032
paterson	0.8006	0.805	−0.0044
gainesville	0.7871	0.803	−0.0159
sao_paulo (the dev split)	0.7578	0.777	−0.0192
manual_gold	0.8990	0.908	−0.0090

Five Mapillary splits agree and five do not, and docs/operating_point.md:70-90 already records why — the GSV production path used a 4096x2048 resample (its detection counts 265/257, 284/281, 205/197, 251/228 match exactly), and manual_gold's committed detections carry flip-TTA (3610/3487). Two different heatmaps, not one heatmap read at two floors.
Failure scenario: the pooled macro clears its 0.005 tolerance by cancellation (+0.003 bend against −0.004 paterson and −0.016 gainesville). A scoring regression that moved gainesville by −0.02 and bend by +0.02 would pass the control unnoticed. And the dev split has the largest discrepancy of the ten, which the doc does not say.

Medium

M1 (new) — scripts/model_comparison/check_yolo_dataset_loads.py:16-20, 67-84: the acceptance test cannot detect the failure mode its docstring says it guards against.
The docstring's first claim is about nf (found) against nm (missing) — that a rebuild which skipped the label-less tiles would still "work", and that a large nm is the signature. The script never reads nm. It reads len(ds.labels), box counts and zero-box counts — and Ultralytics' cache_labels appends a record for a missing label file exactly as it does for an empty one (the append runs for every im_file; only the nm/ne counters differ, and those live in x["results"], which ds.labels does not expose).
Failure scenario: delete all 59,923 zero-byte background label files and re-run --expect-images 557413 --expect-boxes 968227. n_images = 557,413, n_boxes = 968,227, n_empty = 59,923 — every check passes and the script prints PASS, while the dataset now has 59,923 nm where it should have ne. Secondary: augment=False means Ultralytics only warns on nf == 0 rather than raising, so "this is the call that raised the original ValueError" is not what happens as invoked.

M2 — docs/operating_point_parity_51.md:90-99: three per-split reads are below the document's own noise rule, and one is inside a measured instrument difference.
"Gaps" says differences under ~0.02 F1 should not be read; section 2 then reads 0.009 (manual_gold), 0.017 (morgantown) and 0.001 (bend) as a loss, a loss and a tie, and calls manual_gold "the first result in this benchmark where a non-RampNet model takes a split outright".
Failure scenario: RampNet's committed manual_gold detections were exported with flip-TTA (benchmark/manual_gold/detections_meta.json) while the op_cache swept here is no-TTA; at 0.55 those two differ by 0.0090 F1 — the same size as the 0.009 margin being reported as a win. A reader quoting "YOLO beats RampNet on manual_gold" is quoting a TTA setting.

M3 (new) — the parity comparison mildly understates RampNet, and nowhere says so.
Three of the seven pooled splits are GSV, where op_cache runs −0.016 to +0.003 against the bundle path. Macro-meaned that is −0.0024, i.e. RampNet's parity 0.843 would be about 0.845 on a bundle-derived path and the headline gap about 0.041 rather than 0.039. Conservative, not flattering — but a caveat that only travels in one direction is still a missing caveat.

M4 — docs/model_scoreboard.md:362, docs/model_comparison.md:147, 185-187, scripts/model_comparison/yolo_baseline/README.md:88, 136-137: three committed documents state the tiles arms are unevaluated, which this PR makes false, and none of them points at either new document.
Failure scenario: someone reads the scoreboard's headline table (RampNet 0.827 vs YOLO11x 0.575) and its manual_gold column (0.908 vs 0.851), concludes RampNet leads by 0.25 everywhere, and has no path from that page to the finding that at matched operating points the residual is 0.039 and the best YOLO leg is level on three splits.

M5 — provenance gaps beside the numbers. Four, all confirmed: env.txt records repo dirty : 10 paths as a count, so a reader cannot tell untracked outputs from modified scoring code; all ten *_pano.txt say [y11x_pano_h200] all N panos already cached; model load skipped, so the control was re-scored, not re-inferred (the right instrument for a matcher change, but run_yolo_geometry_eval.sh:19 says "re-run"); the durable checkpoint path on klone and the fact that the checkpoints are unpublished are nowhere stated, so "not reproducible from a clean clone" is left to inference; and the rescue run's job ids and --verify PASS live only in the PR body.

M6 — no cost is recorded. driver.log gives 17:10:09 to 18:17:48 on one A40: 67 min 39 s, about 1.13 GPU-h, $0. The standing rule is to record time as well as money at run time.

Low

L1 — scripts/model_comparison/rebuild_yolo_labels_from_cache.py:37, 114: "byte-identical" overclaims, and write_text takes the platform newline. Confirmed in ultralytics 8.4.104's verify_image_label: it calls np.unique(lb, axis=0, return_index=True) and keeps lb[i] when duplicates exist — a file that held a duplicate row is rebuilt deduplicated and row-sorted. Training-equivalent, not byte-identical.

L2 — tests/test_operating_point_parity_51.py:147: the fairness test cannot fail. NON_POOLED is defined as the complement of US_SPLITS, so isdisjoint is a tautology — and the doc cites this test as the guarantee that selection never touches a reported split.

L3 (new) — tests/test_operating_point_parity_51.py:206: test_at_floor_is_flagged... is vacuous. No model has at_floor true in the default build, so the loop body never runs. The flag is real and does fire (the tiles arm selects 0.05 with budapest_district5 as dev), and the doc reports that case as a lower bound.

L4 — PUBLISHED_RAMPNET_F1 = 0.827 and the 0.55 deployed threshold are duplicated across the two scripts, though the parity script already imports from yolo_geometry_51.

L5 — docs/operating_point_parity_51.md:107: the AP table's RampNet 0.849 is in neither artifact — it is the scoreboard's macro AP, uncited.

L6 — scripts/analysis/README.md does not index either new script.

---

Verdicts on the original F1-F12

#	sev	verdict	note
F1	High	real, and now urgent	#152/#153 merged 2026-09-04; measured 12/37 test failures plus a TypeError under main's registry
F2	High	real	mechanism disproved in 200 skimage trials; per-split delta reproduced, and it matches docs/operating_point.md's own counts
F3	Medium	real	0.009 / 0.017 / 0.001 all under the doc's own 0.02 rule; the TTA delta on manual_gold is 0.0090
F4	Medium	real	all five stale sentences confirmed at the cited lines
F5	Medium	real	all four sub-points confirmed (dirty count, cached control, unpublished ckpts, rescue log)
F6	Med-low	real	timings confirmed in driver.log
F7	Low	real	dedup and reorder confirmed in the ultralytics source
F8	Low	real	tautology confirmed
F9	Low	real	duplicated constant
F10	Low	real	0.849 is model_scoreboard.md's, not this study's
F11	Low	half wrong — "load-bearing" is not a defect	it appears 38 times across 20+ committed files, including low_floor_sweep.py, scoreboard.py and four existing docs. House idiom, not slang this PR introduced; changing it only here would make this PR the inconsistent one. "split-brain" was new to this PR (2 occurrences) and is vague — it goes away with F2.
F12	Low	real	neither script indexed

Not defects

• The methodology is sound and I could not break it: uniform threshold per model, selected on a split never reported over, sensitivity over all three candidates, macro-mean matching scoreboard.py's convention. Both floors really are 0.05, so the AP correction in the PR body is right.
• test_every_leg_gains_from_being_given_its_own_operating_point is not a tautology — selection is on the dev split, the assertion is on the pool.
• run_yolo_geometry_eval.sh:82's exit=$? is correct: $? is expanded before the command substitution later in the same word runs.
• The rescue tooling has no destructive step; its .tmp-plus-rename, second source hash and FORCE=1 refusal are all correctly implemented.

🤖 Generated with Claude Code (claude-opus-5)

[jonfroehlich]

Review fixes

8f9c6dd. Eleven of the twelve original findings plus three new ones, fixed largest to smallest. Full suite 1366 passed, 1 skipped (up from 1359/1 — six new tests). Both --check commands pass, and both committed artifacts still regenerate byte-identically under origin/main's current registry as well as this branch's, which is the point of the first fix.

High

• H1 / F1 — the split population is pinned. POOLED_SPLITS and ALL_SPLITS_AS_RUN are now module constants in yolo_geometry_51.py, imported by operating_point_parity_51.py; the held-out keys are pinned to the run while only their reason text follows the registry, so docs/data/yolo_geometry_51.json is byte-unchanged. Two tests assert the tuples literally. Verified by swapping in origin/main's low_floor_sweep.py: 43/43 pass and both --check commands exit 0, against 12 failures and a TypeError before.
• H2 / F2 — the control's mechanism is corrected, and the control is now per split. The peak-extraction explanation is replaced in the script docstring, the doc and the test names, with the documented causes (GSV resample; flip-TTA on manual_gold) and a pointer to docs/operating_point.md. control.per_split is in the artifact and printed by the script: 0.002 tolerance on the five splits where the two paths are the same computation, 0.025 on the five with a written-down reason to differ, and the script now exits 2 if any split fails. Four new tests, including one proving the per-split check can fail and one asserting the dev split really does carry the largest discrepancy.

Medium

• M1 (new) — check_yolo_dataset_loads.py now detects the failure its docstring claimed. It stats the label paths via img2label_paths and fails on any absent file, takes --expect-background, and records that augment=False only warns on nf == 0.
• M2 / F3 — the three near-tied splits are reported as ties, with the manual_gold margin flagged as the size of the measured TTA delta on that split, and the blocked paired test recorded.
• M3 (new) — the ~0.002 conservative bias from the GSV splits is stated in both the docstring and the doc.
• M4 / F4 — the stale sentences are replaced in model_scoreboard.md, model_comparison.md and yolo_baseline/README.md, and a caveat block sits beside the scoreboard's headline table. It is outside the <!-- BEGIN GENERATED --> region — I put it inside first and test_committed_doc_is_current caught it, which is the test doing its job.
• M5 / F5 — a "Provenance and cost" section in yolo_geometry_51.md covering all four gaps, and the driver now logs git status --porcelain in full.
• M6 / F6 — cost recorded in both docs: 67 min 39 s on one A40, ≈ 1.13 GPU-h, $0, ledger row pending Record time as well as money for every non-free run (#143) #147.

Low

• L1 / F7 — the dedup/reorder caveat, and write_text(body, newline="\n").
• L2 / F8 — the dev-split test now asserts on build()'s output for every candidate.
• L3 (new) / F10 / F12 — the at_floor test is no longer vacuous; the AP table cites its source; both scripts are indexed in scripts/analysis/README.md.
• L4 / F9 — PUBLISHED_RAMPNET_F1 is imported, not restated.

Skipped

• F11's "load-bearing" half. It appears 38 times across more than 20 committed files, including low_floor_sweep.py, scoreboard.py, peak_nms_check.py and four existing docs. It is house usage, not slang this branch introduced, and changing it only here would make this PR the inconsistent one — a repo-wide sweep is a separate change if it is wanted at all. "split-brain" was new here and is gone, as part of the H2 rewrite.

Not touched — for you

• Registering y11x_tiles / y11x_pano as roster legs and publishing their detections. Both new docs now state this gap explicitly beside the numbers that need it, but the export itself is a decision and a run.
• Merge order with Add Laurens, IA — two imagery arms over one rural footprint, and the deficit is the rig, not the town (#149, #151) #152/The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153. Both merged 2026-09-04, so this branch is behind main — but after H1 the two are compatible; I verified the tests and both --check commands against main's registry rather than merging.
• Three seeds of y11x_tiles, the doc's own recommendation and a GPU spend.
• Whether to regenerate the GSV splits' op_cache so the two paths agree. Pre-existing GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55/Assess location precision for the unassessed candidate cities — the gate every route to 500k depends on #96 territory, not this PR's.

🤖 Generated with Claude Code (claude-opus-5)