Add Laurens, IA — two imagery arms over one rural footprint, and the deficit is the rig, not the town (#149, #151) - #152
Merged
Merged
Conversation
Phase 0 intent: rural / small-town streetscape on a consumer 360 rig. Every existing split is urban, suburban, or a college town; Laurens is 1.91 km2 and 1,264 people. It answers whether the operating point holds where sidewalks are intermittent and driveway aprons outnumber ramps. Imagery: Mapillary, GoPro Max 5760x2880, all captured 2025-11 by a single uploader in 19 sequences — the most uniform vintage of any split. Detection run: 4,495 panos, 9.3% with an operational detection. Strata are 5 top / 64 random / 25 empty = 94, not the standard 5/95/25. The shortfall is geometric, not a sampling artifact: _spread walks the whole candidate pool greedily, and the town saturates at 64 detection panos held 30 m apart. The unbiased column (random + empty) is therefore 89 panos against the usual 120, so its intervals are correspondingly wider. Strata definitions and the de-clustering rule are unchanged. 111 operational detections to adjudicate. Archive reconciled 1:1, zero decay. Still outstanding per docs/adding_a_benchmark_city.md: GT review, the BENCHMARK_SPLITS allowlist, and the downstream pooled numbers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
digest 7485886ea08f671b over 94 panos, verify clean. Written now rather than after review so it records the bytes the reviewer actually sees — Mapillary re-serves thumb_original_url from a signed, expiring URL, so "the same pano id" is not a guarantee of the same pixels later. docs/adding_a_benchmark_city.md still does not mention this step; it postdates the runbook. Every other split carries one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
94/94 panos reviewed, all 111 detections judged, 190 missed-ramp marks. Precision 0.898 (95% CI 0.827-0.942) unbiased 0.863 Recall 0.390 (95% CI 0.331-0.451) unbiased 0.325 Precision is unremarkable — mid-pack. Recall is roughly half the next-worst split (clovis 0.650), and three things say it is a real failure mode rather than a small or ramp-poor sample: - Laurens is ramp-RICH: 2.65 ramps/pano, 3rd of 10 splits, ahead of richmond, annapolis and morgantown. Misses/pano is 1.62 against a 0.45-1.19 range everywhere else. - The misses are NOT far-field. In all nine other splits missed ramps sit nearer the horizon than detected ones (delta -0.013 to -0.030 in normalized y); Laurens is the only split where the delta is positive (+0.004). It is missing near, well-resolved ramps. - Not a lighting artifact either. The reviewer noted leaf litter on three panos, but measured over a 2%-width window the missed and detected ramps sit in the same light: median luminance 94.6 vs 97.4, 47% vs 42% in shadow. The leaf/shadow reading is not supported. What the miss crops do show, visually and so far unmeasured, is flush or minimal-reveal ramps — a street cross-section close to at-grade, which is the rural design this split was added to probe. The GSV arm of this same town (runs/laurens_gsv, 2,137 panos, 2024-09, leaf-free) is the discriminator and is not yet reviewed: if its recall is also ~0.35 the cause is ramp geometry, if ~0.75 it is rig or season. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
Phase 4-5 of docs/adding_a_benchmark_city.md. Extraction + gate: operating_point_curve.py extract --cities laurens (94 panos, local GPU) low_floor_sweep.py parity --cities laurens -> 111/111, 100.0% EXACT, med R 0.000. Bit-exact, which is what a Mapillary split should be. Registries: US_SPLITS in low_floor_sweep and miss_decomposition, TIER (mapillary), BENCHMARK_SPLITS, SERIES, and SPLIT_HEADER in scoreboard_render (a touchpoint the Phase 5 table does not list). tier_of() already resolves GoPro Max to action-modern, so no new branch. SERIES takes slot 7 (#4a3aa7) -- the categorical slot the manual_gold fold vacated, so no new hue is invented. Pooled numbers move, as the runbook warns they will: thr 0.30 P 0.9002 -> 0.8991 R 0.7927 -> 0.7640 (-2.9 pts) thr 0.55 P 0.9637 -> 0.9594 R 0.7223 -> 0.6864 (-3.6 pts) n_panos 859 -> 953 Precision is untouched; the recall drop is Laurens' own 0.390 entering the pool, not a regression anywhere else. farfield_forensics.py and silent_activation.py both read the live US_SPLITS but are FROZEN studies -- their tests pin the committed result JSONs to numbers quoted in the issues' section 0c. Registering a city silently restated both. Each now carries its own PUBLISHED_SPLITS / US_SPLITS tuple with the reason, and their tests read that instead, so the published findings stay reproducible and a future split cannot rewrite them by accident. Laurens is absent from those two studies for that reason alone; it is pooled everywhere else. Also refreshed: op/*.csv (sweep, hist, gtbias, floor, distance), op_cache/laurens.json, scoreboard.json, model_scoreboard.md and the five scoreboard figures. The split date range test moves to 2026-08-31, Laurens' review date. NOT resolved here -- eight tests in test_scoreboard.py still fail, and they should not be silenced by bumping constants. No challenger model has been scored on Laurens, so every one of them becomes partial coverage (7/8) and drops out of the pooled tables, and test_ap_ordering_is_not_an_artifact_of_the_floor now compares RampNet's 8-split AP against a 7-split YOLO AP -- 0.829 vs 0.730, a margin of 0.0993 against an asserted 0.1. That is a population mismatch, not a stale constant. See the PR for the decision it needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
Laurens is the only city in the benchmark run on both imagery sources over one footprint (area hash d8dd392b...), which is what makes it the control for RampNet#151. With a second arm arriving, the bare city name stops being an identifier and starts being a lie: `laurens` silently meant Mapillary. The convention, now written into docs/adding_a_benchmark_city.md phase 0: A split carries a `_<source>` suffix if and only if its city has more than one imagery arm. So `bend`, `richmond` and the rest stay bare and pay nothing. If bend ever gains a Mapillary arm it gets renamed at that moment, not pre-emptively. Timing is the whole argument for doing this now rather than later. Checked the Hub directly: projectsidewalk/rampnet-benchmark was last modified 2026-08-17, its four configs are per-split parquets named <split>.parquet, and there are zero laurens files -- 0c8e0f4 registered the split in BENCHMARK_SPLITS but never ran build -> verify -> push. So the one irreversible cost of a rename, breaking every existing load_dataset(..., "laurens"), does not exist yet. It will the moment the dataset is republished. Doing it inside this PR also keeps the bare name out of main's history entirely. Mechanical only -- this changes no number, only its label: benchmark/laurens/ -> benchmark/laurens_mapillary/ (4 tracked files; panos/, gallery/ and index.csv are gitignored) analysis_out/op_cache/laurens.json -> laurens_mapillary.json, and its internal "city" key the `split` column of six committed tables under analysis_out/op/, plus scoreboard.json US_SPLITS (low_floor_sweep, miss_decomposition), TIER, BENCHMARK_SPLITS, SERIES, SPLIT_HEADER, and the frozen-study comments in farfield_forensics and silent_activation Two things the blind rewrite got wrong and were fixed by hand. SPLIT_HEADER is documented as "short enough that the by-split matrix stays readable", so it takes `laur_mly` -- mly being this project's own abbreviation for Mapillary, cf. the mly1_public tiles -- not the full 17-character name; the provenance and coverage tables, which have room, keep the full id. And all three model_scoreboard.md hits were inside <!-- BEGIN GENERATED --> blocks, so that file was reverted and regenerated with scoreboard.py rather than edited. Only scoreboard_by_split.png moved; the other four figures rebuilt byte-identical. pytest: 8 failed, 1314 passed, 1 skipped -- byte-identical to the pre-change baseline, same eight test_scoreboard.py tests 0c8e0f4 flagged as unresolved. Nothing here addresses those; they are still the open question on this PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
Phase 1 of docs/adding_a_benchmark_city.md for the second Laurens arm.
Laurens is the only city run on both imagery sources over one footprint
(area hash d8dd392b...), so reviewing the GSV arm separates the two
explanations for its recall 0.390 -- the benchmark's worst by 2x, and the
only split whose misses are NEAR rather than far-field:
GSV recall ~0.35 -> the cause is ramp GEOMETRY. Flush / minimal-reveal
rural ramps are a real blind spot; the fix is training data. Escalate.
GSV recall ~0.75 -> the cause is RIG or SEASON, and Laurens becomes
evidence about consumer 360 rigs and autumn capture instead.
python scripts/export_benchmark.py runs/laurens_gsv/results.jsonl \
--bundle ../RampNet/benchmark/laurens_gsv # in the auto-labeler
86 panos: 5 top / 56 random / 25 empty, from 2,137 GSV panos carrying 473
operational detections over 230 panos. Reconcile is 86/86 with zero decay.
Sampler defaults are untouched (sample 100, empty 25, seed 0, spacing 30 m),
identical to the Mapillary arm, so the two are comparable on the unbiased
column.
The 56 rather than 95 is the same geometric saturation that capped the
Mapillary arm at 64: _spread walks the whole shuffled candidate pool and
the town simply runs out of detection panos 30 m apart. Unbiased column is
81 against that arm's 89 and the usual 120, so both arms' intervals are
wider than other splits'. Do not "fix" this by lowering --min-spacing.
Measured, and it changes the analysis plan: the two independent samples
already overlap heavily, because both are 30 m greedy walks over the same
small road network. 51 of 86 GSV panos (59%) sit within 20 m of a
laurens_mapillary bundle pano; median nearest neighbour 17.2 m, and 7.7 m
across the `top` stratum. So the standard protocol hands us a paired subset
of ~51 corners seen by both rigs for free -- same physical ramps, rig and
season the only variables -- on top of the two independent recalls. A
bespoke site-matched sampler was considered and is not needed, which also
keeps this split comparable with the other ten.
imagery_manifest.json pins the bytes now, before review, so the verdicts
can never be re-paired with re-fetched pixels (--cities laurens_gsv, which
does not require the split to be registered).
NOT registered yet, deliberately. BENCHMARK_SPLITS is test-locked to
ALL_SPLITS minus manual_gold, and split_date_range reads every listed
split's verdicts.json -- so registering an unreviewed split breaks the
suite. Registration is phase 5, after the GT review produces verdicts.
When it lands, the arm goes into HELD_OUT, not US_SPLITS: it samples the
same physical ramps as laurens_mapillary, so pooling both would
double-count them and break the independence the Wilson intervals assume.
That is a third kind of held-out reason, and the convention plus the
CITY_OF mechanism are written up in phase 0 of the runbook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
86/86 panos reviewed. P 0.925 (0.864-0.960) / R 0.505 (0.439-0.570); unbiased 0.904 / 0.459 (0.389-0.531). #151's decision rule was GSV ~0.35 -> ramp geometry, ~0.75 -> rig or season. 0.505 is neither, so the answer is the split verdict the issue listed as a real possible outcome. It decomposes cleanly, though, and the two halves point in different directions: RIG/SEASON IS REAL, AND WORTH ~12 RECALL POINTS. GSV 0.505 vs Mapillary 0.390 is +11.5 pts with barely-overlapping Wilson intervals. The paired subset confirms it is not a sampling artifact -- restricting both arms to the corners both rigs saw reproduces the same gap at every radius: 15 m 33 panos/arm mly 0.415 gsv 0.545 +0.130 20 m 49/51 panos mly 0.393 gsv 0.518 +0.125 25 m 54/57 panos mly 0.398 gsv 0.502 +0.104 THE DEFICIT MOSTLY SURVIVES ANYWAY. 0.505 is still the worst of any US split by ~18 points -- next lowest is paterson at 0.684, on a uniform TP/(TP+FN) recomputation across all ten splits. Sept-2024 leaf-free GSV at 16384x8192 does not rescue Laurens. So roughly a third of the gap is the camera and two thirds is the place, and the escalation in #151 stands. BUT THE SPECIFIC EVIDENCE FOR FLUSH RAMPS WEAKENED, AND THE ISSUE TEXT NEEDS AMENDING. The near-miss signature -- the thing that made Laurens unique among ten splits, and the whole basis for "near, well-resolved ramps are being missed" -- is ARM-SPECIFIC. Median normalized-y delta (missed minus detected), abstentions excluded: nine other splits -0.013 .. -0.030 (misses are far-field) laurens_mapillary +0.0038 (inverted -- the #151 finding) laurens_gsv -0.0053 (back to the normal direction) So on better imagery the inversion disappears. laurens_gsv is still an outlier -- at half the magnitude of the nearest split (richmond -0.0128) it has the weakest far-field tilt in the benchmark -- but it is no longer inverted. "Near, well-resolved ramps are being missed" was substantially a GoPro-Max-in-November artifact, not a property of rural ramps. The flush/minimal-reveal hypothesis therefore survives as an explanation of the LEVEL and loses its distinctive supporting signature. Making it quantitative still needs the curb-reveal measurement from GSV depth (#111), which this arm can supply and the Mapillary arm cannot. Method note: recall here is TP/(TP+FN) with `unsure` misses excluded, which reproduces score_validation.py's all-panos column exactly on both arms; the delta method reproduces every published per-split delta in #151. Missed-ramp abstention is high in both arms -- 20% mapillary, 15% gsv -- itself a signal about how hard these corners are for a human to call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SF2s8wWUM6HphV1xuYju21
…lation mismatch PR #152 left eight test_scoreboard failures unsilenced and asked for a decision: re-run the challengers on Laurens, or hold it out of the comparison basis. This is the first option. Every registered leg now has laurens_mapillary detections, so the scoreboard compares eight splits against eight rather than eight against seven. What the re-run settles: - test_ap_ordering_is_not_an_artifact_of_the_floor passes on its own terms. It read 0.829 (RampNet, 8 splits) vs 0.730 (YOLO, 7) = 0.0993, just under the asserted 0.1. Measured on the same eight splits it is 0.8289 vs 0.7229 = 0.1061. Adding Laurens lowers both -- RampNet 0.849 -> 0.829, y11x 0.730 -> 0.723 -- and the margin widens because the comparison is finally like-for-like. The threshold was never the problem and is untouched. - The pooled operating point moves onto the eight-split basis, and docs/operating_point.md moves with it: 0.55 goes 0.964/0.722/0.826 -> 0.959/0.686/0.800 over n=953. Precision is untouched; the recall drop is Laurens' own 0.390 entering the pool. The F1 optimum is still 0.32 and the recommendation is unchanged. Two results worth recording: - The supervised YOLO baseline beats RampNet on Laurens at the #71 operating point -- y26_pano 0.574 and y11l_pano 0.563 against 0.543 -- the first split where that happens. It is an operating-point result, not a curve result: read full-range at the 0.05 floor RampNet's AP is 0.691, still the highest of the four. The YOLO arms win on recall (0.450, 0.402 vs 0.390) while RampNet keeps precision (0.898 vs 0.794). - The Qwen-32B inversion recurs a fourth time, most extreme instance yet: 0.066 against 8B's 0.210, at 0.27 boxes/pano and recall 0.036. Four splits, three of them HIGH-confidence US GT, so the "it stops firing on unusual infrastructure" mechanism is established rather than suggestive. Against zero-shot challengers RampNet's lead survives its worst split: +0.200 F1 over gemini-3.1-pro. OWLv2's 0.851 recall is density (68.0 boxes/pano), not a usable ceiling. Detections: 11 files, all --verify identical to the cache. The YOLO export needs --tiling none --yolo-imgsz 1280 or the reconstructed signature misses every cache entry and writes empty files without failing; that trap and the exact commands are now in docs/replication.md, whose file-count ledger a test caught going stale at 114/125. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… the rig, not the town The GSV arm was run through the identical leg list as laurens_mapillary, so the two differ only in imagery. That makes it the #151 discriminator, and it answers cleanly. Every zero-shot challenger is flat or WORSE on the GSV arm -- seven of eight move down, none moves up more than 0.008 -- while RampNet gains +0.115 (F1 0.543 -> 0.659, recall 0.390 -> 0.509). If rural streetscape were intrinsically hard, the arm that is easier for RampNet would be easier for them too. It is not. Laurens' 0.390 is RampNet meeting an out-of-domain rig, not a town that defeats detectors. Two consequences: - The RampNet-vs-YOLO ordering FLIPS between the arms. On Mapillary the supervised baseline wins (y26 0.574, y11l 0.563 vs 0.543); on GSV RampNet leads by 0.072 (0.659 vs 0.587). The previous commit's "YOLO beats RampNet on the rural split" is really "on non-GSV imagery" and does not survive changing the rig over the same ground. - Being GSV-trained is not the whole mechanism. The YOLO arms trained on the same GSV-derived dataset barely move (+0.039, +0.024, and y26 goes the other way) against RampNet's +0.115, so RampNet is markedly more rig-sensitive than a YOLO trained on its own data. That points at preprocessing and the 2048x4096 input rather than the training distribution, and is an open question, not a closed one. Caveats travel with the table: the arms are unpaired (249 GT ramps vs 220), and the ~51 corners within 20 m of each other are the paired subset that would turn this into a measurement -- both arms' detections are committed, so it needs no new inference. Registration follows docs/adding_a_benchmark_city.md, which prescribed all of this and described machinery that did not exist. CITY_OF is now real, and so is the one-pooled-split-per-city test it exists for. Held out for non-independence, not GT quality: 59% of gsv panos sit within 20 m of a mapillary one (median NN 17.2 m), so the arms largely see the same physical ramps and pooling both would double-count them. The pooled operating point is unchanged, which is what held-out has to mean. Three latent bugs surfaced, each found by a test rather than by reading: - pool_of() named the three held-out splits literally instead of deriving from HELD_OUT, so a newly held-out split was pooled anyway -- laurens_gsv would have been pooled alongside its own sibling arm while HELD_OUT said otherwise. Now derived. - The AP provenance table had a binary (substituted, or "bundle, already at 0.05") that held only while manual_gold was the sole unsubstituted row. It printed "already at 0.05" against laurens_gsv, whose bundle is truncated at 0.55 with no op_cache to swap in -- a false provenance claim on the page whose job is provenance. Cells now carry bundle_floor and the table distinguishes three cases. - docs/replication.md's file ledger went stale again (125 -> 136). Detections: 11 more files, all --verify identical to the cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
Author
Deep reviewRe-ran the branch head ( Two things changed underneath this PR since it was opened, and the review has to be read against them. PR #153 ( 1. High —
|
| line | prose says | the table beside it says |
|---|---|---|
| 3 | "Eighteen model legs, ten splits" | eleven splits |
| 7–12 | "YOLO11l … 0.604; Gemini 3.1 Pro … 0.608 … a dead heat" | neither model is in the headline table any more (both 7/8, partial) |
| 13–14 | "Its F1 spans 0.80–0.85 across the seven cities, a range of 0.053 … the only strong model that is also stable" | F1 range column: 0.54–0.85 |
| 18–19 | "0.969, above RampNet's 0.958 … against RampNet's 0.728" | RampNet 0.951 / 0.686 |
| 24 | "RampNet is the top score in all ten splits" | eleven; and after #153 it is false on laurens_mapillary (y26_pano 0.574 > 0.543) |
| 26–27 | "clovis … is the worst pooled city for 5 of the 12 models" | RampNet's worst is now laurens |
| 34 | "PR curves, pooled over the seven US splits" | eight |
| 42–44 | "The pool is seven cities, not ten. … The three held-out splits" | eight of eleven; three held-out becomes four after #153 |
The "stability" finding (line 13) is the one that matters scientifically: the sentence is now refuted by its own table, and a reader who trusts the prose gets the pre-Laurens picture.
#153 does not fix this — its model_scoreboard.md diff touches only lines inside the GENERATED markers. Prose edits outside those markers will merge cleanly; write them as "eight US splits" rather than naming laurens wherever possible so the downstream rename does not have to touch them.
4. Medium — the two operating-point figures were not regenerated
plot_operating_point.py gained the laurens SERIES entry and collect() reads ALL_SPLITS, so a regeneration would draw the eighth city — but docs/figures/operating_point_pr.png and docs/figures/storage_floor_ceiling.png are not in the diff (last touched bdd7d55, 2026-08-01). The committed figure is a seven-split plot sitting next to eight-split CSVs. The runbook's Phase 4 checklist has "Both figures regenerated and visually inspected" precisely because "an eighth series changes the legend layout and can collide with the curves". The body's "five figures regenerated" are the scoreboard figures, which are a different script.
#153 does not regenerate them either. Regenerate both here and look at them; #153 will need to run the same two scripts once more after its rename so the legend reads laurens_mapillary and includes the dashed laurens_gsv line it added to HELD_DASH.
5. Medium — the frozen-study fix was applied to two studies; three more have the same exposure
The PR's reasoning is right: a committed study whose script reads the live US_SPLITS is silently restated by every new registration. It was applied to farfield_forensics and silent_activation because their tests failed. But four more committed studies default --cities to the live tuple, and three of them do not even record which splits they covered:
| committed result | script default | records its split list? |
|---|---|---|
analysis_out/miss_decomposition.json |
US_SPLITS (now 8) |
no |
analysis_out/miss_taxonomy.json |
US_SPLITS (now 8) |
no |
analysis_out/silent_witness.json |
US_SPLITS (now 8) |
no |
analysis_out/fp_taxonomy.json |
US_SPLITS (now 8) |
yes — cities: the seven |
A default re-run of any of the first three now overwrites a seven-split result with an eight-split one, and nothing in the payload says which it is. None was re-run for this PR (they are unchanged in the diff), and the body does not say so — per CLAUDE.md, an unstated gap is indistinguishable from a hidden one. The numbers those files feed (docs/operating_point.md's "Pooled over the seven US splits" at lines 212, 265, 409; the #46 taxonomy rates) are therefore still seven-split figures under an eight-split registry.
Fix, minimal: have miss_decomposition.py, miss_taxonomy.py and silent_witness.py write "cities": [...] into their payloads (the same key fp_taxonomy and silent_activation already use), and add a line to the PR body naming these four studies as not re-run for Laurens. Whether to re-run them is a decision (below). #153 touches miss_decomposition.py in a different hunk (the registry lines), so this merges.
6. Medium-low — Phase 6 is not done, and one Phase 5 item is unstated
The body says benchmark/README.md, docs/model_comparison.md and docs/operating_point.md are not updated and gives the reason (the #55 pass). Two things go beyond that:
benchmark/README.mdis untouched by this PR and by The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153. It is the document the runbook calls "where a split actually becomes real": both split tables plus a prose section (what the split is for, its imagery, what fought the rubric, the negative-check result). After both PRs merge it still says "All nine splits are published" (line 81), "the seven US splits" (109), "All nine city splits" (117), "All eight city splits were reviewed" (502). This does not wait on the GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55 pass.BENCHMARK_SPLITSnow includeslaurens, and the HF dataset was not republished. The runbook's Phase 5 row says the allowlist is what the HF package is built from and to "Republish with build → verify → push". Not doing that is defensible — The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 renames the split, and the runbook's new naming section says to rename before the first HF publish because config names are the one irreversible piece — but it needs saying in the body as a deliberate omission, with the note that it must wait for the rename.
Stale split counts that this merge makes wrong and that #153 does not touch, for whoever does the docs pass: docs/model_comparison.md lines 3, 135, 143, 549, 766, 805, 919, 935, 1822; docs/operating_point.md lines 123, 188, 212, 373, 409, 762 (#153 rewrites the pooled block at 262–303 but not these); docs/replication.md "9 splits" at 227, 245, 282, 301, 448, 457 (#153 updates the file ledger, not the config counts).
7. Low
silent_activation.pypins its frozen tuple under the nameUS_SPLITS, the same name the live registry uses in every other module, whilefarfield_forensics.pychosePUBLISHED_SPLITSfor the same thing.from silent_activation import US_SPLITSnow returns a different tuple fromfrom miss_decomposition import US_SPLITS, and the test has to alias it (as PUBLISHED_SPLITS) to say what it means. Rename toPUBLISHED_SPLITSin the script and the test.- The
test_ap_ordering_is_not_an_artifact_of_the_floormargin, 0.0993 against an asserted 0.1, is a population mismatch as the body says — confirmed the other way round by The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153, which only moves the constants after scoring the challengers on the new split. Leaving the eight tests red rather than nudging the thresholds was the right call. - Strata 5/64/25 = 94: the geometric explanation checks out (
sample.jsonrecordsmin_spacing_m: 30,selected: 94from a 4,495-pano run), the unbiased column is 89 panos, and the body says the intervals are wider. Fine as stated.
Decisions for Jon, not fixes
- The body's "decision this needs" is moot. The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 took option 1 — all 11 challengers scored on both Laurens arms — and it is what turns this branch's CI green. The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153's own note gives the merge order: merge The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 into
benchmark/laurensfirst, confirm Add Laurens, IA — two imagery arms over one rural footprint, and the deficit is the rig, not the town (#149, #151) #152 is green, then Add Laurens, IA — two imagery arms over one rural footprint, and the deficit is the rig, not the town (#149, #151) #152 intomain. Merging Add Laurens, IA — two imagery arms over one rural footprint, and the deficit is the rig, not the town (#149, #151) #152 alone puts a red commit onmain. - Reviewer confidence for
review_notes(finding 1): only the person who did the 2026-08-31 pass can supply it. Given 38 abstained miss marks and the leaf notes, "medium" would be the honest default, but that is the reviewer's call, not the reviewer's proxy's. - Re-run the four seven-split studies for Laurens, or pin them as published?
miss_decompositionandmiss_taxonomyare CPU-only from committed caches;fp_taxonomyandsilent_witnessneed challenger detections on the split, which exist only after The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 and only under the new name. Either way the answer belongs in the doc beside the numbers. - HF republish waits for the rename in The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153; say so in the body.
What holds up
- Every script-derived number in the body reproduces to the digit, and the parity gate is bit-exact as a Mapillary split should be.
- The manifest was written before review, its digest re-derives, and the three id sets agree. The archive reconciliation claim (8.3 GB Mapillary, 40 GB GSV) is the one thing a clean clone cannot check; the body says where it lives.
- The frozen-study pinning is correct and tested:
test_the_recorded_scope_is_the_pooled_populationasserts the committed payload'scitiesequals the frozen tuple, andtest_farfield_forensicsbuilds its population fromPUBLISHED_SPLITS. Registering a city can no longer restate either study. - Registration is complete on every touchpoint the runbook lists, plus
SPLIT_HEADERinscoreboard_render.py, which the runbook does not list and which The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 keeps.tier_of()resolvesGoPro Maxtoaction-modernthrough the existing"max" in modelbranch, so no new tier was needed. SERIES slot 7 is the vacated categorical slot, not an invented hue. - Camera provenance is present on all 94 records; the
copyrightfield matches the shape every other Mapillary split carries. - The eight CI failures are exactly the ones named, and refusing to bump the constants was correct.
Fix list
- [F1] High Add a
review_notesblock tobenchmark/laurens/verdicts.jsonin gainesville's shape (reviewer,reviewed_at= 2026-08-31,confidence,summary,caveats). Caveats must include: 38 of 190 missed marks abstained asunsure; leaf litter noted on three panos; strata 5/64/25 with an 89-pano unbiased column. If the reviewer's confidence is not known to you, write the block withconfidenceset to the string"unrecorded — reviewer to fill in"and say so in the PR comment; do not invent a level. Write withnewline="", keep the existing key order, and confirmscore_validation.py benchmark/laurensprints the block above the numbers — files: benchmark/laurens/verdicts.json - [F2] High Edit the PR body (
gh pr edit 152): add an amendment block under "The result" withdrawing the "misses are near, not far-field (+0.004)" bullet, citing Recall collapses on flush / minimal-reveal curb ramps: Laurens R 0.390, and the experiment that isolates it #151's 2026-09-03 amendment and The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 (laurens_gsv−0.0053); state that the y-delta, luminance and shadow figures came from an ad hoc measurement with no committed script; add a "Deliberate omissions" list naming the four studies in F5 as not re-run and the HF republish as waiting on The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153's rename; replace "The decision this needs" with a note that The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 resolves it and gives the merge order. No session links, keep the existing attribution line — files: PR body only - [F3] Medium Rewrite the stale hand prose in
docs/model_scoreboard.mdoutside the GENERATED markers so it agrees with the regenerated tables: line 3 split count; findings 1–3 (lines 7–19: the YOLO/Gemini dead-heat numbers are no longer in the headline table, RampNet's F1 range is 0.54–0.85 across eight cities and the "only stable model" sentence must go or be qualified by Laurens, RampNet's P/R are 0.951/0.686); line 24 "all ten splits"; lines 26–27 worst-city claim; line 34 caption; lines 42–44 pool description. Say "eight US splits" rather than naminglaurenswhere the sentence allows, so The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153's rename does not have to touch it — files: docs/model_scoreboard.md - [F4] Medium Regenerate
docs/figures/operating_point_pr.png(python scripts/analysis/plot_operating_point.py) anddocs/figures/storage_floor_ceiling.png(python scripts/analysis/plot_storage_floor.py), open both and check the eighth series does not collide with the legend or the curves; note in the PR comment that The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 must re-run both after its rename — files: docs/figures/operating_point_pr.png, docs/figures/storage_floor_ceiling.png - [F5] Medium Make
scripts/analysis/miss_decomposition.py,scripts/analysis/miss_taxonomy.pyandscripts/analysis/silent_witness.pywrite a"cities"list into their output payload (same keyfp_taxonomy.pyandsilent_activation.pyuse), without regenerating the committed JSONs; add a module comment in each saying the committed result covers the seven pre-Laurens splits. Add one test that every committed study JSON underanalysis_out/produced by a script with a--citiesdefault either carriescitiesor is named in an explicit allowlist with the reason — files: scripts/analysis/miss_decomposition.py, scripts/analysis/miss_taxonomy.py, scripts/analysis/silent_witness.py, tests/ - [F6] Medium-low Update
benchmark/README.md: addlaurensrows to both split tables (all-panos and unbiased, fromscore_validation.pyoutput) and a prose section in the shape of the other cities (what it is for, GoPro Max / Mapillary / 2025-11 / single uploader / 19 sequences, the leaf and abstention caveats, empty-group result), and fix the split counts at lines 81, 109, 117, 502. Name the splitlaurensas this branch does; The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153 will rename — files: benchmark/README.md - [F7] Low Rename
silent_activation.py's frozen tuple fromUS_SPLITStoPUBLISHED_SPLITSand updatetests/test_silent_activation.pyto import it under that name, matchingfarfield_forensics.py— files: scripts/analysis/silent_activation.py, tests/test_silent_activation.py
🤖 Generated with Claude Code (claude-fable-5-1)
The README was the last place either arm was invisible: no table row, no
section, and a set of hand-maintained split counts that were already stale
before Laurens landed ("All nine splits", "All eight city splits" — while
there were ten).
Adds both arms to the headline and unbiased tables, a § footnote explaining
that the two rows are one town shot twice, and a section on what the second
arm is for: holding the town fixed and changing only the rig separates "rural
is hard" from "this rig is out of domain", and it comes out the second way.
Three counts corrected, and one dropped rather than corrected. "Every city
split was reviewed at model resolution" is a property of gt_gallery.py
(RUBRICS.md §1, issue #26), not of a number someone has to remember to
increment, so the count is gone.
Two gaps recorded rather than papered over:
- The Laurens arms are in BENCHMARK_SPLITS but have NOT been pushed to the
Hub, so "all nine splits are published" would now be false. Nine of eleven.
- Neither arm carries a review_notes block, so the "high confidence" rating
both are quoted at in docs/model_comparison.md is not reproducible from the
bundle — RUBRICS.md §1 says that block is where confidence lives, and
score_validation.py looks for it there. annapolis has the same gap.
sao_paulo, gainesville and budapest_district5 all carry theirs.
Numbers re-derived from the committed bundles with score_validation.py:
laurens_mapillary 94 panos 0.898/0.390 (unbiased 89, 0.863/0.325),
laurens_gsv 86 panos 0.925/0.505 (unbiased 81, 0.904/0.459).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…flat (#151) Runs the benchmark's strongest zero-shot model on both arms of Laurens, which is the sharpest available version of #151's test. If the GSV arm were simply easier imagery, the best general model should gain on it too. It does not. claude-opus-5 at effort low: laurens_mapillary P 0.485 R 0.386 F1 0.430 96/102/153 laurens_gsv P 0.489 R 0.395 F1 0.437 87/91/133 dF1 +0.007 RampNet gains +0.115 across the same two rigs. So the gain is a domain effect specific to RampNet's training distribution, not the imagery being easier to read -- which is what the write-up now says, in place of the weaker version that rested only on models that were already far behind. Claude is the best non-RampNet model on BOTH arms, ahead of molmo2-8B and gemini-3.1-pro on each. That changes a published number: laurens_mapillary's "0.200 lead over the best zero-shot challenger" was measured against gemini-pro and is now 0.113. The doc says so explicitly rather than quietly editing it. The "seven of eight move down" recount is now seven of nine, and neither of the two that rise clears +0.008. The mapillary arm needed a second pass. Two panoramas (2102336717175440, 2281219182305735) died on Vertex's transient 404 even after the detector's four retries; compare.py isolated them and scored the other 92, which would have published a recall against 247 GT ramps under a caption saying 249. Recovered for 12 calls and $0.14. Same defect class as the annapolis sonnet/low pano. tests/test_claude_annapolis_leg.py -> test_claude_published_legs.py, generalized from one split to three. It still recomputes every published Claude number from committed detections with no cache, no key and no GPU, and it now asserts whole-split coverage per split (the guard that catches exactly the failure above) and that both Laurens arms are the SAME leg -- a rig comparison whose two halves came from different effort levels would attribute to imagery a difference that is really a configuration change. test_scoreboard's single-split assertion is now per-leg: opus/low covers 2 of 8 pooled splits, not 1, and partial coverage still means excluded from the pooled tables. The point of that test is the exclusion, not the number. Spend: 1,080 calls, 2,193,480 in / 72,534 out tokens, ~$12.79, logged to analysis_out/usage_log.jsonl at run time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jonfroehlich
changed the title
jonfroehlich
added a commit
that referenced
this pull request
Sep 4, 2026
…the review findings (#51) Twelve findings were raised on this PR. Eleven are real; the fixes are below, largest first. Pin the split population in both scripts (was: the live registry). Both read US_SPLITS/ALL_SPLITS from analysis.low_floor_sweep, and macro_at/pooled return None unless every pooled split has a report. #152 and #153 have since merged, so main now carries laurens_mapillary in US_SPLITS and laurens_gsv in CITY_SPLITS, and neither has a YOLO report: measured against main's registry, this branch fails 12 of 37 tests, both --check commands, and crashes with a TypeError in _render. Worse than failing, RampNet has an op_cache for laurens_mapillary while the YOLO legs do not, so a merged version would re-pool one side of the comparison over a different population than the other. POOLED_SPLITS and ALL_SPLITS_AS_RUN are now module constants asserted literally in the tests, and the held-out keys are pinned to the run while only their reason text follows the registry. Both committed artifacts are unchanged, and both --check commands pass under main's registry as well as this branch's. Correct the control's stated mechanism. The -0.0025 between RampNet's op_cache re-score and its published bundle at 0.55 was attributed to peak extraction -- that a 0.05 floor finds a different peak set than a 0.55 floor. That cannot happen: peak_local_max suppresses on a maximum filter, so lowering threshold_abs can only add candidates, and the >= 0.55 subset of a 0.05-floor extraction is exactly a 0.55-floor extraction. The real causes are already measured in docs/operating_point.md: the four GSV splits' shipped detections came from a 4096x2048 resample rather than the native-resolution cache, and manual_gold's carry flip-TTA. Five Mapillary splits agree to 0.0005; the other five differ by up to 0.019. That matters beyond the wording. The pooled macro cleared its 0.005 tolerance partly by cancellation (+0.003 bend against -0.004 paterson and -0.016 gainesville), so a scoring regression could have passed it. The control is now asserted per split as well: 0.002 on the five splits where the two paths are the same computation, 0.025 on the five with a documented reason to differ. The per-split table is in the artifact and printed by the script, and a test proves the per-split check can fail. Two further consequences are now stated with the numbers: the default dev split has the largest discrepancy of the ten, and because three pooled splits are GSV, RampNet's parity F1 is about 0.002 low -- the reported gap is slightly conservative. Make check_yolo_dataset_loads.py detect the failure its docstring claims. It said a large nm was the signature of a rebuild that dropped the zero-byte background labels, but it never read nm: Ultralytics appends a zero-box record for a missing label file exactly as for an empty one, so deleting all 59,923 background labels leaves the image count, box count and zero-box count unchanged and the script prints PASS. It now stats the label paths and fails on any absent file, takes --expect-background, and says that augment=False only warns on nf == 0 rather than raising. Qualify the per-split reads in docs/operating_point_parity_51.md. The document's own rule is that differences under ~0.02 F1 should not be read; section 2 then read 0.009, 0.017 and 0.001 as two losses and a tie. All three are now reported as ties, with the manual_gold margin flagged as the same size as the measured TTA-vs-no-TTA delta on that split, and the paired test that would settle it recorded as blocked on the tiles arm's detections being unpublished. Replace the stale "tiles arms are still training" statements in model_scoreboard.md, model_comparison.md and yolo_baseline/README.md, and add a caveat beside the scoreboard's headline table -- outside the generated region -- saying the YOLO rows are at the unselected 0.25 default and pointing at both new documents. Add provenance and cost to docs/yolo_geometry_51.md: the eval was 67 min 39 s on one A40, about 1.13 GPU-hours at no cost; the control leg was re-scored from cached detections rather than re-inferred, so it isolates the matcher and does not test inference; the checkpoints live only on cluster storage, so the run is not reproducible from a clean clone; and the rescue job ids are not in the repo. The driver now logs git status --porcelain in full instead of a path count. Smaller items: rebuild_yolo_labels_from_cache.py no longer claims byte-identical output for files that held duplicate rows (Ultralytics dedups and re-sorts them) and writes with newline="\n"; the dev-split fairness test now asserts on build()'s output for every candidate instead of a tautology; the at_floor test is no longer vacuous; PUBLISHED_RAMPNET_F1 is imported rather than restated; the AP table cites where RampNet's 0.849 comes from; and both scripts are indexed in scripts/analysis/README.md. Not fixed: the request to replace "load-bearing" with plain wording. It appears 38 times across more than 20 committed files in this repo, including low_floor_sweep.py, scoreboard.py and four existing docs, so it is house usage rather than something this branch introduced. "split-brain", which was new here, is gone as part of the control rewrite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Laurens, Iowa — the first rural split — and works it through
docs/adding_a_benchmark_city.md. Companion to ProjectSidewalk/sidewalk-auto-labeler#48; findings in #149 and #151.Why this split exists (Phase 0)
Rural / small-town streetscape on a consumer 360 rig. Every existing split is urban, suburban or a college town; Laurens is 1.91 km² and 1,264 people. Pooled — US deployment city, complete verdict-grade GT.
The result
Recall is roughly half the next-worst split (clovis 0.650). It is not a weak or ramp-poor sample: Laurens is ramp-rich (2.65 ramps/pano, 3rd of 10), misses/pano 1.62 vs 0.45–1.19 elsewhere, and — uniquely among all ten splits — its misses are near, not far-field (delta +0.004 in normalized y; every other split is −0.013 to −0.030). The reviewer's leaf-litter reading did not survive measurement. Full analysis and the discriminating experiment: #151.
Parity gate: 111/111, 100.0% exact, med R 0.000 — bit-exact, as a Mapillary split should be.
Pooled numbers move
n 859 → 953 panos. Precision untouched; the recall drop is Laurens' own 0.390 entering the pool.
Two published studies pinned
farfield_forensics.pyandsilent_activation.pyread the liveUS_SPLITSbut are frozen studies — their tests pin the committed result JSONs to numbers quoted in the issues' §0c. Registering a city silently restated both published findings. Each now carries its own explicit split tuple with the reason. This is a latent trap for every future split, not something specific to Laurens, and is probably worth a line in the runbook.Checklist
Phase 1 — bundle
d8dd392b…)export_benchmark.py --bundleexited zero;index.csvwritten, zero decayrecords.jsonlcommitted;panos/archived durably on makelab2 (8.3 GB Mapillary + 40 GB GSV, both reconciled 1:1, zero decay)_spreadwalks the whole pool and the town saturates at 64 detection panos held 30 m apart. Unbiased column is 89 vs the usual 120, so intervals are wider. Strata definitions and de-clustering unchanged.Phase 2–3 — ground truth
gt_gallery.pyat model resolutionreview_noteswritten · [x]verdicts.jsoncommitted · [x]score_validation.pyrun, unbiased recordedGoPro/GoPro Max, all 94)imagery_manifest.jsonwritten before review (7485886ea08f671b) — a step the runbook still does not mentionPhase 4 — operating point
sweep/hist/gtbias/floor/distancere-run;op_cache/laurens.json+op/*.csvcommittedcorrected/tagcheck— blocked on the abovePhase 5 — code
US_SPLITS/CITY_SPLITS/ALL_SPLITS· [x]TIER· [x]BENCHMARK_SPLITS· [x]SPLIT_HEADERHELD_OUT— n/a, pooledtier_of()already resolves GoPro Max →action-modernSERIESslot 7 (#4a3aa7), vacated by the manual_gold fold — no invented huepytest -q— 8 failures intest_scoreboard.py, deliberately not silenced (see below)Phase 6 — docs
benchmark/README.md,docs/model_comparison.md,docs/operating_point.md— not updated; the corrected tables need the GT-completeness correction: spot-check the low-confidence incremental FPs from the operating-point curve #55 pass, and the scoreboard question below changes what the model-comparison text should say.The decision this needs
No challenger model has been scored on Laurens. Consequences, all real rather than cosmetic:
test_ap_ordering_is_not_an_artifact_of_the_floornow compares RampNet's 8-split AP (0.829) against YOLO's 7-split AP (0.730) — margin 0.0993 against an asserted 0.1. That is a population mismatch, not a stale constant, and bumping the threshold would paper over exactly the artifact the test exists to catch.Two ways out — your call, which is why the constants are untouched:
🤖 Generated with Claude Code (claude-opus-5[1m], effort: high)