Recall collapses on flush / minimal-reveal curb ramps: Laurens R 0.390, and the experiment that isolates it

[jonfroehlich]

⚠️ AMENDED 2026-09-03 — the GSV arm has been reviewed, and it answers this issue partly against the text below

GSV recall is 0.509 (verdict scorer 0.505), between the two pre-registered outcomes, so the answer is the split verdict this issue listed as a real possible outcome.

• Rig/season is real and worth ~12 recall points, confirmed on the paired subset of corners both rigs saw.
• The deficit mostly survives anyway — 0.509 is still the worst US split by ~18 points (next lowest paterson 0.684) — so the escalation below stands.
• The second bullet under "The finding" is withdrawn. The positive near-miss delta is arm-specific: laurens_gsv reads −0.0053, back to the normal far-field direction. "Near, well-resolved ramps are being missed" was substantially a GoPro-Max-in-November artifact, not a property of rural ramps. The flush-ramp hypothesis is not refuted, but its most specific evidence is gone.
• Separately: the town is not the problem. Every zero-shot challenger is flat or worse on the arm RampNet prefers, while RampNet gains +0.115 F1 — so Laurens is RampNet meeting an out-of-domain rig, not a town that defeats detectors.

Full numbers, the paired-radius table and the caveats are in the comment below.

The finding

Laurens (#149), the first rural split, scores precision 0.898 / recall 0.390 — precision is mid-pack, recall is roughly half the next-worst split (clovis 0.650). The evidence says this is a distinct failure mode, not a weak sample:

• Ramp-rich, not ramp-poor. 2.65 ramps/pano, 3rd of 10 splits. Misses/pano is 1.62 against 0.45–1.19 everywhere else.
• Not the far-field falloff. [WITHDRAWN 2026-09-03 — arm-specific; see the amendment above.] In all nine other splits missed ramps sit nearer the horizon than detected ones (delta −0.013 to −0.030 in normalized y). Laurens is the only split with a positive delta (+0.004): near, well-resolved ramps are being missed. This is orthogonal to the annapolis far-field finding and to The recall-by-distance axis is stretched ~25-30%: recompute it from GSV depth #112.
• Not shadow or leaf litter. The imagery is November Iowa and the reviewer flagged leaves on three panos, but over a 2%-width window missed and detected ramps sit in the same light — median luminance 94.6 vs 97.4, 47% vs 42% in shadow.

Inspecting the miss crops, the ramps being missed look flush or minimal-reveal: a street cross-section close to at-grade, little or no curb face, the ramp readable mostly from a subtle grade change and a joint line. That is the standard rural/small-town design, and it is absent from every other split — all nine are urban, suburban, or a college town.

Hypothesis: RampNet keys substantially on curb-face contrast, so recall degrades as curb reveal goes to zero — independent of range, resolution and lighting. If true it is a systematic blind spot over a large fraction of the rural US, and it is invisible in the current pooled numbers because no other split contains the geometry.

The experiment

Laurens is the only city in the benchmark with both imagery sources over one footprint (area hash d8dd392b…), which makes it a natural control:

arm	panos	capture	rig	reviewed
Mapillary	4,495	2025-11 (leaf litter, low sun)	GoPro Max 5760×2880	✅ R 0.390
GSV	2,137	2024-09 (leaf-free, high sun)	GSV, 16384×8192 native	❌

Reviewing the GSV arm — one ~94-pano bundle, an afternoon — discriminates cleanly:

• GSV recall ≈ 0.35 → the cause is ramp geometry. Flush ramps are a real blind spot; the fix is training data, not imagery. Escalate.
• GSV recall ≈ 0.75 → the cause is rig or season, and Laurens becomes evidence about consumer 360 rigs and autumn capture rather than about rural design.

Either outcome is worth having, and no other city can produce it.

Cost: python scripts/export_benchmark.py runs/laurens_gsv/results.jsonl --bundle benchmark/laurens_gsv in the auto-labeler, then the usual review. Detection is already done — 2,137 panos, 473 operational detections, 10.8% of panos.

Caveats

• "Flush / minimal-reveal" is a visual read of ~20 miss crops, not a measurement. Making it quantitative needs a curb-reveal estimate per ramp. The GSV arm can supply one — GSV serves per-pano depth (GSV serves a native per-pano depth map (streetlevel already parses it): an independent anchor for #101 and the visibility test #102 needs — archive it before the endpoint changes #111, Harvest GSV depth for every processed pano before it goes away (~1 GB for all four GSV cities) sidewalk-auto-labeler#41), already archived, so curb reveal is recoverable there. Mapillary serves none.
• The two arms differ in rig, resolution, season and 14 months at once, so a split verdict ("some of each") is a real possible outcome and should be reported as such rather than forced either way.
• Laurens' unbiased column is 89 panos against the usual 120 (the town saturates the 30 m de-clustering at 64 detection panos), so its intervals are wider than other splits'.

Related

• Add Laurens, Iowa from Mapillary? #149 — the Laurens split
• The recall-by-distance axis is stretched ~25-30%: recompute it from GSV depth #112, annapolis far-field — the failure mode this is not
• Experiment: curb-ramp detection performance vs. input resolution (Mapillary high-res) #25 — input-resolution sweep. Related but a different question: Experiment: curb-ramp detection performance vs. input resolution (Mapillary high-res) #25 varies what the model is fed (above 4096×2048); this varies the source detail reaching a fixed input. Also, Experiment: curb-ramp detection performance vs. input resolution (Mapillary high-res) #25's premise that "GSV z3 is natively 4096×2048, no headroom" is stale — GSV is natively 16384×8192 for ~92% of panos, more headroom than Mapillary.
• GSV serves a native per-pano depth map (streetlevel already parses it): an independent anchor for #101 and the visibility test #102 needs — archive it before the endpoint changes #111 — GSV depth, the route to a measured curb reveal

🤖 Generated with Claude Code (claude-opus-5[1m], effort: high)

[jonfroehlich]

An off-the-shelf instrument for the rig/season branch: the Global Streetscapes quality classifiers

While surveying ZenSVI (MIT, NUS Urban Analytics Lab, CEUS 2025) for reuse opportunities, one piece landed on this issue: it packages the eight Global Streetscapes classifiers — per-image trained scores for glare, lighting, quality, reflection, weather, view direction, platform, and panorama status — behind a two-line API (ClassifierGlare, ClassifierQuality, etc.).

Two places it fits:

1. If the GSV-arm review lands on the rig/season branch (GSV recall ≈ 0.75), the follow-up question is which attribute of the Mapillary arm is responsible. The luminance/shadow probe above already rules out simple lighting at the miss sites; these classifiers add trained, independent signals (glare, reflection, overall quality) that can be compared across the two arms on the same footprint — the same control structure this issue already exploits.

2. Regardless of branch: a one-script sweep over the benchmark bundles (local files, no network; the classifiers are cheap) would put per-split quality-attribute distributions next to per-split recall. The motivating pattern: the two lowest-recall splits are the two consumer GoPro rigs — laurens R 0.390 (GoPro Max) and clovis R 0.650 (GoPro Fusion) — and the Mapillary records already store camera make/model, so rig identity and measured image quality can be separated rather than conflated.

Known limitation: these are whole-image scores. Within-split FN-vs-TP contrasts are the weak use (misses and hits frequently share a pano at 2.65 ramps/pano); the cross-arm and cross-split comparisons above are the strong ones.

Cost: pip install zensvi in a scratch env — it pulls a torch model zoo, so it should stay out of both repos' requirements — one script, single GPU or CPU.

🤖 Generated with Claude Code (claude-fable-5)

[jonfroehlich]

The GSV arm is reviewed, and it splits the difference: R 0.509

86/86 panos reviewed against the same rubric, same reviewer, same footprint. P 0.933 / R 0.509 / F1 0.659 on the scoreboard's uniform TP/(TP+FN) (112 tp / 8 fp / 108 fn of 220 GT ramps); the verdict scorer with abstentions excluded reads P 0.925 / R 0.505 (Wilson 0.439–0.570).

This issue pre-registered a decision rule — GSV ≈ 0.35 → ramp geometry, GSV ≈ 0.75 → rig or season. 0.505 is neither, so the answer is the split verdict the issue listed as a real possible outcome. It decomposes cleanly, and the two halves point in different directions.

1. Rig/season is real, and worth ~12 recall points

GSV 0.509 against Mapillary 0.390 is +11.9 points with barely-overlapping Wilson intervals. That is not a sampling artifact — restricting both arms to the corners both rigs saw reproduces the same gap at every radius:

radius	panos (mly/gsv)	mly R	gsv R	Δ
15 m	33 / 33	0.415	0.545	+0.130
20 m	49 / 51	0.393	0.518	+0.125
25 m	54 / 57	0.398	0.502	+0.104

2. The deficit mostly survives anyway

0.509 is still the worst of any US split by ~18 points — next lowest is paterson at 0.684, on a uniform recomputation across all ten splits. September-2024, leaf-free GSV at 16384×8192 does not rescue Laurens. So roughly a third of the gap is the camera and two thirds is the place, and the escalation in this issue stands.

3. But the specific evidence for flush ramps weakened — the issue text is amended

The near-miss signature is what made Laurens unique among ten splits and was the entire basis for "near, well-resolved ramps are being missed". It is arm-specific. Median normalized-y delta (missed minus detected), abstentions excluded:

split	delta
the nine other splits	−0.013 … −0.030 (misses are far-field)
laurens_mapillary	+0.0038 (inverted — the finding in the issue text)
laurens_gsv	−0.0053 (back to the normal direction)

On better imagery the inversion disappears. laurens_gsv is still an outlier — at half the magnitude of the nearest split (richmond −0.0128) it has the weakest far-field tilt in the benchmark — but it is no longer inverted. "Near, well-resolved ramps are being missed" was substantially a GoPro-Max-in-November artifact, not a property of rural ramps. The flush-ramp hypothesis is not refuted; its most specific piece of evidence is withdrawn. The issue body now carries an amendment block saying so.

The challenger cross-check: the rig, not the town

The GSV arm was then run through the identical leg list as laurens_mapillary — 11 challengers, every model on both arms, so imagery is the only thing that moves. ΔF1 is laurens_gsv minus laurens_mapillary:

model	Mapillary (GoPro Max)	GSV (zoom 5)	ΔF1
rampnet	0.543	0.659	+0.115
y11x_pano_h200	0.529	0.568	+0.039
y11l_pano	0.563	0.587	+0.024
grounding-dino-base	0.045	0.054	+0.008
gemini-3.6-flash	0.277	0.274	−0.003
owlv2-large-patch14-ensemble	0.062	0.055	−0.007
gemini-3.7-flash	0.281	0.261	−0.020
allenai/Molmo2-8B	0.339	0.307	−0.032
y26_pano	0.574	0.538	−0.036
Qwen3-VL-32B-Instruct	0.066	0.018	−0.048
Qwen3-VL-8B-Instruct	0.210	0.161	−0.049
gemini-3.1-pro-preview	0.343	0.279	−0.064

Three readings, in decreasing order of how well the data supports them:

1. The town is not the problem. Every zero-shot challenger is flat or worse on the imagery RampNet prefers — seven of eight move down, none moves up by more than 0.008. If rural streetscape were intrinsically hard, the arm that is easier for RampNet would be easier for them too. It is not. Laurens' 0.390 is RampNet meeting an out-of-domain rig, not a town that defeats detectors.

2. The RampNet–YOLO ordering flips between the arms. On Mapillary the supervised baseline wins (y26_pano 0.574, y11l_pano 0.563, against 0.543); on GSV RampNet leads by 0.072 (0.659 against y11l_pano's 0.587). "YOLO beats RampNet on the rural split" is really "on non-GSV imagery", and it does not survive changing the rig over the same ground. This qualifies the matched-operating-point result in Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51 rather than contradicting it.

3. Being GSV-trained is not the whole mechanism — this is the open question. The YOLO arms were trained on the same GSV-derived dataset, so if in-domain imagery were sufficient they should gain like RampNet did. They barely move: +0.039, +0.024, and y26_pano goes the other way. RampNet's +0.115 is three to five times larger, so RampNet is markedly more rig-sensitive than a YOLO trained on its own data. That points at its preprocessing and 2048×4096 input rather than at the training distribution alone — and it connects directly to Experiment: curb-ramp detection performance vs. input resolution (Mapillary high-res) #25.

Caveats

• The cross-arm table is unpaired. Different panorama sets, different GT (249 ramps against 220), so ΔF1 compares two samples of one town rather than the same corners twice. The ~51 corners inside 20 m of each other are the paired subset that would turn this into a measurement, and both arms' detections are now committed, so that analysis needs no new inference. (The recall half of it is already done — the radius table above.)
• AP is not comparable across the two arms for RampNet. laurens_mapillary has a 0.05-floor op_cache and laurens_gsv does not, so its 0.494 is truncated at the deployed 0.55 while the other arm's 0.691 is not. P, R and F1 are comparable — both arms are read at 0.55.
• laurens_gsv is held out of the pooled basis, for non-independence and not for GT quality: 59% of its panos sit within 20 m of a laurens_mapillary one (median nearest neighbour 17.2 m), so the two arms largely see the same physical ramps and pooling both would double-count them. The pooled operating point is unchanged, which is what held-out has to mean.
• The curb-reveal measurement this issue proposed is still not done. GSV depth (GSV serves a native per-pano depth map (streetlevel already parses it): an independent anchor for #101 and the visibility test #102 needs — archive it before the endpoint changes #111) is the route, and the GSV arm can now supply it.

Where it landed

Branch fix/laurens-challenger-coverage, commits e9dfd86 (GT review) and d388f17 (registration + all 11 challengers). Docs: docs/model_comparison.md (both arm sections and the rig-not-town table), docs/model_scoreboard.md, docs/adding_a_benchmark_city.md. Registering the arm surfaced three latent bugs, each found by a test rather than by reading — the load-bearing one is that pool_of() named the three held-out splits literally instead of deriving from HELD_OUT, so laurens_gsv would have been pooled alongside its own sibling arm while HELD_OUT said otherwise.

🤖 Generated with Claude Code (claude-opus-5[1m])