Vistas at resolution parity: the handicap was real, and it was recall (#126) by jonfroehlich · Pull Request #137 · ProjectSidewalk/RampNet

jonfroehlich · 2026-08-18T22:03:05Z

Closes the open caveat on the Mapillary Vistas arm from #129: the published richmond numbers were
measured at 384×384 while every other tiled leg sees 1024×1024, so the 0.338 F1 gap to RampNet was
an upper bound, not a measurement. --vistas-input-size landed in #129 for exactly this run.

Docs only. No code, no detection artifacts, no roster change.

Pre-registered read

Posted to #126 before the scored output was read (comment 5334506091): gap closes by
< 0.05 F1 ⇒ "transfers but does not compete" stands, one split, stop. More ⇒ revise and cost
3–4 further splits.

It stands, and not narrowly

RampNet re-ran alongside as the comparability check and reproduced its committed richmond row to
every digit (0.964 / 0.768 / 0.855, AP 0.763, 238/9/72).

arm	model input	P	R	F1	AP	tp/fp/fn
rampnet (committed, reproduced)	—	0.964	0.768	0.855	0.763	238/9/72
vistas curb-cut — published (RTX 3070)	384×384	0.411	0.697	0.517	0.513	216/309/94
vistas curb-cut — same-env control (A40)	384×384	0.411	0.694	0.516	0.510	215/308/95
vistas curb-cut — parity (A40)	1024×1024	0.383	0.884	0.534	0.649	274/442/36

F1 0.516 → 0.534, +0.018 — about a third of the bar. RampNet's lead goes 0.339 → 0.321.

The mechanism is the finding, and it is not what the caveat predicted

The handicap was real and large. It was almost entirely a recall handicap:

Recall 0.694 → 0.884 (+0.190); misses fall 95 → 36, a 62% reduction. At parity this arm
out-recalls RampNet (0.884 vs 0.768) while never having seen one of our labels.
AP 0.510 → 0.649 (+0.139), 27% relative.
Precision 0.411 → 0.383 (−0.028) — slightly worse. FPs rise faster than TPs.

F1 stayed flat only because precision is the binding constraint and resolution does nothing for
it. That sharpens the conclusion #126 was built to test: the concept is findable, the
discrimination is not now holds against the supervised arm at equal input, no longer confounded
with how many pixels the model was given.

An env control that wasn't in the plan, and earned its place

The published run was on an RTX 3070 on an older transformers; makelab2 is a major version on,
and the transformers version is not in the detection signature — a hazard the doc already
flags. So parity-vs-published would have differed in two things.

Re-running 384 in the parity env separates them: it lands within one detection out of 523 of
the published run (215/308/95 vs 216/309/94, F1 0.516 vs 0.517). The 4.x→5.15 jump is benign for
this checkpoint, the residual is fp16 kernel nondeterminism, and the whole parity delta is
attributable to input size. That also retires "an upgrade could have changed every mask under an
unchanged cache key" as a measurement rather than an assurance.

Two claims at 384 that parity corrects

The YOLO AP comparison said richmond AP 0.748 / 0.724 / 0.536 are "all above 0.513". At parity
the arm is 0.649, so y26_pano (0.536) no longer clears it. The sentence's conclusion
holds — the two stronger YOLO arms still lead — its arithmetic does not.
min_area_px=16 is inert at 384 but not at 1024. At 384 the mask upsamples 10.67× so the
smallest blob is ~114 px; at 1024 it upsamples 4× so the smallest blob is ~16 px, exactly the
floor. The two rows share the value, not its meaning. Left untouched so as not to confound the
one variable.

Deliberate omissions

Parity detections are not published. --vistas-input-size does not change the arm's label,
so exporting would overwrite mask2former-vistas-curb-cut__richmond.json. Publishing them needs
a distinct name (the --publish-as pattern from Wire Claude into the model-comparison harness (#122): Opus 5 at low effort takes the top challenger slot on annapolis #123). Out of scope here.
richmond only, per the decision rule.
richmond only for the complementarity read below too.

Verification

Ran on makelab2 (A40, fp16, transformers 5.15.0 / torch 2.13.0+cu130) in a scratch worktree with
a private --cache-dir, so nothing shared a cache directory with the published detections.
Instrument checked before the run, not assumed: the override reaches the model —
pixel_values (1, 3, 384, 384) → (1, 3, 1024, 1024), mask logits (1, 100, 96, 96) →
(1, 100, 256, 256). A silently no-opping processor.size on a new major version would have made
"parity" a second 384 run wearing a different cache key. Also re-confirmed the 65-class head
(class 9 Curb Cut, class 2 Curb).
Cost measured, not extrapolated — the estimate going in was 3–4× and was wrong in the cheap
direction: full 124-pano run 3m38s on one A40; GPU forward 0.078 → 0.092 s per view
(1.17×), because Swin's windowed attention scales far better than pixel count. Peak GPU
memory 1.64 GB.
pytest -q: 1238 passed, 1 skipped, including Actually fix the two collapsed runbook snippets from #129, and guard the class #136's runbook line-continuation test against
the new shell block.

Follow-on in this PR: complementarity (the #35 gate)

The parity result raised the recall-first question, so it is answered here rather than filed.
Scoring-side only, on the same cached detections.

	vistas @384	vistas @1024 (parity)
challenger ONLY (rampnet-miss ∩ hit)	21	54
found by NEITHER	51	18
of rampnet's 72 misses, recovered	21 (29%)	54 (75%)
null on that subset	0.090	0.143
attributable after the null	~15	~44
oracle-union recall	0.835	0.942
boxes/pano · above chance	4.5 · 0.657	6.2 · 0.864

Discounted for chance, a free zero-training model finds ~44 of the 72 ramps RampNet misses —
61%. The null is measured on the miss subset, not extrapolated from the split-wide one, because
RampNet's misses are a biased sample and that is where density differs; it landed at 0.143 against
0.145, so the extrapolation would have been fair — a result, not an assumption.

The 384 control is the interesting half. Parity moved F1 by +0.018 and was correctly judged not
to change the ranking — but it nearly tripled the attributable complementary gain (~15 → ~44)
and shrank the found-by-nobody core from 51 to 18. A flat headline metric hid a large change in
the structure underneath it.

A naive union is still dead: F1 0.555 vs RampNet's 0.855. The 54 ramps arrive with 442 false
positives — ~8.2 FPs per recovered ramp, against the 9 FPs RampNet pays for 238 TPs. The oracle
0.942 is recorded as the ceiling it is, with the real union arithmetic beside it.

The cascade was then run — see below.

Tooling changes

complementarity.py generalized past its Gemini-only form (it called
build_detector("gemini", ...), so this arm could not go through the gate at all). Takes a model
spec now; the documented positional form still works. Verified against the committed paterson
numbers — 36 of 124, 88 found by no model, oracle-union 0.777, reproduced exactly.
--vistas-* args added to null_recall.py. build_detector reads them via getattr, so
without them the null there was silently the 384 arm even when analysing parity.
Provider defaults now come from roster.PROVIDER_DEFAULTS via the same shim shape
fp_taxonomy uses, so the reconstructed cache key cannot drift from the one compare.py wrote
under — a wrong default does not crash, it silently misses every entry and reports zero.
The oracle-union number now prints its counterweight: the FP bill a real union pays and the
resulting P/R/F1. On paterson/gemini-pro the union genuinely wins (0.821 vs 0.805), which is the
check that the line is not hardcoded to one verdict.

pytest -q: 1238 passed, 1 skipped after every step.

Also in this PR: the operating-point correction, and the cascade gate

The complementarity numbers above are against a RampNet we do not recommend

The committed bundle detections are the shipped operating point (≥ 0.5519 on richmond); this
document has recommended 0.30 since #54/#55 (PR #79). Verified op_cache is the same source
before re-basing: filtered at ≥ 0.5519 it reproduces the published row exactly (238/9/72). At 0.30
it gives 257/28/53, P 0.9018 / R 0.8290 / F1 0.8639, matching committed corrected_at_0.3.csv.

	rampnet @0.55	rampnet @0.30 (recommended)
rampnet recall / F1	0.768 / 0.855	0.829 / 0.864
rampnet misses	72	53
challenger recovers	54 (75%)	38 (72%)
attributable after null	~44	~30
naive union F1	0.555	0.549

~14 of the recoveries are ramps RampNet already has at the recommended point. Deployable gain
is ~30, not ~44 — the correction is applied in the doc, next to the original figure rather
than replacing it. The recovery rate barely moves (75% → 72%): fewer misses to find, not a
different kind of miss.

`cascade_gate.py` (new): the cascade is live, ceiling ~19 ramps

Partitions all 310 GT ramps into the four cells and reads RampNet's heatmap at each, reusing #46
Phase 1's instrument verbatim. Read pre-registered on #126 before running. At rampnet@0.30, of the
38 genuinely-complementary ramps:

what RampNet has there	n	meaning
floor peak in radius, 0.05–0.30	19	promotable — the cascade's real target
floor peak ≥0.30 but unmatched	4	greedy matcher gave it to an adjacent GT — matcher/σ (#130), not threshold
no floor peak in radius	15	nothing to promote; `act` 0.215 is unpeaked mass

Ceiling ≈ 19 ramps, +6.1 recall points (0.829 → 0.890), FP cost unmeasured. Far below the 54
the raw complementarity suggested.

Negative worth keeping: RampNet's activation does not discriminate — challenger_only null
percentile 0.88 vs the hard core's 0.925, so the ramps nobody finds look stronger on raw heatmap
mass. No self-gating shortcut exists.

An instrument bug I caught mid-run

nearest_peak expects floor peaks (≥0.05); I first fed it the shipped ≥0.5519 detections, so
"is there a peak near this missed ramp?" was really "is there a 0.55+ detection near it?" — close
to definitionally no for a miss. It made every miss look peakless (nearest peak a median 104 px
away) and I nearly wrote up "the cascade is dead". Fixed in dcb9c32, artifacts regenerated. The
greedy match that defines the miss set still uses the bundle records, since that is what
produced the published 238/9/72 — two inputs, two questions, now documented.

New committed artifacts: analysis_out/cascade_gate.json, analysis_out/cascade_gate_op030.json
(both whitelisted in .gitignore, written with newline="" so a Windows re-run stays
byte-comparable).

🤖 Generated with Claude Code (claude-opus-5[1m])

…#126) The published Vistas richmond numbers were measured with the checkpoint's own 384x384 preprocessor in force, so every 1024x1024 view was downsized to ~1/7 the pixel area while every other tiled leg sees the full view. That made the 0.338 F1 gap to RampNet an upper bound rather than a measurement. #129 added --vistas-input-size for exactly this run; this is it. Read pre-registered on #126 before the scored output was read: gap closes by < 0.05 F1 => "transfers but does not compete" stands, one split, stop. Result: it stands, and not narrowly. F1 0.516 -> 0.534 (+0.018), about a third of the bar. RampNet's lead goes 0.339 -> 0.321. The mechanism is the actual finding, and it is not what the caveat predicted. The handicap was large but almost entirely a RECALL handicap: recall 0.694 -> 0.884 (+0.190; misses 95 -> 36, a 62% reduction) AP 0.510 -> 0.649 (+0.139, 27% relative) precision 0.411 -> 0.383 (-0.028, slightly worse) At parity this arm out-recalls RampNet (0.884 vs 0.768) while never having seen one of our labels. F1 stayed flat only because precision is the binding constraint and resolution does nothing for it -- which sharpens the conclusion #126 was built to test: "the concept is findable, the discrimination is not" now holds against the supervised arm at equal input, no longer confounded with pixels. Also ran a same-env 384 control, which was not in the plan and earned its place. The published run was on an RTX 3070 on an older transformers; makelab2 is a major version on, and the transformers version is not in the detection signature -- a hazard this document already flags. So parity-vs-published would have differed in two things. The control lands within ONE detection out of 523 of the published run (215/308/95 vs 216/309/94), so the 4.x->5.15 jump is benign for this checkpoint and the whole delta is attributable to input size. That retires the "an upgrade could have changed every mask under an unchanged cache key" worry as a measurement. Two claims stated at 384 are corrected: y26_pano's richmond AP 0.536 no longer clears this arm at 0.649 (the YOLO sentence's conclusion holds, its arithmetic does not), and min_area_px=16 is inert at 384 but sits exactly at the smallest achievable blob at 1024, so the two rows do not share that setting's meaning. Parity detections are deliberately not published: --vistas-input-size does not change the arm's label, so exporting would overwrite the published richmond file. Publishing them needs a distinct name (the --publish-as pattern from #123). Measured cost, since the estimate going in was 3-4x and wrong in the cheap direction: the full 124-pano run is 3m38s on one A40, GPU forward 0.078 -> 0.092 s per view (1.17x). Verified before the run that the override reaches the model (pixel_values 384 -> 1024, mask logits 96x96 -> 256x256) rather than silently no-opping on a new major version. Docs only; no code, no detection artifacts, no roster change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The #35 gate script was hardcoded to build_detector("gemini", ...), so the parity Vistas arm could not be run through it at all. Generalized to a model spec (provider or provider:model_id), with the documented positional form preserved -- a bare non-provider token still reads as a Gemini model id, which is how three call sites in docs/model_comparison.md spell it. Verified against the committed paterson numbers: 36 of 124 misses recovered (29%), 88 found by no model, oracle-union 0.777, all reproduced exactly. Provider defaults now come from roster.PROVIDER_DEFAULTS via the same shim shape fp_taxonomy and null_recall use, so the reconstructed cache key cannot drift from the one compare.py wrote under -- a wrong default there does not crash, it silently misses every entry and reports zero detections. Also added the --vistas-* args to null_recall.py. build_detector reads them off the namespace with getattr, so without them the vistas arm there was silently the published 384x384 one even when the run being analysed was at parity -- a wrong-arm read that looks like a valid answer. And printed the counterweight to the oracle-union number: the FP bill a naive union actually pays, with the resulting precision/F1 against rampnet alone. The oracle ceiling assumes a combiner that keeps every right call, which does not exist; on paterson/gemini-pro the union genuinely wins (0.821 vs 0.805), which is the check that the line is not hardcoded to one verdict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

#126) The headline of a complementarity run is "the challenger recovers N of rampnet%s M misses". At any real box density some of that N is the match radius handing out hits for free, and the repo doctrine is to discount every recall-ceiling and union-oracle claim before believing it. null_recall.py already measures that, but over the WHOLE split. Applying its number to the miss subset assumes the coincidence rate is uniform across GT, and it need not be: rampnet%s misses are a biased sample (far-field, adjacent pairs) and those are exactly where box density differs. So this measures the null on the subset the claim is about, using null_recall%s own cyclic-shift construction -- pano A%s missed GT against pano B%s predictions, averaged over every non-identity shift, so box count and clustering stay real and only the pairing is wrong. On the paterson/gemini-pro regression case the committed numbers are unchanged (36 of 124, 88 found by no model) and the new line reads: null 0.081, so ~10 of the 36 are coincidence and ~26 attributable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…till loses (#126) Runs the #35 gate on the parity arm, plus the 384 arm as the control. Answers the recall-first question the parity result raised, and answers it against chance rather than at face value. Discounted for the null, a free zero-training model finds ~44 of the 72 ramps RampNet misses on richmond (61%). Raw is 54 of 72 (75%); the null on that subset is 0.143. The recall is real detection, not density -- at 6.2 boxes/pano the arm above-chances 0.864, higher than RampNet own 0.754, nothing like OWLv2 0.733 at 74 boxes/pano. The control is the interesting half. Parity moved F1 by +0.018 and was correctly judged not to change the ranking -- but it nearly TRIPLED the attributable complementary gain (~15 -> ~44 ramps) and shrank the found-by-nobody core from 51 to 18. A flat headline metric hid a large change in the structure underneath it, and only the complementarity read surfaced it. A naive union stays dead and it is not close: F1 0.555 against RampNet 0.855. The 54 ramps arrive with 442 false positives, ~8.2 FP per recovered ramp against the 9 FP RampNet pays for 238 TP. Recorded the oracle-union 0.942 as the ceiling it is, with the real union arithmetic beside it so the two cannot be confused. What is left open is a gated cascade -- this arm candidates as a spatial prior to locally relax RampNet threshold -- which is decidable but NOT run: #131 measured most silent misses as adjacent-tail or faint, but nobody has checked whether sub-threshold signal exists at these 54 locations specifically. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The complementarity result rules out a union but leaves a gated cascade -- the challenger candidates as a spatial prior to locally relax RampNet threshold. That has one precondition and it is measurable without building anything: RampNet must already produce sub-threshold response AT those ramps. If the heatmap is flat there the miss is genuine absence and no prior can raise it. Partitions every GT ramp into the four complementarity cells and reads the heatmap at each, with #46 Phase 1 instrument imported rather than reimplemented -- site_profile, null_percentile, nearest_peak, class_of and its ABSENT_MAX/PEAK_FLOOR cutoffs -- so the numbers are directly comparable to that phase 8% absent / 62% adjacent-tail / 30% faint decomposition, and a fix to the probe fixes both analyses. The three-way class is the point, not the raw activation. faint_local with the argmax at the site is signal a prior could raise. tail is an ADJACENT mode reaching in, so relaxing a threshold there fires on the neighbour rather than the ramp -- that argues for sigma/NMS work, not for a prior, even though it reads as "signal present" if you only look at act. Read pre-registered on #126 before running. Nulls computed only for the two RampNet-miss cells; the hit cells are high by construction and are the positive control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ons (#126) Self-inflicted, caught before the write-up. nearest_peak expects peak_local_max output down to the 0.05 floor. I fed it records[pid]["detections"], which on richmond are all >= 0.5519 -- the SHIPPED operating point. So "is there a peak near this missed ramp?" was really asking "is there a 0.55+ detection near it?", which for a missed ramp is close to definitionally no, and it made every miss look peakless with the nearest peak a median 104 px away. analysis_out/op_cache/richmond.json is committed and holds the floor peaks: 537 for richmond, 270 of them below 0.55. That is the input the sub-threshold question needs. The greedy match that DEFINED the miss set still uses the bundle records, because that is what produced the published 238/9/72 -- two different inputs to two different questions, now documented as such. Also replaced the class_of column in the printed table with peak_in_radius. class_of is #46 Phase 1 cutoffs, and there "tail" could only mean an outside mode reaching in because that population was SILENT misses by definition. This population is every RampNet miss, so act >= 0.05 has two causes the class cannot separate: a floor peak inside the radius (the model localized it and the detection was lost downstream -- recoverable WITHOUT a second model) versus unpeaked heatmap mass (nothing for a prior to promote). The old table also printed "100% tail" for the two hit cells, which is meaningless there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The committed bundle detections are the SHIPPED operating point -- on richmond every one scores >= 0.5519 -- while this document own recommendation since #54/#55 (PR #79) is 0.30. For a complementarity read those are different models, and the difference decides who gets credit for a recovery. Verified the op_cache is the same source before trusting it: filtered to >= 0.5519 it reproduces the published richmond row exactly (P 0.9636 R 0.7677 F1 0.8546, 238/9/72). At 0.30 the same peaks give P 0.9018 R 0.8290 F1 0.8639, 257/28/53 -- matching the committed corrected_at_0.3.csv. So RampNet at the recommended point already finds 19 of the ramps the shipped point misses. --rampnet-op-threshold re-sources RampNet from analysis_out/op_cache/<split>.json at a given threshold. Default is the bundle records, so the published roster numbers and the committed #35 gate results are unchanged -- paterson/gemini-pro still reproduces 36 of 124, 88 nobody finds, oracle-union 0.777. Also regenerates analysis_out/cascade_gate.json, which was first committed from a run whose nearest_peak input was the shipped detections rather than the floor peaks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… too (#126) same reasoning as the complementarity flag: the bundle detections are the shipped point and the recommendation is 0.30, and the cells move between them. peak_in_radius and the heatmap probe are threshold-independent, but which cell a ramp lands in is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…e gain was ours (#126) Two findings, one of which corrects my own headline from earlier in this PR. 1. THE OPERATING-POINT CORRECTION. The complementarity read scored RampNet from the committed bundle detections, which are the SHIPPED point (>=0.5519 on richmond), while this document has recommended 0.30 since #54/#55 (PR #79). Verified op_cache is the same source before re-basing: filtered at 0.5519 it reproduces the published row exactly (238/9/72). At 0.30 it gives 257/28/53, P 0.9018 R 0.8290 F1 0.8639, matching committed corrected_at_0.3.csv. Re-based, the challenger recovers 38 of 53 misses, ~30 after the null -- not 54 of 72 / ~44. About 14 of the ramps it got credit for are ramps RampNet already has at the point we recommend. The recovery RATE barely moves (75% -> 72%), which is the honest read: there are simply fewer misses to find. 2. THE CASCADE CEILING. Of the 38 genuinely-complementary ramps at 0.30: 19 floor peak in radius at 0.05-0.30 -> promotable, the real target 4 floor peak >=0.30 but unmatched -> greedy matcher gave it to an adjacent GT; a matcher/sigma problem (#130), not a threshold one 15 no floor peak at all -> nothing to promote; act 0.215 is unpeaked mass So the cascade is LIVE with a ceiling of ~19 ramps, +6.1 recall points (0.829 -> 0.890), before an unmeasured FP cost. A long way below the 54 the raw complementarity suggested. Also a negative worth keeping: RampNet activation does NOT discriminate. challenger_only sits at null percentile 0.88 and the hard core at 0.925 -- the ramps nobody finds look stronger on raw heatmap mass. There is no self-gating shortcut that skips the second model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This work branched at 5e20d11, before eccadda (wrap the 360 seam in the matcher) and f4c71c8 (peaks_to_dets dropped peaks beside the seam) landed on main. Rather than argue that away, measured it. The greedy match used here does not wrap. score_pano own docstring records that wrapping moves no metric on any committed split while #130 duplicate ground truth masks it, so aggregates are safe -- but a cell assignment is finer-grained than an aggregate. And analysis_out/op_cache/richmond.json is UNCHANGED by those commits, so it was never regenerated after the peaks_to_dets fix and can still be missing peaks beside the seam; that can only UNDERSTATE the promotable count. Measured: 6 of richmond 310 GT ramps straddle the seam, and only ONE is in challenger_only. The other five are in both, where neither fix can move the partition in a direction that matters. Worst case for the headline is one ramp in 38 and no conclusion turns on it. Re-running on a post-#140 main retires the caveat instead of bounding it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

jonfroehlich · 2026-09-03T13:17:26Z

Deep review

Re-ran everything from the branch head (54a98c6) in a clean detached worktree, against the committed inputs only. The headline result stands and the arithmetic in the doc is right almost everywhere it can be checked. What reproduced:

Full suite: 1238 passed, 1 skipped, as claimed.
RampNet's committed richmond row from records.jsonl + verdicts.json: 238/9/72, P 0.9636 / R 0.7677 / F1 0.8546, AP 0.7635. analysis_out/op_cache/richmond.json filtered at ≥ 0.5519 reproduces it exactly; at ≥ 0.30 it gives 257/28/53, 0.9018 / 0.8290 / 0.8639, matching the doc and corrected_at_0.3.csv.
The committed 384 detections (benchmark/model_detections/mask2former-vistas-curb-cut__richmond.json) reproduce the "published" row 216/309/94, 0.411 / 0.697 / 0.517, AP 0.513 and the ≥ 0.30 row 0.419 / 0.697 / 0.524.
complementarity.py gemini-3.1-pro-preview paterson (legacy positional form) reproduces the committed paterson numbers exactly: 36 of 124, 88 found by no model, oracle-union 0.777, naive union 0.821 vs 0.805.
cascade_gate_op030.json: 38 challenger_only sites split 19 / 4 / 15 as the table says; null percentile 0.88 vs 0.925; 6 of 15 neither sites carry a ≥ 0.30 peak; 6 seam sites, 5 in both and 1 in challenger_only. All match.
Every number in the union tables re-derives from the cell counts (0.393 / 0.942 / 0.555; 0.549 at 0.30; 5.8%; 8.2 FP per ramp; 3.6 vs 0.07 FP/pano; +6.1 recall points).
Both pre-registrations are on Mapillary Vistas as a second supervision source: off-the-shelf baseline, and a path to non-ramp label types #126 with timestamps before the corresponding results.
git merge-tree against origin/main, Actually fix the two collapsed runbook snippets from #129, and guard the class #136, claude-opus-5 on nine splits: the annapolis displacement does not generalise (#139) #146 and Record time as well as money for every non-free run (#143) #147: no conflicts on any of the four, despite the shared docs/model_comparison.md.
Every symbol cascade_gate.py imports from silent_activation, farfield_forensics, operating_point_curve, miss_gallery and threshold_sweep exists on this branch and on origin/main, so the merge does not break its imports.
No licensing or terms-of-service text in the diff. The new shell block carries real line continuations (Actually fix the two collapsed runbook snippets from #129, and guard the class #136's class).

Everything below is about what a clean clone cannot re-derive, one column that quietly comes from a different run than it says, and a docstring quote that lost its second half.

1. High — the parity run and everything downstream of it rest on detections that exist on one machine

The parity detections (1024×1024) are, per the doc, "deliberately not published", and the same-env 384 control is not published either. Both live in a private --cache-dir on makelab2. That single sentence is true, but its consequence is not stated where the replication rule requires it — beside every number it affects. Those are:

the parity row (274/442/36, AP 0.649) and the control row (215/308/95) in the first table;
the entire Complementarity table at parity (54 / 18 / 0.143 / ~44 / 0.942 / 6.2 · 0.864);
the entire operating-point correction table (38 / ~30 / 15 / 0.952 / 0.549);
both committed cascade_gate*.json — the four-cell partition inside them was computed against those detections, so the artifacts are committed but cannot be regenerated;
the seam-exposure count (1 in 38).

The only side of any of those a clean clone can rebuild is RampNet's. The doc's own "Reproducing it" block gives three compare.py commands that re-run the model, which is a different thing from re-deriving the committed numbers from committed inputs. The mechanism to close this exists on main (this branch is 23 commits behind it): export_model_cache.py --publish-as from #123 gives the parity arm a distinct published name so it does not overwrite mask2former-vistas-curb-cut__richmond.json.

Fix: a one-line statement next to each of the five items above: "the 1024 and same-env 384 detections are not published; these numbers are re-derivable only from the makelab2 cache. Publishing them needs --publish-as (#123) after this branch takes main." Publishing is a decision for Jon (below).

2. Medium — the "vistas @384" column is the unpublished control run, not the published arm, and it is not labelled

Re-running complementarity.py vistas:curb-cut richmond against the published 384 detections gives:

	doc says	published 384 detections give
challenger ONLY	21 (29%)	22 (31%)
found by NEITHER	51	50
oracle-union recall	0.835	0.839
naive union F1	0.585	0.586
above chance (`null_recall.py`)	0.657	0.661
attributable after the null	~15	~15

The doc's column matches the same-env control (215/308/95), which differs from the published run by one detection. The text says "on the same cached detections" without saying which of the two 384 runs that is. Nothing turns on one ramp, but as written the column is not reproducible from committed data and the reader has no way to know why.

Fix: either replace the column with the published-arm values above (then it reproduces from a clean clone with the command in the doc), or label it "same-env control, unpublished". Replacing is better. The rate becomes 22 (31%).

3. Medium — the seam paragraph quotes half of `score_pano`'s docstring, and the half it drops is the one that applies

The doc cites score_pano's docstring as saying wrapping "moves no metric on any committed split". The sentence on main is: "Wrapping moves no RampNet or YOLO metric on any committed split — but it does move the challengers." The challenger is the side being partitioned here. The paragraph's conclusion may still be right, but the evidence offered for it is a misquote.

Measured on this branch with a wrapping copy of matched_gt: 0 cell flips on RampNet's side at ≥ 0.5519, ≥ 0.30 and ≥ 0.05, and 0 flips on the published 384 arm. The parity arm cannot be checked (finding 1), so the doc's own count — 1 seam site in 38 — remains the bound, and it holds.

One more consequence for the merge. On main, score_pano wraps by default, so after this branch takes main, the r_fp / c_fp and union P/R/F1 in complementarity.py will come from a wrapping matcher while the four cells come from the non-wrapping matched_gt in the same script. Two matchers in one output.

Fix: quote the docstring in full and record the measured zero-flip result for RampNet and the 384 arm. Merge origin/main into the branch (verified clean), then route matched_gt through rampnet.geometry.dist_sq(..., wrap_x=True) so the cells and the FP counts use one matcher, and state beside the committed JSONs that they were computed before the wrap with the 1-in-38 bound.

4. Medium-low — `act` 0.215 is the median of the wrong subset

The cascade table's third row, "no floor peak in radius | 15 | act is 0.215 median": 0.2152 is the median act of the whole challenger_only cell (n = 38, cells[].act_median in the JSON). The 15 no-peak sites have median act 0.272 (mean 0.369). The 19 promotable sites have median 0.153. The point the row makes — unpeaked mass with nothing to promote — is unaffected, but the number attached to it is the cell's, not the row's.

Fix: 0.215 → 0.272.

5. Medium-low — three different numbers for how many recoveries were RampNet's own: 19, 16, ~14

At 0.55 → 0.30 RampNet gains 19 hits (72 → 53 misses). Of those 19, 16 came out of challenger_only (54 → 38) and 3 out of neither (18 → 15). So:

complementarity.py docstring: "19 of the 'misses' a challenger gets credit for recovering are ramps RampNet already has" — 19 is the miss reduction; the challenger's credited recoveries fall by 16.
doc: "~14 of the ramps the challenger got credit for recovering are ramps RampNet already has" — ~14 is the null-discounted difference (~44 → ~30); the raw count is 16.
cascade_gate.py help text: "19 of the shipped point's misses are ramps RampNet already has" — correct as phrased.

Fix: docstring → 16 (of the 19); doc → "16 of the 54, ~14 after the null".

6. Medium-low — none of the new code has a test

complementarity.py (spec parsing, compare_args, complementary_null, the union arithmetic), cascade_gate.py (cell_of, summarize) and the new null_recall.py arguments add zero tests; the suite count is unchanged at 1238. One concrete hole: model_spec("foo:bar") — an unknown provider with a colon — is silently read as the Gemini model id "foo:bar", and the script then reports zero cached detections rather than a bad spec.

Fix: tests for model_spec (provider, provider:id, legacy bare Gemini id, and unknown-provider-with-colon should raise), compare_args producing a vistas signature equal to compare.py's with and without --vistas-input-size, complementary_null on a synthetic three-pano case, cell_of/summarize on synthetic rows, and the 384 regression (194 / 44 / 22 / 50) from the committed model_detections file written into a temporary DetectionCache.

7. Low — `analysis_out/cascade_gate.json` predates its own schema

It was last written at b7342dc; 4c192ca then added rampnet_op_threshold to the payload, and cascade_gate_op030.json (written at 730baff) carries it while the shipped-point file does not. A regeneration would change bytes ("rampnet_op_threshold": null) even with identical results — which is exactly the comparison newline="" was added to protect. It cannot be regenerated without the parity cache (finding 1).

Fix: state it beside the artifact list in the doc; regenerate when the parity detections are published.

8. Low — the run instructions for the complementarity and cascade sections are not in the doc, and the README has no row for the new script

"Reproducing it" lists three compare.py commands and nothing else. The exact invocations for the complementarity table, the 0.30 re-base, null_recall.py at parity and both cascade_gate.py runs exist only in docstrings and the PR body. scripts/analysis/README.md gained a complementarity.py row but no cascade_gate.py row; every other script in the directory has one.

Fix: add the five commands to the block; add the README row.

9. Low — cost is wall-clock for one of three runs

3m38s is given for the parity run; the same-env control and the RampNet re-run alongside are not timed; there is no GPU-hour or dollar figure. Three runs on one lab A40 is about 0.2 GPU-hours and $0, and the rule is to write both units down.

Fix: one line: three runs, total wall-clock, ~0.2 GPU-h, $0 (lab hardware).

10. Low — two silent paths in `cascade_gate.py`

Under --rampnet-op-threshold, a pano missing from op_cache is scored as RampNet-blank (floor_peaks.get(pid, [])), so every GT on it becomes a miss with no warning. All 124 richmond panos are present, so nothing here is affected; another split would not be told.
preds = floor_peaks.get(pid) or [bundle records] substitutes the bundle per-pano whenever op_cache lists a pano with zero floor peaks, while the header still prints floor peaks from: op_cache. Harmless (the bundle is a subset of the floor) but the two-input story in the docstring is no longer exactly true.

Fix: warn and count panos absent from op_cache; drop the or fallback when floor_src is op_cache.

11. Low — register

"a matching bug wearing a threshold costume", "wearing a different cache key", "dead" (three times), "the economics are the whole story", "earns its place", "the honest way to read it". These are published under Jon's name; each has a plain statement behind it.

Decisions for Jon, not fixes

Publish the parity detections. --publish-as (e.g. mask2former-vistas-curb-cut-1024) once the branch takes main, assuming the private cache directory on makelab2 still exists. Without it, finding 1 stays open and the three tables and two artifacts remain results that live on one host.
Merge main into the branch before it lands, so matched_gt can use rampnet.geometry and the two matchers in complementarity.py agree. Verified conflict-free; measured zero effect on RampNet's and the 384 arm's cells.

What holds up

The decision rule was read fixed on Mapillary Vistas as a second supervision source: off-the-shelf baseline, and a path to non-ramp label types #126 before the result, and the result is not close: +0.018 F1 against a 0.05 bar.
The env control is the right design and it did its job — one detection in 523 between transformers 4.x and 5.15.
The mechanism read is correct and the doc says it plainly: resolution bought recall and AP, not precision.
The null measured on the miss subset rather than extrapolated (0.143 vs 0.145) is the right instrument and the doc reports that it landed where the extrapolation would have.
cascade_gate.py asks the one question that decides whether a cascade is possible, uses Qualitative failure taxonomy: bucket each model's FP/FN from cached detections #46 Phase 1's instrument by import rather than copy, and the peak_in_radius column — not the class — is correctly identified as the one that decides it.
The op-0.30 re-base is the most important correction in the PR and it is applied next to the original number rather than over it.
The mid-run instrument bug (feeding ≥ 0.5519 detections to nearest_peak) was caught, recorded, and the artifacts regenerated.
compare_args reads provider defaults from roster.PROVIDER_DEFAULTS; every vistas_* key it needs is present, and null_recall.py's new arguments follow the deviation-only convention exactly.
The two JSONs are whitelisted correctly (git check-ignore returns nothing) and written LF-pinned.

Fix list

[F1] High — Next to the parity/control table, the complementarity table, the op-0.30 table, the cascade tables and the artifact list, state that the 1024 and same-env 384 detections are not published and these numbers re-derive only from the makelab2 cache; name export_model_cache.py --publish-as (Wire Claude into the model-comparison harness (#122): Opus 5 at low effort takes the top challenger slot on annapolis #123, on main) as the way to close it — files: docs/model_comparison.md
[F2] Medium — Replace the "vistas @384" column with the published-arm values (194 / 44 / 22 (31%) / 50, oracle-union 0.839, naive-union F1 0.586, above-chance 0.661) and add one sentence that the same-env control differs by one detection — files: docs/model_comparison.md
[F3] Medium — Merge origin/main into the branch (clean); quote score_pano's docstring in full; route matched_gt through rampnet.geometry.dist_sq(wrap_x=True) so cells and FP counts use one matcher; record the measured zero-flip result for RampNet (all three thresholds) and the 384 arm, and note the committed JSONs predate the wrap with the 1-in-38 bound; re-run the suite after the merge — files: scripts/analysis/complementarity.py, scripts/analysis/cascade_gate.py, docs/model_comparison.md
[F4] Medium-low — Cascade table row 3: act 0.215 → 0.272 (median of the 15 no-peak sites; 0.215 is the whole cell) — files: docs/model_comparison.md
[F5] Medium-low — "19 of the misses a challenger gets credit for recovering" → 16 of the 54 (3 of the 19 were in neither); doc "~14" → "16, ~14 after the null" — files: scripts/analysis/complementarity.py, docs/model_comparison.md
[F6] Medium-low — Add tests: model_spec (four cases, unknown-provider-with-colon raises), compare_args vistas signature with/without input size equals compare.py's, complementary_null synthetic, cell_of/summarize synthetic, and the 384 regression 194/44/22/50 from the committed model_detections file via a temp DetectionCache — files: tests/test_complementarity.py (new), tests/test_cascade_gate.py (new), scripts/analysis/complementarity.py
[F7] Low — State beside the artifact list that cascade_gate.json was written before rampnet_op_threshold entered the payload and will differ by that key when regenerated — files: docs/model_comparison.md
[F8] Low — Add the exact complementarity.py (parity, and --rampnet-op-threshold 0.30), null_recall.py --vistas-input-size 1024 1024 and both cascade_gate.py commands to "Reproducing it"; add a cascade_gate.py row to the README table — files: docs/model_comparison.md, scripts/analysis/README.md
[F9] Low — Record cost in both units: three runs on one A40, total wall-clock, ~0.2 GPU-h, $0 — files: docs/model_comparison.md
[F10] Low — cascade_gate.py: warn and count panos absent from op_cache under --rampnet-op-threshold; drop the per-pano or bundle fallback when floor peaks come from op_cache — files: scripts/analysis/cascade_gate.py
[F11] Low — Replace "wearing a threshold costume", "wearing a different cache key", "dead", "the economics are the whole story", "earns its place", "honest way to read it" with plain statements — files: docs/model_comparison.md, scripts/analysis/cascade_gate.py

🤖 Generated with Claude Code (claude-fable-5-1)

Conflict-free. main's score_pano now wraps x by default, so the FP counts and union arithmetic in complementarity.py come from a wrapping matcher; its own matched_gt is routed through rampnet.geometry in the next commit so the four cells and the FP counts agree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

) F3 from the #137 review. matched_gt re-derived the pano distance inline and did not wrap, while score_pano -- which supplies the r_fp / c_fp and the union P/R/F1 printed in the same table -- wraps by default since #132. It now calls rampnet.metrics.greedy_match with wrap_x=True, the same core score_pano uses. Measured effect, cells (both / rampnet-only / challenger-only / neither): richmond vistas 384 rampnet bundle 194/44/22/50 identical wrapped richmond vistas 384 rampnet@0.30 202/55/14/39 identical wrapped richmond vistas 384 rampnet@0.05 213/66/3/28 identical wrapped paterson gemini-pro rampnet bundle 188/83/36/88 identical wrapped paterson gemini-pro rampnet@0.30 194/90/30/81 identical wrapped paterson gemini-pro rampnet@0.05 201/98/23/73 identical wrapped Zero flips, so the committed #35 gate numbers and this PR's 384 column are unchanged. The parity 1024 arm cannot be re-checked -- those detections are not published -- so its bound stays the doc's own count of 1 seam site in 38. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ent paths (#126) F5, F6 and F10 from the #137 review. F6 -- tests/test_complementarity.py (26) and tests/test_cascade_gate.py (18). Both read only committed files: benchmark/richmond, benchmark/model_detections and the two analysis_out/cascade_gate JSONs. No GPU, no .model_cache, no network. They cover matched_gt (including the seam case, so the inline non-wrapping distance cannot come back), model_spec, compare_args against the signature recorded in the published export, complementary_null on cases with an arithmetic answer, cell_of, summarize, and the published 384 richmond column cell for cell (194/44/22/50). model_spec("foo:bar") -- an unknown provider WITH a colon -- was read as the Gemini model id "foo:bar". That builds a detector whose signature nothing ever cached, so the run reported a model with zero detections instead of a mistyped argument. A colon now means provider:model_id and an unrecognised provider there is rejected; a bare token still means a Gemini model id, which is the #35 gate's committed form. F10 -- cascade_gate.py: panos absent from analysis_out/op_cache/<split>.json under --rampnet-op-threshold scored as RampNet-blank, turning every GT ramp on them into a miss with no warning. They are now counted and reported. And the per-pano `floor_peaks.get(pid) or <bundle records>` fallback substituted the shipped detections whenever the op_cache listed a pano with zero floor peaks, while the header still said the floor came from op_cache; the source is now one per run. All 124 richmond panos are present with peaks, so neither changes a committed number. F5 -- the 19 misses RampNet gains going 0.55 -> 0.30 are 16 out of challenger_only and 3 out of neither, so the complementary-gain headline falls by 16, not 19. Corrected in complementarity.py's docstring and cascade_gate.py's --rampnet-op-threshold help. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@384

…clone cannot rebuild (#126) F1, F2, F4, F7, F8, F9, F11 and the documentation half of F3 from the #137 review. F2 -- the "vistas @384" complementarity column was the unpublished same-env A40 control, not the published arm, and it was not labelled either way. Replaced with the published arm's own numbers, re-run here against benchmark/model_detections/mask2former-vistas-curb-cut__richmond.json: challenger-only 22 (31%) not 21 (29%), found-by-nobody 50 not 51, oracle-union 0.839 not 0.835, above-chance 0.661 not 0.657, naive-union F1 0.586 not 0.585. The control differs by one detection in 523; that is now stated rather than left as the unmarked source. F1 -- neither A40 run's detections are published, so five things re-derive only from the private makelab2 cache: the parity and control rows, the 1024 complementarity column, the whole operating-point-correction table, both cascade tables and both cascade_gate JSONs, and the seam-exposure count. Each now says so where it appears, and export_model_cache.py --publish-as (#123, on main) is named as what closes it. F3 (doc half) -- score_pano's docstring was quoted as "moves no metric on any committed split"; the full sentence is "Wrapping moves no RampNet or YOLO metric on any committed split -- but it does move the challengers", and the challenger is the side being partitioned. Quoted in full, with the measurement that replaces the citation: zero cell flips on the published 384 arm and on the paterson #35 gate, at all three RampNet thresholds. The parity arm cannot be re-checked, so its bound stays 1 seam ramp in 38. F4 -- the no-peak row's activation median is 0.272 over those 15 sites (0.369 mean); 0.215 was the whole 38-ramp cell's, and the 19 promotable sites are at 0.153. F5 (doc half) -- "~14 of the ramps the challenger got credit for recovering" is 16 raw and ~14 after the null; 16 of the 19 hits RampNet gains at 0.30 come out of challenger_only and 3 out of neither. F7 -- cascade_gate.json predates the rampnet_op_threshold key, so a regeneration changes its bytes with identical results. Stated beside the artifact list. F8 -- the five missing invocations (complementarity at parity and at 0.30, null_recall at parity, both cascade_gate runs) added to "Reproducing it", with the step a clean clone needs first: complementarity.py and null_recall.py read .model_cache rather than the published export, so the export has to be written into a cache directory. cascade_gate.py now has a row in scripts/analysis/README.md. F9 -- cost in both units: $0 on lab-owned hardware, and 3m38s for the one leg of three that was timed. The 384 control was not timed and RampNet's row uses no GPU, so there is no session total to quote. F11 -- plain statements in place of six phrases. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

jonfroehlich · 2026-09-03T13:58:59Z

Review fixes

All eleven items applied. Four commits: 1b634e2 (merge main), a3ebe37 (the matcher),
7e8c9d0 (tests + the two silent paths), 70ce265 (the write-up).

Merge with main and the post-merge suite. git merge origin/main was conflict-free as the
review found, and the suite went 1238 passed / 1 skipped → 1322 passed / 1 skipped on the
merge alone (main brought 84 tests, mostly the #132 seam work and the scoreboard).

The zero-flip measurement. Cells are both / rampnet-only / challenger-only / neither, wrapping
against not wrapping, on the committed detections:

arm	RampNet's side	cells	wrapped
vistas 384 (published)	bundle, ≥ 0.5519	194 / 44 / 22 / 50	identical
vistas 384 (published)	op_cache ≥ 0.30	202 / 55 / 14 / 39	identical
vistas 384 (published)	op_cache ≥ 0.05	213 / 66 / 3 / 28	identical
gemini-3.1-pro-preview, paterson	bundle	188 / 83 / 36 / 88	identical
gemini-3.1-pro-preview, paterson	op_cache ≥ 0.30	194 / 90 / 30 / 81	identical
gemini-3.1-pro-preview, paterson	op_cache ≥ 0.05	201 / 98 / 23 / 73	identical

Zero cell flips everywhere, so the published 384 column and the committed #35 gate numbers are
unchanged by the wrap. The paterson gate reproduces exactly as the review reported it: 36 of 124,
88 found by no model, oracle-union 0.777, naive union 0.821 vs 0.805. The parity arm cannot be
re-checked — its detections are not published — so its bound stays the doc's own 1 seam ramp in 38.

Per item

F1 — 70ce265. Statements next to all five: the parity/control rows, the 1024
complementarity column, the operating-point table, the cascade artifacts, and the seam count.
export_model_cache.py --publish-as (Wire Claude into the model-comparison harness (#122): Opus 5 at low effort takes the top challenger slot on annapolis #123, now on this branch via the merge) named as the fix.
F2 — 70ce265. Column replaced with the published arm, re-run here rather than copied:
challenger-only 22 (31%), found-by-nobody 50, oracle-union 0.839, naive-union F1
0.586, above-chance 0.661 (null_recall.py gives 4.5 boxes/pano · 0.661). Every value
matched the review's. found by BOTH 194 and rampnet ONLY 44 were already right — the two
arms agree on those cells — and attributable after the null stays ~15 (null 0.090).
F3 — merge 1b634e2, matcher a3ebe37, docstring and measurement 70ce265. matched_gt
now calls rampnet.metrics.greedy_match(..., wrap_x=True), the same core score_pano uses, so
the four cells and the FP counts printed in one table come from one matcher.
F4 — 70ce265. 0.215 → 0.272 (0.369 mean). Checked against
cascade_gate_op030.json: 0.2152 is cells[].act_median over the 38-ramp cell, the 15 no-peak
rows are at 0.2723, the 19 promotable rows at 0.1525. All three are now named.
F5 — 7e8c9d0 (docstring, --rampnet-op-threshold help) and 70ce265 (doc). 19 is the
miss reduction; 16 of the 19 come out of challenger_only and 3 out of neither, so the
headline falls by 16, and ~14 is the null-discounted figure. Verified from the two JSONs.
F6 — 7e8c9d0. tests/test_complementarity.py (21) and tests/test_cascade_gate.py (18).
Committed files only — benchmark/richmond, benchmark/model_detections, the two
cascade_gate JSONs — no .model_cache, no network, no GPU. Covers model_spec (including
unknown-provider-with-colon, which now exits rather than being read as a Gemini model id),
compare_args against the signature recorded inside the published export with and without
--vistas-input-size, complementary_null, cell_of, summarize, the seam case on
matched_gt, and the 384 regression 194/44/22/50 through a tmp_path DetectionCache. Also
summarize(sites) == cells on both artifacts, so a hand-copied cascade number now fails CI.
F7 — 70ce265. Stated beside the artifact list.
F8 — 70ce265. Five commands added; cascade_gate.py row added to
scripts/analysis/README.md. One thing found while writing it: complementarity.py and
null_recall.py read .model_cache, not the published export the way fp_taxonomy.py and
silent_witness.py do, so a clean clone has to materialise the export into a cache directory
first. That step is now written down, and tests/test_complementarity.py is the working example.
F9 — 70ce265. $0 (lab-owned hardware) and 3m38s for the one leg of three that was timed.
No session total is quoted: the 384 control was untimed and RampNet's row loads no model.
F10 — 7e8c9d0. Panos absent from op_cache under --rampnet-op-threshold are counted and
warned about; the per-pano or <bundle records> fallback is gone when the floor came from
op_cache. All 124 richmond panos are present with peaks, so no committed number moves. No test
for the warning path — reaching it in main() requires the model load, so it would need a GPU
or a refactor; cell_of/summarize are tested directly instead.
F11 — 70ce265. Six phrases replaced with plain statements.

Nothing skipped. Nothing published: the parity detections stay unpublished, which is the decision
left for you.

Final suite: 1361 passed, 1 skipped.

🤖 Generated with Claude Code (claude-opus-5)

jonfroehlich mentioned this pull request Aug 18, 2026

Mapillary Vistas as a second supervision source: off-the-shelf baseline, and a path to non-ramp label types #126

Open

jonfroehlich and others added 9 commits August 18, 2026 15:48

jonfroehlich mentioned this pull request Sep 3, 2026

Actually fix the two collapsed runbook snippets from #129, and guard the class #136

Open

This was referenced Sep 3, 2026

The Laurens deficit is the rig, not the town: add the GSV arm and score all 11 challengers on both (#151) #153

Merged

The RampNet-vs-YOLO gap at matched operating points is 0.039, not 0.252 — plus the label rescue and the equirect control (#51) #154

Merged

jonfroehlich and others added 3 commits September 3, 2026 06:26

jonfroehlich mentioned this pull request Sep 6, 2026

Make RampNet rig-native for open imagery: retrain on Mapillary/Panoramax hard positives mined from multi-view consensus, precision check first #158

Open

Navigation Menu

Uh oh!

Vistas at resolution parity: the handicap was real, and it was recall (#126) - #137

Vistas at resolution parity: the handicap was real, and it was recall (#126)#137
jonfroehlich wants to merge 14 commits into
mainfrom
exp/vistas-parity-126

jonfroehlich commented Aug 18, 2026 •

edited

Loading

Uh oh!

jonfroehlich commented Sep 3, 2026

Uh oh!

jonfroehlich commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jonfroehlich commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-registered read

It stands, and not narrowly

The mechanism is the finding, and it is not what the caveat predicted

An env control that wasn't in the plan, and earned its place

Two claims at 384 that parity corrects

Deliberate omissions

Verification

Follow-on in this PR: complementarity (the #35 gate)

Tooling changes

Also in this PR: the operating-point correction, and the cascade gate

The complementarity numbers above are against a RampNet we do not recommend

cascade_gate.py (new): the cascade is live, ceiling ~19 ramps

An instrument bug I caught mid-run

Uh oh!

jonfroehlich commented Sep 3, 2026

Deep review

1. High — the parity run and everything downstream of it rest on detections that exist on one machine

2. Medium — the "vistas @384" column is the unpublished control run, not the published arm, and it is not labelled

3. Medium — the seam paragraph quotes half of score_pano's docstring, and the half it drops is the one that applies

4. Medium-low — act 0.215 is the median of the wrong subset

5. Medium-low — three different numbers for how many recoveries were RampNet's own: 19, 16, ~14

6. Medium-low — none of the new code has a test

7. Low — analysis_out/cascade_gate.json predates its own schema

8. Low — the run instructions for the complementarity and cascade sections are not in the doc, and the README has no row for the new script

9. Low — cost is wall-clock for one of three runs

10. Low — two silent paths in cascade_gate.py

11. Low — register

Decisions for Jon, not fixes

What holds up

Fix list

Uh oh!

jonfroehlich commented Sep 3, 2026

Review fixes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jonfroehlich commented Aug 18, 2026 •

edited

Loading

`cascade_gate.py` (new): the cascade is live, ceiling ~19 ramps

3. Medium — the seam paragraph quotes half of `score_pano`'s docstring, and the half it drops is the one that applies

4. Medium-low — `act` 0.215 is the median of the wrong subset

7. Low — `analysis_out/cascade_gate.json` predates its own schema

10. Low — two silent paths in `cascade_gate.py`