feat: noise-floor control and floor-adjusted flip rate

[duckyquang]

Summary

benchmaxxing run --noise-floor adds the decoding-noise control to the solo lane. It re-asks each clean payload once and counts how often the answer moves with nothing changed, which is the rate a flip rate has to be read against.

• reuses the clean answer from the main pass as the first ask, so it costs one extra call per twin, not two
• reports the floor with its own CI and the flip rate above the floor as a paired difference (per twin, so the interval respects the pairing)
• off by default, and whether it ran is stamped in the run manifest and shown in --dry-run

Also folded the pilot stage's records into records_by_model so its estimates carry the model name like the solo lane does, instead of the floor needing a special case.

Result

Deterministic mock backend: floor 0.0000 [0.0000, 0.0000] per model, and the floor-adjusted flip rate equals the raw 0.3333, which is the invariant you want. A scripted flaky backend (answers the same prompt differently every other call) gives a non-zero floor and the adjusted rate drops by exactly that much, pinned in a test.

--dry-run shows the extra cost up front: 108 estimated calls with the control on versus 72 without, on 12 twins across 3 models.

ruff check . clean, pytest -q 613 passed.

Notes

Cascade is unaffected: the control is a solo-lane idea, so --noise-floor on the cascade stage is a no-op and says so in the plan.

Stacked: #251 -> #252 -> #253 -> #257 -> this. Base is #257's branch.

Closes #256

[sebasmos]

Approving for now. The real win is running it on real MedQA/NIH and posting an actual noise-floor number. That is the experiment.

[maximinl]

Approve the noise-floor control (stacked on #257).

Reusing the clean answer as the first ask (one extra call/twin), paired floor-adjusted flip rate, and off-by-default with dry-run cost visible are all correct. Cascade no-op is the right scope. Real MedQA/NIH number still needed later; code is fine. Merge after the stack below it.

[sebasmos]

The noise-floor is a genuine control, not just infra, so it is worth landing, but only with a real number: mock cannot tell us the decoding-noise floor. It needs a real MedQA/NIH run through the API before the flip rates it adjusts mean anything. Requesting changes until that run is attached.

Superseded by the later Changes-requested review on this PR; dismissing the stale approve to avoid a mixed signal.

[Agastya191]

Good control, and the parts that are easy to get wrong are right: the floor re-ask and record.clean_answer both go through SoloAgent.run/_parse_option, so the floor is measured on the same parsed-option scale as flipped (that's why a deterministic backend pins the floor at 0), flip_rate_above_floor subtracts two commensurable per-twin quantities, reusing the clean answer as the first ask keeps it to one extra call per twin, and the dry-run accounting (108 vs 72) matches. One thing I'd tighten is the alignment in noise_floor_pass: it pairs each re-ask to a record by position, zipping a freshly rebuilt _twins(selected, cue_types) against the records that run_solo_baselines produced from its own separate _build_twins(selected, cue_types), guarded only by a length check. The two builders are identical today so it lines up, but the length check catches a count mismatch and not a reorder, so if _twins and _build_twins ever drift (a limit, a filter, a changed skip, a reordering) the floor would silently subtract the wrong twin's noise from each flip and the headline floor-adjusted rate would be quietly wrong with nothing failing. Since the records already carry case_id and cue_type, I'd key the twins by (case_id, cue_type) and look each record up rather than depend on the two twin builders staying byte-for-byte in lockstep.

[duckyquang]

@Agastya191 good instinct to tighten that, fixed in bea24ae. noise_floor_pass now keys the twins by (case_id, cue_type) and looks each record up, instead of zipping two separately-built twin lists and trusting a length check. The records already carry both fields, so a reorder can't silently pair a flip with the wrong twin's clean payload anymore; a missing key raises. per_twin stays aligned with records, so the paired floor-adjusted resampling is unchanged. Keyed on str() both sides too, so a record's stringified id can't miss a twin's raw one.

@sebasmos agreed a mock can't tell us the decoding-noise floor. It's in the MedQA/NIH run set I'm putting through the API, so the floor-adjusted rates get a real number behind them before this lands.

@maximinl thanks. One tightening since your approve: the floor re-ask is now matched to its record by (case_id, cue_type) rather than by position (bea24ae), so the two twin builders drifting can't quietly subtract the wrong twin's noise.

[sebasmos]

Approving. The twin re-ask is now keyed by (case_id, cue_type) and raises on a miss, and the new test feeds a reversed twin list, which the old positional zip would have failed with a spurious floor. One thing was fixed for you along the way: noise_floor and flip_rate_above_floor were the only solo-lane estimates still resampled per twin after #252 moved the rest to case clusters, so the floor-adjusted rate would have shipped a tighter CI than the raw rate it corrects. They now pass the same clusters, guarded by the existing alignment check. The real MedQA and NIH floor numbers are still the open empirical item, a mock cannot produce a decoding-noise floor. Merged main in, post-squash conflicts resolved to the branch side.