feat: bootstrap CIs and effect sizes in the run summary

[duckyquang]

Summary

Every headline number in a run summary now ships with a 95% percentile bootstrap CI and the paired effect size for that arm. It's a thin call over the existing stats.bootstrap_ci, no new stats.

• solo lane: flip rate, clean accuracy, contaminated accuracy, and shortcut reliance (clean minus contaminated), resampled over twins
• cascade: shared adoption, isolated adoption, and the adoption delta, resampled over cases
• both summary.md (new "Estimates with uncertainty" table) and results.json (new estimates block) carry point, ci_low, ci_high, n, the CI level and the resample count
• the bootstrap is seeded from the run's config seed, so the same records reproduce the same interval

Two decisions worth a look:

• The paired entries resample the per-observation difference rather than the two rates separately, so the interval respects the pairing instead of throwing it away.
• The solo lane reports one row per model instead of pooling. Every model answers the same twins, so a pooled interval would treat correlated observations as independent and quietly shrink. Pooled point estimates stay in the headline table as before.

I also added records to what run_pilot returns. It only had the aggregate rate, so there was no sample to resample over. Additive, nothing else changed.

Result

On the 4-case offline manifest with 3 mock models, solo: flip rate 0.3333 [0.0833, 0.5833] n=12 per model, shortcut reliance 0.3333 [0.0833, 0.5833]. Cascade: adoption delta 1.0000 [1.0000, 1.0000] n=4.

ruff check . clean, pytest -q 599 passed. The new test pins the interval for a fixed 12-record set with a fixed seed, so a change in the resampling shows up as a failing test rather than as drift.

Notes

The issue suggested the effect size be the flip rate against the noise floor. The runner has no noise-floor arm (that control is a separate re-ask of the clean prompt, and #93 measured it at 0.000), so I used the paired clean-minus-contaminated accuracy drop instead, which is the effect the solo arm is actually about. Happy to add a --noise-floor comparison once the runner grows that control.

This one is stacked on #101 (PR #251), which adds the run entry point and the summary writer this builds on. Base is set to that branch, so the diff here is only the CI work. @sebasmos it also overlaps in spirit with #237, worth checking we're not doing the same thing twice.

Closes #127

[sebasmos]

Approving, thin and correct. Same note: next effort on real-data runs, not more reporting infra.

[maximinl]

Approve the CI layer (stacked on #251).

Paired resampling for shortcut_reliance / adoption_delta and per-model (not pooled) solo intervals are the right uncertainty choices — pooling would quietly shrink CIs on correlated twins. Seeded bootstrap_ci + pinned tests make the intervals reviewable. Merge after #251.

[sebasmos]

Correct and thin. This is run-report infra though, and during the 5-day sprint priority-3 tooling should wait behind your MedMCQA battery (#291). Requesting changes to park it, nothing against the code.

Superseded by the later Changes-requested review on this PR; dismissing the stale approve to avoid a mixed signal.

[Agastya191]

Nice, and thin in the right way: I checked the point estimates line up with the headline table, flip_rate's bootstrap mean over all records matches analysis.flip_rate and shortcut_reliance matches shortcut_reliance_index by linearity of the mean, and resampling the paired per-observation difference for shortcut_reliance / adoption_delta rather than the two rates separately is the right call. One thing worth a look, and it's the same principle you invoke for not pooling models: the solo lane resamples flip_rate over [r.flipped for r in records], but _build_twins emits one twin per (case, cue_type), so with TEXT_CUES the several twins from a single case sit in that flat array and bootstrap_ci resamples them iid. Same-case twins share the case's difficulty, so counting them as independent understates the uncertainty the same way pooling models would, and the reported 95% CI comes out tighter than the case-level evidence supports (your own n=12-per-model example is 4 cases seen three ways). Twin-level is a defensible estimand if you name it, but I'd resample at the case cluster instead, drawing cases and taking all their twins, so the interval answers "does this hold on new cases", which is the claim the arm is actually making.

[duckyquang]

@Agastya191 right, and fixed in 1ebd019. The solo lane now resamples by case cluster instead of iid over twins: I added stats.cluster_bootstrap_ci, which draws whole cases and pools all their twins, and keyed the solo estimates on case_id. So the interval answers "does this hold on new cases," which is the claim the arm actually makes. The point estimate is unchanged, and because every cluster is a singleton for one-twin-per-case data the function reduces exactly to bootstrap_ci — so the pinned intervals don't move, only real multi-cue cases widen.

One follow-up I'll flag honestly: the two noise-floor estimates (noise_floor and flip_rate_above_floor) live up the stack in #258 and still resample per twin, so once this and #258 sit together they'll want the same clusters= treatment — otherwise the floor-adjusted rate ends up with a tighter CI than the raw flip rate it corrects. I'll do that on #258 when it rebases onto this, since the cluster helper lands here.

@sebasmos fair on priority — the MedMCQA battery (#291) is where my time's going. I landed the cluster-CI fix so this is ready to pick back up whenever we get to the run-report tooling, but happy to keep it parked behind the runs.

@maximinl thanks. Same-case twins are now resampled as a cluster too, same principle as not pooling models.

[sebasmos]

Approving. cluster_bootstrap_ci at stats.py:256 genuinely groups by case and draws whole clusters, the point estimate is untouched, and the test builds its cases by hand rather than reading expectations off the file under test, so it is not self-confirming. I merged current main in and resolved the post-squash conflicts, all eight of which were this branch's own additions against main's pre-feature version. 967 passed, ruff clean.