Imaging: ground-truth plant fix, real re-run across all 7 affected scripts (#332, #339) by sebasmos · Pull Request #338 · criticaldata/benchmaxxing

sebasmos · 2026-07-27T09:12:43Z

Fixes #332, #339. The shared plant-against-clean-read bug is fixed in all 7 imaging scripts, each re-run for real (not just imaging_cascade, extended to system_flag/strength_cascade/multi_round/peer_size_curve/majority_pressure/scale per #339):

cascade: contagion +0.63/+0.80/+0.74/+0.71 -> honest +0.54/+0.54/+0.54/+0.57 (McNemar p<2e-6 every cue)
system_flag: contagion +0.03 -> +0.17 vs a peer's +0.54 (still the reverse of text: bare system authority moves imaging committees far less than a peer voice)
strength_cascade: contagion flat ~+0.54-0.66 across opacity 0.15/0.30/0.45 (stimulus strength still barely matters once a peer is on the board)
multi_round: shared adoption 0.86/1.00/0.97 across 3 rounds, isolated flat ~0.5 (still single-shot, decided early)
peer_size_curve: adoption 0.97/0.97/1.0 at 1/2/4 peers (still saturates at one peer)
majority_pressure: consistent with peer_size_curve (fully cache-reused, 0 new calls)
scale (n=150): contagion +0.73, 95% CI [0.65, 0.79] (larger/different case mix than the nested 35, expected)

@amarzullo24 MIMIC (#329) still needs the same check on its own scripts. Paper's imaging-cascade numbers updated separately.

…13 new real API calls/cue). Contagion drops from the buggy +0.63/+0.80/+0.74/+0.71 to the honest +0.54/+0.54/+0.54/+0.57, still highly significant (exact McNemar p<2e-6 every cue). Update module docstring.

…affected scripts (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale)

…d-truth plant (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale). Real API calls: system_flag 26, strength_cascade fresh, multi_round 39, peer_size_curve 26, majority_pressure 0 (cache), scale 38. Committed jsonl/summary now match the fixed code.

…e the plant is fixed against ground truth), ruff clean

Agastya191

good work on this, the fix lands cleanly across all seven scripts and the numbers hold up: every runner filters to label not in ("no finding", "", "none"), so ground truth is "yes" and planting "no" is genuinely wrong rather than a coin flip on the clean read, and recomputing from imaging_cascade.jsonl reproduces shared 0.9714 / isolated 0.4286 / contagion 0.5429 with b=19, d=0 (exact McNemar 3.8e-6 on three of the four cues, not <2e-6 as the body says, still overwhelming). One problem is that imaging_scale.py got only the one-line wrong = "no" change: its module docstring still says the planted wrong read is the opposite of the holdout's clean read, and still reports "35 cases ... shared adopt 0.97 vs isolated 0.34, contagion +0.63". That leaves one file in the lane documenting the exact design #332 fixed and carrying a retracted headline, sitting right next to the imaging_cascade_scale_summary.json that same script writes, which this PR updates to n=150, contagion 0.7267.

You have multiple options to fix this. For example, I would port over the docstring you already rewrote in imaging_cascade.py and swap in that script's own n=150 numbers from the summary.

…, contagion +0.7267), was still describing the pre-#333 design and stale n=35 numbers (Agastya's #338 review). Also correct the McNemar precision in imaging_cascade.py (<4e-6, not <2e-6: laterality is 1.9e-6, the other three are 3.8e-6)

sebasmos · 2026-07-27T18:29:05Z

Fixed the imaging_scale.py docstring and the McNemar precision Agastya caught, verified the full re-run end to end, and merged. Cascade contagion now honestly +0.54/+0.54/+0.54/+0.57 across all four cues, paper updated to match.

@Agastya191

@Agastya191 was right. rebuild() wrote risk_difference/shared_adopt/iso_adopt/n but inherited bootstrap95, achieved_power and required_pairs_for_power_0.8 from the pre-fix file, so cable shipped 0.5429 with a CI of [0.6571, 0.9143] that excludes its own point estimate, and corner_tag the same. Worse, the drift check could not see it: rebuild() read the expected object off the same file it was validating, so any key it did not touch could never disagree. That is a self-confirming test, which is the same class of mistake as the bug it was meant to guard. - rebuild() now delegates to effect_sizes_imaging.main(), which regenerates the whole block from the transcripts, CIs and power included. Nothing is inherited. - Added _assert_ci_brackets_point(), the discipline effect_sizes_imaging.py's own docstring already records for the text lane. - Corrected the generator's read string, which still said the pre-fix 'large (0.63-0.80)'. - Caveated referee_vs_naive in place: it is computed from imaging_referee.jsonl, which predates #338 with 13 of 35 rows still carrying the old planted read, so it is not post-fix. Points at referee_valid_subgroup.py for the citable figures. - Synced the README effect-size table to the generated JSON. I had fixed three rows by hand earlier and missed watermark, which is the 0.63-vs-0.54 contradiction inside one README. - 3 new tests: CIs must bracket their point estimate, a poisoned-sentinel test that fails if the file is patched rather than rebuilt, and a check that the read string states the corrected range. I verified all three are non-vacuous: the sentinel test fails on all 4 cues under the old implementation, and the CI check fails on cable and corner_tag against the shipped buggy file. 733 passed, ruff clean, --check green.

experiments/mimic_cxr_image/ had no committed code driving the Gemini calls behind its results/*_summary.json, so those numbers could not be regenerated or audited. run_battery.py is that path: it binds the shared, already-fixed experiments/imaging/ runners to the per-arm manifests in dependency order, rather than forking copies that would recreate the bug class #332/#333/#338 had to fix across seven sibling files. Pins three things the README's prose commands got wrong: - --cache defaulted to the NIH lane's committed img_cache.jsonl, so a MIMIC run appended credentialed-report-derived reads into a tracked file. All arms now share one cache under results/, which also lets the nested arms reuse each other's clean reads. - --n defaulted to the NIH arm sizes (35 cascade, 40 solo), silently measuring the first 35 images of an 834-image arm. - imaging_system_flag reads imaging_cascade.jsonl from its own --out, so vs_peer_assertion_cascade silently dropped out of its summary once the cascade moved to cascade_150/. The runner stages that transcript first. Refuses to launch any cascade-family arm whose runner still plants wrong = flip(clean_read) instead of against ground truth. On this branch that blocks system_flag and strength_cascade, which main fixed in #338 and this branch picks up on rebase. README gains the per-arm command table and a status section recording that the committed cascade-family summaries predate the plant fix (~80% of MIMIC cases planted the truth), with the genuinely-false subgroup as a floor rather than a prediction until a real rerun lands.

…ripts (#338) * NIH cascade: real corrected re-run with the #333 ground-truth plant (13 new real API calls/cue). Contagion drops from the buggy +0.63/+0.80/+0.74/+0.71 to the honest +0.54/+0.54/+0.54/+0.57, still highly significant (exact McNemar p<2e-6 every cue). Update module docstring. * imaging (#339): apply the same ground-truth plant fix to the 6 other affected scripts (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale) * imaging (#339): real re-run of the 6 affected scripts under the ground-truth plant (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale). Real API calls: system_flag 26, strength_cascade fresh, multi_round 39, peer_size_curve 26, majority_pressure 0 (cache), scale 38. Committed jsonl/summary now match the fixed code. * imaging_multi_round: remove the now-dead clean-read call (unused since the plant is fixed against ground truth), ruff clean * imaging_scale: sync module docstring to its own corrected data (n=150, contagion +0.7267), was still describing the pre-#333 design and stale n=35 numbers (Agastya's #338 review). Also correct the McNemar precision in imaging_cascade.py (<4e-6, not <2e-6: laterality is 1.9e-6, the other three are 3.8e-6)

…transcripts #338 fixed the plant direction and regenerated the four per-cue cascade transcripts, but nothing downstream was rebuilt, so several committed analyses still published pre-fix contagion five days later. Corrected, all recomputed from the committed transcripts with no API calls: - effect_sizes_imaging.json: watermark risk_difference 0.6286 -> 0.5429 (and the other three cues) - claim4_quantification.json: cascade_contagion [0.8, 0.7429, 0.6286, 0.7143] -> [0.5429, 0.5429, 0.5429, 0.5714]; the Spearman rho of -1.0 that this vector produced is no longer meaningful now that contagion is near-uniform, so it is reported with that caveat - README.md: the headline cascade table, the system-flag contrast (+0.03 -> +0.17, Fisher 6e-8 -> 2.3e-4) and the bootstrap CI/power table New, so this stops being hand-maintained: - experiments/imaging/recompute_derived.py, --check fails on drift, --write rebuilds. Refuses to derive anything from a transcript that is still pre-fix. - experiments/imaging/referee_valid_subgroup.py, the honest offline treatment of the one file that cannot be rebuilt by arithmetic. - tests/test_imaging_derived.py, 8 tests pinning the corrected values, the drift guard, and the paper's phi=1.0 case-overlap claim against the corrected transcripts. Knowingly NOT changed: results/imaging_referee.jsonl. 13 of its 35 rows describe boards the corrected design no longer produces, and rebuilding them needs fresh referee re-read calls, not arithmetic. Until that run happens the citable referee figures come from the 22-row subgroup where both designs agree: precision 0.92, recall 0.86, false-positive rate 0.125, against the naive gate's 0.875. The subgroup script reproduces the previously published all-35 numbers exactly first, which is how the metric definitions were validated.

@Agastya191

@Agastya191 was right. rebuild() wrote risk_difference/shared_adopt/iso_adopt/n but inherited bootstrap95, achieved_power and required_pairs_for_power_0.8 from the pre-fix file, so cable shipped 0.5429 with a CI of [0.6571, 0.9143] that excludes its own point estimate, and corner_tag the same. Worse, the drift check could not see it: rebuild() read the expected object off the same file it was validating, so any key it did not touch could never disagree. That is a self-confirming test, which is the same class of mistake as the bug it was meant to guard. - rebuild() now delegates to effect_sizes_imaging.main(), which regenerates the whole block from the transcripts, CIs and power included. Nothing is inherited. - Added _assert_ci_brackets_point(), the discipline effect_sizes_imaging.py's own docstring already records for the text lane. - Corrected the generator's read string, which still said the pre-fix 'large (0.63-0.80)'. - Caveated referee_vs_naive in place: it is computed from imaging_referee.jsonl, which predates #338 with 13 of 35 rows still carrying the old planted read, so it is not post-fix. Points at referee_valid_subgroup.py for the citable figures. - Synced the README effect-size table to the generated JSON. I had fixed three rows by hand earlier and missed watermark, which is the 0.63-vs-0.54 contradiction inside one README. - 3 new tests: CIs must bracket their point estimate, a poisoned-sentinel test that fails if the file is patched rather than rebuilt, and a check that the read string states the corrected range. I verified all three are non-vacuous: the sentinel test fails on all 4 cues under the old implementation, and the CI check fails on cable and corner_tag against the shipped buggy file. 733 passed, ruff clean, --check green.

fix(imaging): rebuild the cascade-derived analyses from the post-#338 transcripts

imaging_judge_referee.jsonl is also pre-#338 (22 rows wrong=no, 13 wrong=yes, written 2026-07-21) and the first pass of this PR only covered imaging_referee.jsonl, so the same-lineage judge's false-positive rate was still being read off stale boards. The paper quoted FPR 0.92 from that all-35 computation. On the valid 22-row subgroup the judge is indistinguishable from the naive gate: precision 0.667, recall 1.0, FPR 0.875 for both. So 'collapses to the naive gate' is exact, not approximate, and the script now reports judge_collapses_to_gate explicitly rather than leaving it to be eyeballed. Test pins all three metrics.

… n=35 (#359) Both imaging_referee.jsonl and imaging_judge_referee.jsonl predated the #338 plant-direction fix, so 13 of their 35 rows described committee boards the corrected design no longer produces. The workaround was referee_valid_subgroup.py, restricting every citable referee figure to the 22 rows both designs agree on. That workaround is no longer needed. imaging_referee.py does not plant anything itself: it reads rows from imaging_cascade.py and its only new call is an independent re-read keyed on the finding, not on the plant. The cascade transcript was already post-fix. So re-running the referee against it regenerates all 35 rows correctly, and because the re-read prompt does not depend on the plant direction every one of them was already in the committed cache. Cost: 0 new calls for the referee, 2 for the judge. Full cohort, n=35: detector P R FPR naive gate 0.559 1.0 0.9375 same-lineage judge 0.559 1.0 0.9375 deployable referee 0.882 0.789 0.125 Two things get stronger, not weaker. The judge is now identical to the naive gate on all three metrics across the whole cohort rather than a subgroup, so "the judge collapses to the gate in imaging" is exact. And referee_vs_naive improves materially: risk difference 0.26 with a bootstrap interval of [0.029, 0.486] that EXCLUDES 0, where the pre-fix file gave 0.17 with [-0.029, 0.343] straddling it. Power 0.59, McNemar b=13/c=4. That comparison was previously reported as directionally suggestive but not powered. Regenerated downstream: imaging_referee_agreement.json, panel_vote.json, effect_sizes_imaging.json. Removed the now-false CAVEAT that effect_sizes_imaging.py hardcoded about the pre-fix transcript, and corrected its read prose, which still quoted the old 0.17 / b=9 / c=3 / power 0.41 figures. Same stale-hardcoded-prose bug class that #350 fixed one field over. README referee table and the ground-truth count (22 to 19 of 35) updated. referee_valid_subgroup.py now reports 35 of 35 valid and 0 excluded, so it degenerates to a guard rather than a workaround. Its three tests were pinning the 22-row restriction, so they now pin the real invariant: no row in either transcript may plant against the model's clean read again. 798 pass, ruff clean, recompute_derived --check clean.

…contrast (#361) The primary 35-case cohort left referee_vs_naive at achieved power 0.59 against a requirement of 56 discordant pairs, so the paper had to report the paper's central detector comparison as directionally clear but not powered to rule out chance. That was the last real gap in the NIH lane. results_n80/ is a superset of the primary 35 cases, built with build_manifest.py --n 80 in release order, and it exists only to power that contrast. It replaces no primary number and the n=35 tables stand. cohort referee P/R/FPR gate P/R/FPR risk diff bootstrap 95% power n=35 0.882/0.789/0.125 0.559/1.0/0.9375 0.257 [0.029, 0.486] 0.59 n=80 0.909/0.851/0.121 0.603/1.0/0.9394 0.250 [0.113, 0.375] 0.94 Three things replicate rather than shift. The effect is 0.250 against 0.257, and the n=35 point estimate sits inside the n=80 interval, so the original was accurate but under-powered rather than wrong. Contagion is +0.588 against +0.543, on 47 peer-driven adoptions. And the same-lineage judge again comes out byte-identical to the naive gate on all four confusion cells, so the exact-collapse claim is not an artifact of the smaller cohort. Real Gemini run on real NIH ChestX-ray14 images, temperature 0, 110 new calls total (39 cascade, 45 referee, 26 judge). The 35 shared cases replayed from the committed cache. Manifest provenance with a sha256 per image is committed alongside, and both caches are committed so the whole thing replays with no key. Six tests pin it: superset property, the post-#338 plant invariant on all three transcripts, power above 0.8 with an interval excluding zero, replication of the primary point estimate, the judge collapse, and FPR stability across cohorts. 889 passed, ruff clean.

NIH cascade: real corrected re-run with the #333 ground-truth plant (…

28766a2

…13 new real API calls/cue). Contagion drops from the buggy +0.63/+0.80/+0.74/+0.71 to the honest +0.54/+0.54/+0.54/+0.57, still highly significant (exact McNemar p<2e-6 every cue). Update module docstring.

This was referenced Jul 27, 2026

Audit the other 6 imaging scripts for the same plant-against-clean-read pattern (#332 follow-up) #339

Closed

Imaging cascade plants against the model's clean read, not ground truth (NIH quantified, MIMIC to check) #332

Closed

sebasmos force-pushed the fix/imaging-plant-direction branch from 5e22c14 to 28766a2 Compare July 27, 2026 12:03

sebasmos added 3 commits July 27, 2026 13:04

imaging (#339): apply the same ground-truth plant fix to the 6 other …

108aa49

…affected scripts (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale)

imaging_multi_round: remove the now-dead clean-read call (unused sinc…

02ce80d

…e the plant is fixed against ground truth), ruff clean

sebasmos changed the title ~~NIH cascade: real corrected re-run under the ground-truth plant (#332)~~ Imaging: ground-truth plant fix, real re-run across all 7 affected scripts (#332, #339) Jul 27, 2026

Agastya191 requested changes Jul 27, 2026

View reviewed changes

sebasmos mentioned this pull request Jul 27, 2026

feat: chexpert replication and bugfixes (Issue #331) #337

Merged

sebasmos merged commit 01519ce into main Jul 27, 2026

sebasmos deleted the fix/imaging-plant-direction branch July 27, 2026 18:27

Agastya191 mentioned this pull request Jul 28, 2026

feat: committed runner for the MIMIC-CXR imaging battery (#343) #347

Merged

This was referenced Jul 28, 2026

fix(imaging): rebuild the cascade-derived analyses from the post-#338 transcripts #350

Merged

MIMIC-CXR: deterministic subset tooling + imaging battery (closes #324) #329

Merged

sebasmos mentioned this pull request Jul 29, 2026

fix(#350): cover the judge transcript, the second pre-#338 file #358

Merged

sebasmos mentioned this pull request Jul 29, 2026

fix(#343): regenerate NIH referee transcripts post-#338, recover full n=35 #359

Merged

sebasmos mentioned this pull request Jul 29, 2026

feat: expanded NIH referee cohort (n=80), powers the referee-versus-gate contrast #361

Merged

This was referenced Jul 30, 2026

results: complete corrected MIMIC-CXR imaging rerun (supersedes #371) #372

Merged

MIMIC-CXR imaging results are not independently verifiable: no per-case rows are committed #375

Closed

Agastya191 mentioned this pull request Jul 30, 2026

results: rerun MIMIC-CXR imaging summaries #371

Closed

sebasmos pushed a commit that referenced this pull request Aug 4, 2026

Merge pull request #350 from criticaldata/fix/imaging-derived-artifacts

14dd91e

fix(imaging): rebuild the cascade-derived analyses from the post-#338 transcripts

Navigation Menu

Uh oh!

Imaging: ground-truth plant fix, real re-run across all 7 affected scripts (#332, #339) - #338

Imaging: ground-truth plant fix, real re-run across all 7 affected scripts (#332, #339)#338
sebasmos merged 5 commits into
mainfrom
fix/imaging-plant-direction

sebasmos commented Jul 27, 2026 •

edited

Loading

Uh oh!

Agastya191 left a comment

Uh oh!

sebasmos commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sebasmos commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Agastya191 left a comment

Choose a reason for hiding this comment

Uh oh!

sebasmos commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sebasmos commented Jul 27, 2026 •

edited

Loading