Imaging cascade: plant against ground truth, not the clean read (#332) by sebasmos · Pull Request #333 · criticaldata/benchmaxxing

sebasmos · 2026-07-26T01:28:08Z

Fixes #332. The shared imaging-cascade planted wrong = flip(clean_read), which on finding-present cases coincides with ground truth whenever the holdout was already wrong (~37% at NIH accuracy), scoring an accept-the-correction as contagion.

imaging_cascade.py: plant against ground truth (the definitely-false read); record clean_correct.
plant_direction_check.py: offline re-analysis of the committed NIH transcripts. The genuinely-false subgroup matches the overall (~+0.73 across cues), so NIH's effect is real harmful sycophancy, not an artifact (unlike the all-pneumothorax CheXpert pilot, CheXpert: scale the imaging battery to ~600 solo / 150 cascade (match MIMIC) #331).

@amarzullo24 MIMIC (#329) reuses this same path; running plant_direction_check.py on the MIMIC transcripts would confirm it is likewise robust. MedQA text is unaffected (it plants a distractor).

…; add plant_direction_check.py showing NIH's genuinely-false subgroup matches the overall (~+0.73), so the effect is real sycophancy not accept-the-correction

…#339) Offline split of the committed transcripts into genuinely-false (wrong==no, the honest harmful-sycophancy measure) vs planted-truth (wrong==yes). On MIMIC ~80% of finding-present cases plant the truth (poor pneumothorax clean reads), so ALL contagion is inflated: - cascade_150 (#311): ALL +0.82 -> genuinely-false +0.49 - referee_300 cascade: ALL +0.81 -> +0.51 - strength_cascade (#313): ALL ~+0.80 -> ~+0.53-0.55 (flat across opacity) - system_flag (#312): robust, +0.225 ~ ALL +0.200 Qualitative conclusions unchanged. No API/network; reads transcripts only. imaging_cascade fixed on main (#333); system_flag/strength_cascade tracked in #339. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…13 new real API calls/cue). Contagion drops from the buggy +0.63/+0.80/+0.74/+0.71 to the honest +0.54/+0.54/+0.54/+0.57, still highly significant (exact McNemar p<2e-6 every cue). Update module docstring.

…, contagion +0.7267), was still describing the pre-#333 design and stale n=35 numbers (Agastya's #338 review). Also correct the McNemar precision in imaging_cascade.py (<4e-6, not <2e-6: laterality is 1.9e-6, the other three are 3.8e-6)

…ripts (#338) * NIH cascade: real corrected re-run with the #333 ground-truth plant (13 new real API calls/cue). Contagion drops from the buggy +0.63/+0.80/+0.74/+0.71 to the honest +0.54/+0.54/+0.54/+0.57, still highly significant (exact McNemar p<2e-6 every cue). Update module docstring. * imaging (#339): apply the same ground-truth plant fix to the 6 other affected scripts (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale) * imaging (#339): real re-run of the 6 affected scripts under the ground-truth plant (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale). Real API calls: system_flag 26, strength_cascade fresh, multi_round 39, peer_size_curve 26, majority_pressure 0 (cache), scale 38. Committed jsonl/summary now match the fixed code. * imaging_multi_round: remove the now-dead clean-read call (unused since the plant is fixed against ground truth), ruff clean * imaging_scale: sync module docstring to its own corrected data (n=150, contagion +0.7267), was still describing the pre-#333 design and stale n=35 numbers (Agastya's #338 review). Also correct the McNemar precision in imaging_cascade.py (<4e-6, not <2e-6: laterality is 1.9e-6, the other three are 3.8e-6)

…#339) Offline split of the committed transcripts into genuinely-false (wrong==no, the honest harmful-sycophancy measure) vs planted-truth (wrong==yes). On MIMIC ~80% of finding-present cases plant the truth (poor pneumothorax clean reads), so ALL contagion is inflated: - cascade_150 (#311): ALL +0.82 -> genuinely-false +0.49 - referee_300 cascade: ALL +0.81 -> +0.51 - strength_cascade (#313): ALL ~+0.80 -> ~+0.53-0.55 (flat across opacity) - system_flag (#312): robust, +0.225 ~ ALL +0.200 Qualitative conclusions unchanged. No API/network; reads transcripts only. imaging_cascade fixed on main (#333); system_flag/strength_cascade tracked in #339. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

experiments/mimic_cxr_image/ had no committed code driving the Gemini calls behind its results/*_summary.json, so those numbers could not be regenerated or audited. run_battery.py is that path: it binds the shared, already-fixed experiments/imaging/ runners to the per-arm manifests in dependency order, rather than forking copies that would recreate the bug class #332/#333/#338 had to fix across seven sibling files. Pins three things the README's prose commands got wrong: - --cache defaulted to the NIH lane's committed img_cache.jsonl, so a MIMIC run appended credentialed-report-derived reads into a tracked file. All arms now share one cache under results/, which also lets the nested arms reuse each other's clean reads. - --n defaulted to the NIH arm sizes (35 cascade, 40 solo), silently measuring the first 35 images of an 834-image arm. - imaging_system_flag reads imaging_cascade.jsonl from its own --out, so vs_peer_assertion_cascade silently dropped out of its summary once the cascade moved to cascade_150/. The runner stages that transcript first. Refuses to launch any cascade-family arm whose runner still plants wrong = flip(clean_read) instead of against ground truth. On this branch that blocks system_flag and strength_cascade, which main fixed in #338 and this branch picks up on rebase. README gains the per-arm command table and a status section recording that the committed cascade-family summaries predate the plant fix (~80% of MIMIC cases planted the truth), with the genuinely-false subgroup as a floor rather than a prediction until a real rerun lands.

…ripts (#338) * NIH cascade: real corrected re-run with the #333 ground-truth plant (13 new real API calls/cue). Contagion drops from the buggy +0.63/+0.80/+0.74/+0.71 to the honest +0.54/+0.54/+0.54/+0.57, still highly significant (exact McNemar p<2e-6 every cue). Update module docstring. * imaging (#339): apply the same ground-truth plant fix to the 6 other affected scripts (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale) * imaging (#339): real re-run of the 6 affected scripts under the ground-truth plant (system_flag, strength_cascade, multi_round, peer_size_curve, majority_pressure, scale). Real API calls: system_flag 26, strength_cascade fresh, multi_round 39, peer_size_curve 26, majority_pressure 0 (cache), scale 38. Committed jsonl/summary now match the fixed code. * imaging_multi_round: remove the now-dead clean-read call (unused since the plant is fixed against ground truth), ruff clean * imaging_scale: sync module docstring to its own corrected data (n=150, contagion +0.7267), was still describing the pre-#333 design and stale n=35 numbers (Agastya's #338 review). Also correct the McNemar precision in imaging_cascade.py (<4e-6, not <2e-6: laterality is 1.9e-6, the other three are 3.8e-6)

* refactor(chexpert): delete duplicate runners, use shared imaging_*.py (#331) Addresses all feedback from Agastya191 on Issue #331 and PR #328: Adapter (benchmaxxing/datasets/chexpert.py): - Multi-label extraction with CLINICAL_HIERARCHY ordering (14 findings) - Explicit uncertainty policy: -1.0 treated as negative (cannot anchor a plant) - Fixed limit bug: deterministic subsample via budget.subsample_cases, not CSV truncation Structural refactor: - Deleted chexpert_solo.py, chexpert_cascade.py, chexpert_referee.py, chexpert_matched_temp_floor.py (copies of imaging_*.py differing by 12-26 lines) - Deleted build_manifest.py (replaced by build_subset.py) - New build_subset.py following the MIMIC-CXR pattern (#329): deterministic nested per-arm manifests (solo_600, referee_300, cascade_150, nih_match_35) with finding-absent as a separate subgroup - Shared runners inherit all fixes: ground-truth plant (#333), clean_correct field, matched temp floor, strength cascade, blind metric, judge referee Tests: - Updated test_chexpert_adapter.py: multi-label, clinical hierarchy, uncertainty policy, limit subsampling (8 tests, all passing) * feat(staging): move per-image SHA256 from custom builders to stage_dataset * fix: address review comments on staging and subsets * compliance: implement preregistration endpoints E1/E2 for CheXpert * feat: add chexpert blind metric substitution probe (Issue #308) * docs: add cross-dataset comparison table for blind metric (Issue #294) * docs: update CheXpert metrics with solo_600 partial run results * style: fix remaining ruff lint errors in CheXpert runner scripts * feat(imaging): Phase 1 Issue #331 structural improvements - Implement strict round-robin clinical stratification in build_manifest.py using clinical hierarchy to prevent monocultures (e.g. pneumothorax dominating). - Centralize --n sampling into build_manifest.py and remove legacy hardcoded --n dependencies from solo, cascade, and temp_floor scripts. - Introduce n_genuinely_false and genuinely_false_contagion metrics to imaging_cascade.py to strictly isolate cases where clean reads failed. - Parameterize --cue in imaging_matched_temp_floor.py (removing 'watermark' hardcodes from both CLI and summary read outputs) to dynamically support all 4 injectors. - Ensure cross-dataset reproducibility by asserting random.shuffle is performed natively across the normalized Case outputs, agnostic to the dataset adapter. * feat: standardise MIMIC-CXR for universal imaging lane and fix matched-temp floor (#343, #344) This commit integrates the MIMIC-CXR adapter fully into the universal imaging orchestrator. - Decoupled FINDING_COLUMNS and CLINICAL_HIERARCHY into a shared benchmaxxing/utils/clinical_labels.py to prevent tight coupling between chexpert and mimic_cxr. - Updated mimic_cxr.py to extract multi-labels (pipe-separated) matching CheXpert's conservative policy (treating -1.0 uncertainty as negative) to enable proper dataset stratification. - Used safe parameter injection (label_format="legacy") inside build_manifest.py via inspect.signature to guarantee 100% backward compatibility for legacy text-lane scripts without poisoning Python's module-level global state. - Updated CROSS_DATASET_COMPARISON.md confirming that all 3 imaging datasets now strictly run under the matched-temperature floor. * fix(#337): meta assertion broke on the additive labels field test_join_reports_and_labels asserted exact dict equality on Case.meta, so it failed as soon as this branch surfaced the full CheXpert label vector as meta["labels"] for natural-cue selection. The new field is legitimate, the assertion was brittle. Now asserts the three fields the imaging lane depends on exactly, bounds the allowed key set so an unrelated field still fails loudly, and validates the shape of labels when present. Strictness is kept, additivity is allowed. Merged with current main this gives 800 passed, 7 skipped, ruff clean. * fix(#337): make --n consistent across the imaging runners, and keep the cohort audit trail Two review asks, both about not silently changing the cohort every published NIH number sits on. 1. The --n removal was only half applied. imaging_cascade, imaging_solo and imaging_matched_temp_floor lost the flag, while imaging_system_flag, imaging_strength_cascade and imaging_blind_metric still accept it, so the nine documented invocations in this directory's README worked for some runners and exited with "unrecognized arguments: --n 35" for others. Restored it on the three, which is the least invasive way to make the six consistent and keeps every documented command valid. If the intent is really that the manifest alone dictates n, that is a reasonable design, but it needs applying to all six runners and the README rewritten in the same change. 2. build_manifest.py had switched the default from release order to seeded round-robin stratification and dropped the docstring sentence promising that a rebuild reproduces the committed 35-case set. Stratification is genuinely useful for building a NEW cohort with better finding coverage, so it stays, but behind an explicit --stratify flag with release order restored as the default. label_format is now only requested when stratifying, since asking unconditionally moved the default cohort too. Verified both paths against the committed manifest: a default release-order rebuild at --n 35 recovers the same 35-case set, and --case-ids-file reproduces it byte-identically, which is what the docstring promises. Four tests pin the contract, one of them asserting the reproducibility sentence is present, since it was deleted once already. The plant direction, the guard result and the subgroup work on this branch are untouched. --------- Co-authored-by: YEHUDHAH KENNEDY RODRIGUEZ MORAN <yehu@192.168.1.4> Co-authored-by: YEHUDHAH KENNEDY RODRIGUEZ MORAN <yehu@192.168.1.6> Co-authored-by: sebasmos <sebasticajas@gmail.com>

imaging cascade: plant against ground truth, not the clean read (#332)…

5e22c14

…; add plant_direction_check.py showing NIH's genuinely-false subgroup matches the overall (~+0.73), so the effect is real sycophancy not accept-the-correction

sebasmos requested a review from amarzullo24 July 26, 2026 01:30

sebasmos merged commit ca2088e into main Jul 26, 2026

sebasmos deleted the fix/imaging-plant-direction branch July 26, 2026 01:31

Agastya191 mentioned this pull request Jul 26, 2026

CheXpert: scale the imaging battery to ~600 solo / 150 cascade (match MIMIC) #331

Closed

8 tasks

sebasmos restored the fix/imaging-plant-direction branch July 27, 2026 09:12

This was referenced Jul 27, 2026

Imaging: ground-truth plant fix, real re-run across all 7 affected scripts (#332, #339) #338

Merged

Audit the other 6 imaging scripts for the same plant-against-clean-read pattern (#332 follow-up) #339

Closed

amarzullo24 mentioned this pull request Jul 27, 2026

MIMIC-CXR: deterministic subset tooling + imaging battery (closes #324) #329

Merged

This was referenced Jul 28, 2026

MIMIC-CXR imaging: no committed runner script for cascade/solo/system_flag/blind_metric (mirrors #337's CheXpert gap) #343

Closed

Port the matched-temperature noise floor (#205) to CheXpert and MIMIC-CXR imaging solo claims #344

Closed

Agastya191 mentioned this pull request Jul 28, 2026

feat: committed runner for the MIMIC-CXR imaging battery (#343) #347

Merged

sebasmos mentioned this pull request Jul 28, 2026

fix(imaging): rebuild the cascade-derived analyses from the post-#338 transcripts #350

Merged

This was referenced Jul 30, 2026

results: complete corrected MIMIC-CXR imaging rerun (supersedes #371) #372

Merged

MIMIC-CXR imaging results are not independently verifiable: no per-case rows are committed #375

Closed

Agastya191 mentioned this pull request Jul 30, 2026

results: rerun MIMIC-CXR imaging summaries #371

Closed

sebasmos mentioned this pull request Aug 1, 2026

MIMIC-CXR imaging: same-lineage judge detector never run (table gap vs NIH) #393

Open

sebasmos added a commit that referenced this pull request Aug 4, 2026

Imaging cascade: plant against ground truth, not the clean read (#333)

2942c44

Navigation Menu

Uh oh!

Imaging cascade: plant against ground truth, not the clean read (#332) - #333

Imaging cascade: plant against ground truth, not the clean read (#332)#333
sebasmos merged 1 commit into
mainfrom
fix/imaging-plant-direction

sebasmos commented Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

sebasmos commented Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant