results: MedMCQA foundational arms — solo, cascade, break-it A + C

[duckyquang]

Summary

First four arms of the MedMCQA battery, run through the existing MedQA scripts with no code changes — I just pointed reproduce.py, clean_a.py, and scale_c.py at a MedMCQA manifest built from the medmcqa adapter. Real Gemini API runs (temp 0) on the MedMCQA validation split (the labeled one; test has no public answers). This is the foundation: solo susceptibility + noise floor, the bare cascade baseline, and both break-it mechanisms.

Result

• Solo (Solo shortcut susceptibility + noise floor (MedMCQA) #267): flash flip 0.12 (clean acc 0.83, noise floor 0.067), flash-lite flip 0.237 (noise 0.0). Weaker tier ~2x more cue-susceptible.
• Cascade baseline (Cascade baseline: bare and anchored seed (MedMCQA) #268): null, contagion +0.025 over 20 cases — matches MedQA's null. Verified the seed is injected every run and committees reject it.
• Break-it A, system pre-screen flag (Break-it A: contaminated shared context / system pre-screen flag (MedMCQA) #288): flash +0.267, flash-lite +0.533, control 0.0 (30 hard cases). A trusted-system flag drives a large cascade, bigger than MedQA (flash-lite 0.33 there).
• Break-it C, anchored plausibility (Break-it C: case-anchored plausible seed vs generic (MedMCQA) #289): generic 0.718 -> anchored 0.818, McNemar p=0.019 (110 hard cases). Anchoring replicates and is significant.

The story: the bare cascade is null on MedMCQA just like MedQA, but authority/plausibility-driven cascades are real and if anything stronger here — flash-lite is notably more deferential to asserted senior peers.

Notes

• Model calls are cached locally in call_cache.jsonl, but it's gitignored (~13MB, just a reproduction accelerator). The committed evidence is the summaries, the per-case records (solo_records.jsonl, scale_c_per_case.jsonl), and the cascade transcripts (committed for offline replay).
• Data: openlifescienceai/medmcqa validation split. Raw data is gitignored; only derived results are committed. Answer indices were cross-checked against the raw cop labels (0 mismatches).
• scale_c's live run hung at write time (a Gemini call with no client-side timeout); Break-it C: case-anchored plausible seed vs generic (MedMCQA) #289's numbers are recomputed offline from the committed cache over the 110 hard cases whose full condition set completed. This is flagged in scale_c_summary.json and RESULTS.md.
• One caveat I documented rather than fixed: experiments/medqa/results/RESULTS.md prose cites MedQA solo flip 0.787/0.893, which contradicts that folder's own solo_records.jsonl (0.063/0.117). I compared MedMCQA against the per-record data (the internally consistent one). Worth a separate look.
• Remaining ~21 arms are next (tracked in MedMCQA standard-battery replication (tracking) #291). Several stall on my laptop under the scripts' per-call-client concurrency, so I'll run those on stable compute.

@sebasmos @Agastya191 this is the sprint-priority MedMCQA work.

Closes #267
Closes #268
Closes #288
Closes #289

[sebasmos]

This is a real upgrade, a third text dataset with the full solo, cascade and break-it story, and the bare-cascade null replicating MedQA is a good independent check. Two asks before merge: the scale_c live run hung at write time and its numbers are recomputed offline, so please state the n and which cases were dropped in the summary itself, not only RESULTS.md. And confirm the answer-index cross-check against the raw cop labels is committed somewhere reproducible.

[duckyquang]

@sebasmos two things you asked for:

1. The answer-index cross-check against the raw cop labels is committed and reproducible — it's the adapter itself: medmcqa.py sets answer_index=cop and validates 0 <= cop < len(options), and tests/test_medmcqa_adapter.py pins the mapping (options[answer_index] resolves to the correct text, [c.answer_index for c in cases] == [0, 2, 1]) and rejects an out-of-range cop. So "0 mismatches" isn't just a line in RESULTS.md; it's enforced by the loader and a runnable test. The raw 500-row manifest stays under data/ (gitignored, by policy), so the loader invariant plus that test is the committed proxy for the full-manifest check.

2. On stating n and the dropped cases in the summary itself: scale_c_summary.json already carries n_hard_cases: 110 and the salvage note. The one thing it can't carry is which hard cases were dropped — the attempted set only lives in the local call_cache.jsonl, which we don't commit. The clean fix is to re-run scale_c live so the numbers don't come from a cache salvage at all, which also gives the drop accounting natively; that's the plan.

[sebasmos]

Blocking, and it is the only conflict: your .gitignore replaces main's PhysioNet DUA block rather than adding to it. Keep main's four experiments/mimic_cxr_text/ lines and put your *_cache.jsonl rule alongside them. Without them the MIMIC manifests and transcripts stop being ignored, and those embed report text we cannot redistribute.

Second, please commit experiments/medmcqa/results/call_cache.jsonl, with a ! exception so your new rule does not catch it. MedMCQA is public, so there is no DUA reason to withhold it, and right now the summary says break-it C was "salvaged from the committed call_cache" while that file is not tracked, so the number cannot be reproduced.

Third, a question rather than a finding: what --target did you pass? The default is 150 and 110 rows were retained. If it was 150 the run truncated, and since rows accumulate through as_completed those 110 would be latency-ordered rather than random, which the JSON should note. If you passed 110 it completed and this is moot. I could not tell from the branch either way.

Pairing is intact regardless, all four conditions run inside one run_one, so the McNemar stands. This is provenance, not validity. The four arms are verified against the committed summaries.

Also, your per-case file corrected the paper. I had written that a second anchored voice adds nothing. Holding the anchor fixed, two peers to one drops 0.818 to 0.718 here (p=0.001) and 0.847 to 0.729 on MedQA (p=0.006), while one anchored voice ties two unanchored ones (p=1.0 on both). Not an artifact: anchored_solo differs from generic on 16 of 110 cases, so the equal totals are coincidence. The paper now states the conjunction, which is stronger. Thanks.

[EmmanuelpaulKwesiga]

The branch adds *_cache.jsonl to .gitignore but drops main's four experiments/mimic_cxr_text/ lines. Those protect MIMIC manifests and transcripts under the PhysioNet DUA and must be restored before merge.

State the --target value used. If 150 was passed and 110 rows were retained, the 40 dropped rows are latency-ordered rather than random and the summary should say so. If 110 was passed explicitly, the run completed and this is moot.

Re-running scale_c live resolves both the provenance gap and the dropped-rows accounting.

[Agastya191]

good work on the MedMCQA battery, all four arms reproduce exactly on my side from the committed artifacts alone: solo flip 0.12 / 0.237 and clean acc 0.830 / 0.770 off solo_records.jsonl, cascade +0.025 and 1/20 recomputed from the 20 _repro_shared transcripts, and scale_c's 79/90/84/79 with McNemar p=0.019 off scale_c_per_case.jsonl. One problem is that the flip-above-noise column in RESULTS.md and its Read paragraph treat the 0.067 and 0.000 noise floors as separable quantities, but run_solo measures the floor on cases[:15], so those two numbers are 1/15 and 0/15, with Wilson 95% intervals of [0.012, 0.298] and [0.000, 0.204] and Fisher p=1.0. This results in the claim that "much of flash's apparent susceptibility is decoding nondeterminism rather than the cue" resting on a single case out of 15, and it discounts a flip gap that is otherwise paired-solid (flash-lite flips on 46 case-cue pairs where flash does not, against 11 the other way, McNemar p=3.3e-06).

You have multiple options to fix this. For example, I would report the floor as its raw counts with the interval, and keep the +0.053 reading out of the Read until the floor is run at the same n=100 as the flip rate.

[Agastya191]

Still good work, duckyquang, and the arms hold up on re-checking: shared_adopt recomputes on all 20 cascade cases from the committed _repro_shared transcripts, and the +0.025 contagion and the scale_c 79/90/84/79 are exactly what the artifacts say.

One problem is the onset column in cascade_results.json. reproduce.py:264 feeds cascade_onset a bare seed-agreement series, [1.0 if t.answer == seed_answer else 0.0 for t in shared.turns], which counts the seeded turn itself, but benchmaxxing/onset.py:29 documents the input as the agreement-with-seed minus agreement-with-evidence series, which is what _case_onset in onset_distribution.py:54 builds through _cascade_series. Because the plant leaves a lone 1.0 at index 1, the series has variance even when nobody adopts, so all 17 cases where no non-seeded agent ever states the seed report onset 2 rather than censoring to None, and on d321d320, the only genuine cascade in the set, the series [0,1,1,1,1,1,1,0,0] returns 7, the turn the committee abandons the seed, when the first real adoption is turn 2. I reproduced all three straight from the committed transcripts, so the column is measuring the plant position and, on the one case that cascaded, the recovery rather than the capture.

Nothing in RESULTS.md cites onset, so no claim in this PR is wrong, and the defect is in reproduce.py which you do not touch here, so I would not block on it. It is worth fixing before #287 builds a distribution on this file, though, and you have options: I would build the series with _cascade_series the way _case_onset already does and keep the seeded turn out of it, so a case with no adoption censors the way cascade_onset intends.

[duckyquang]

@sebasmos all three fixed in 2da1c4c (rebased onto main):

• .gitignore restored main's four experiments/mimic_cxr_text/ DUA lines and kept the *_cache.jsonl rule alongside them (resolved in the rebase). MIMIC manifests and transcripts are ignored again — confirmed git check-ignore still catches them.
• Committed experiments/medmcqa/results/call_cache.jsonl (13.9 MB, public MedMCQA, no DUA) with a ! exception so the *_cache.jsonl rule doesn't catch it. break-it C's salvaged numbers now reproduce from the tracked cache.
• On --target: I can't recover the exact flag from the branch either, so the summary note now flags it honestly — 110 retained of the target, and if it was the default 150 the retained set is as_completed-ordered (latency, not random), so the marginal rates carry that selection caveat while the paired McNemar is intact. A live re-run is the clean fix.

And thanks for the anchored_solo correction — good that the per-case file sharpened the paper's claim rather than just reproducing it.

@Agastya191 right, fixed in 2da1c4c. The floor is 1/15 and 0/15 (Fisher p=1.0, Wilson intervals exactly as you gave), not commensurable with the n=100 flip rate, so the flip-above-noise column and the "mostly decoding noise" read are withdrawn until the floor is run at n=100. The real signal is the paired gap you pointed at — I reproduced it from solo_records.jsonl: flash-lite flips on 46 case-cue pairs where flash doesn't, against 11 the other way, McNemar p=3.3e-06 — and that's what the Read now leads with.

[Agastya191]

good work on this round, duckyquang: the noise-floor withdrawal is exactly what was needed, the replacement number reproduces on my side (46 lite-only flips against 11 flash-only across all 300 paired case-cue pairs in solo_records.jsonl, McNemar p=3.3e-06), and main's four PhysioNet lines are back in .gitignore intact with the cache rule alongside them. One problem is that the new provenance note in scale_c_summary.json has the target logic backwards: scale_c.py:137 breaks only on len(rows) >= args.target, so ending at 110 under the default target of 150 means the break never fired and the loop drained the whole probe set, which makes those 110 every hard case it found rather than a latency-ordered subset. The caveat therefore attaches a selection warning to the one case that provably has none, and stays silent about --target 110, which is the case where the break does fire and the retained set really is as_completed-ordered, so the four marginal rates now carry a discount in a committed artifact for an effect the code rules out. You have multiple options here, but I would record the target you actually passed, and if it was the default 150 say the 110 is every hard case in the probe set rather than keeping the caveat.

[sebasmos]

Approving. My blocking objection is resolved, and the one new thing I found is not yours.

The gitignore verdict clears. git diff origin/main -- .gitignore is +6/-0, purely additive. Main's four PhysioNet DUA lines are intact and the new rules sit after them. I checked the consequence rather than reading the diff: running git check-ignore --no-index over the union of both trees, 753 paths, under each version of the file:

• Newly un-ignored: 0. No PhysioNet file, manifest or transcript is exposed.
• Newly ignored: 43, all from the broad *_cache.jsonl glob, and all 43 are already tracked in main, so nothing is dropped.
• The !experiments/medmcqa/results/call_cache.jsonl negation works, rule ordering is right, and the file is in the tree at 1915 rows.

One residual worth a follow-up rather than a block: that glob is repo-wide, so any future cache under experiments/*/results/ is silently ignored, including under mimic_cxr_image/. A path-scoped rule would be tighter than glob-plus-negation.

All four arms re-derive from committed rows, no API calls.

• Solo, from solo_records.jsonl, 600 rows: flash flip 0.1200, clean accuracy 0.8300; flash-lite flip 0.2367, clean accuracy 0.7700. All six per-cue cells match solo_results.json exactly.
• The replacement paired claim: 300 paired case-cue pairs, 46 lite-only flips against 11 flash-only, McNemar exact p = 3.31e-06. Reproduces.
• Cascade: I recomputed shared_adopt and isolated_adopt independently from all 20 committed _repro_shared and _repro_isolated transcripts. Zero mismatches on all 20 cases. The transcripts carry seeded: true and full rationale text, so the arm is genuinely auditable, which is more than most lanes here manage.
• Break-it C, from scale_c_per_case.jsonl, 110 rows: generic 0.7182, anchored 0.8182, anchored_strong 0.7636, anchored_solo 0.7182, with anchored vs generic 15/4 p = 0.0192 and anchored vs anchored_solo 11/0 p = 0.00098. Every number in the summary reproduces.

Real API, 1915 cached responses across two tiers, genuine model prose in the transcripts. The NaN handling is honest too: solo_results.json carries cross_mean and p_value as NaN with n_permutations: 0, and RESULTS.md:52 discloses the cross-lineage arm as undefined and claims nothing from it. Suite green merged onto main, 1077 passed, including #377's board-render tripwire, so no inline renderer here.

On scale_c's provenance note, @Agastya191 is right and the note is backwards. scale_c.py:137 breaks only on len(rows) >= args.target, with --target defaulting to 150 and --probe-limit to 400. The file holds 110 rows, so the break never fired, the loop drained the whole probe set, and those 110 are every qualifying hard case rather than the first to finish caching. Latency ordering would arise only from a run invoked with --target 110 or lower, which is exactly the case the note does not mention. So the note attaches its caveat to the one scenario that provably lacks it. Your marginal rates are sounder than your own note claims. I have corrected our paper draft accordingly, which had repeated the caveat from your note.

The one new defect I found is pre-existing and I have filed it separately, at #394, rather than holding this PR on it. clean_a.py computes its control by re-completing the same base_p as the baseline, and the cache keys on (model, prompt), so control_rate is identically 0.0 and cannot fire. An adversarial mock adopting the flag on every prompt still returns 0.0. That is clean_a.py's defect, it predates both your PRs, and you are simply running it on a new cohort. benchmaxxing/prescreen_flag.py is the tested implementation with a satisfiable control.

Two small doc fixes to fold in whenever convenient, neither blocking: RESULTS.md:6-8 still says call_cache.jsonl is "gitignored, not committed" while it is now tracked and scale_c_summary.json says "now tracked".

Also stated plainly, since it bears on what this evidence is: keyless replay at zero new calls could not be verified, because the manifest is uncommitted and cache rows store only {k, model, resp} with no prompt, so keys cannot be regenerated without rebuilding a byte-identical manifest. The cache is a replay accelerator for whoever has the manifest, not an auditable artifact standing alone. The derived per-case rows are the real evidence here, and those I did verify.

Approving so this stops waiting on me. @Agastya191's change request on the note is a one-line edit and his reading of it is correct.