You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Table 1 of the paper compares four break-it channels across three text cohorts. Channel B, deliberation framing, is the one blank cell left in that table, and it is blank on MIMIC-CXR text only:
Channel
MedQA
MedMCQA
MIMIC-CXR text
A. system pre-screen flag
0.26/0.30
+0.27/+0.53
+0.40/+0.50
B. deliberation framing
0.64 to 0.12
0.74 to 0.28
blank
C. vignette-anchored seed
+0.12
+0.10
+0.03 (n.s.)
D. hidden rubric
0.24 to 0.24
0.31 to 0.31
0.00 to 0.05
This is a never-run gap, not a null.git ls-files experiments/mimic_cxr_text/ has no framing runner and no framing result; MedQA has experiments/medqa/deliberation_framing.py and MedMCQA has committed deliberation_framing_summary.json. Every other cell in that table is now filled.
Why it is worth running
B is the only protective channel in the table: on both cohorts where it has run, instructions that license dissent lower adoption rather than raise it, and by a lot. MedQA goes 0.64 unframed to 0.12 critical, MedMCQA 0.74 to 0.28, with every level significant against unframed at $p \leq 1.1 \times 10^{-4}$.
That makes it the paper's one cheap mitigation result, and it currently rests on two text cohorts of the same MCQ shape. MIMIC-CXR text is the cohort that would show whether the mitigation survives a different provenance, since its items are derived from real radiology reports rather than exam questions. If framing does not protect there, that is a more interesting finding than if it does.
Suggested approach
experiments/medqa/deliberation_framing.py is the template, and MedMCQA was ported from it, so this should be the same shape of change as your #357 referee port:
Point the runner at the MIMIC-CXR text manifest and write to experiments/mimic_cxr_text/results/.
Four conditions: unframed, collaborative, independent, critical, on the same hard-case cohort the other MIMIC text arms use.
Report adoption per condition plus exact McNemar for each level against unframed, matching the two existing cohorts so the row is comparable.
Commit the summary, the per-case rows and the cache, so it replays keylessly.
Roughly 4 conditions times the cohort size in calls, so cheap relative to a full battery.
Related
Channel C on this cohort is already in and does not replicate the anchoring increment (+0.03, McNemar 3/1, $p=0.625$), though dropping to one anchored peer still costs 0.15 ($p=0.004$). Filled from your committed push_c_summary.json.
Channel D is in as a raw rate, from blind_metric_summary.json, baseline 0.00 to blind 0.05.
The gap
Table 1 of the paper compares four break-it channels across three text cohorts. Channel B, deliberation framing, is the one blank cell left in that table, and it is blank on MIMIC-CXR text only:
This is a never-run gap, not a null.
git ls-files experiments/mimic_cxr_text/has no framing runner and no framing result; MedQA hasexperiments/medqa/deliberation_framing.pyand MedMCQA has committeddeliberation_framing_summary.json. Every other cell in that table is now filled.Why it is worth running
B is the only protective channel in the table: on both cohorts where it has run, instructions that license dissent lower adoption rather than raise it, and by a lot. MedQA goes 0.64 unframed to 0.12 critical, MedMCQA 0.74 to 0.28, with every level significant against unframed at$p \leq 1.1 \times 10^{-4}$ .
That makes it the paper's one cheap mitigation result, and it currently rests on two text cohorts of the same MCQ shape. MIMIC-CXR text is the cohort that would show whether the mitigation survives a different provenance, since its items are derived from real radiology reports rather than exam questions. If framing does not protect there, that is a more interesting finding than if it does.
Suggested approach
experiments/medqa/deliberation_framing.pyis the template, and MedMCQA was ported from it, so this should be the same shape of change as your #357 referee port:experiments/mimic_cxr_text/results/.Roughly 4 conditions times the cohort size in calls, so cheap relative to a full battery.
Related
push_c_summary.json.blind_metric_summary.json, baseline 0.00 to blind 0.05.