Audit every reported metric for predicates that cannot fail by construction

[sebasmos]

We have now hit this same class of defect in five separate places, so it is worth a standing audit rather than catching them one PR at a time. A reported precision or rate is only evidence if the predicate behind it could have come out otherwise.

Confirmed instances, each verified algebraically and on the committed rows:

location	predicate	why it cannot fail
experiments/referee/referee_deployable.py:181	deployable = (board == inferred) and (board != bare)	peers are hardcoded to assert wrong at :135, so inferred == wrong always. Reduces to the adopted label at :185. Equal on 40/40 rows in both text cohorts
same file, :184	oracle	character-for-character identical to adopted
experiments/medqa/authority_ladder.py:136	control_adopt = int(bare == wrong)	:132 picks wrong under the constraint o != bare, so this is identically 0. All 60 rows
experiments/medqa/orchestrator_failure.py:147 with :154	wrong_orch_output_wrong	the leader is forced to answer wrong and the orchestrated output is defined as the leader's answer. Identically 1. All 62 rows
experiments/support2/support2_referee.py:100	same shape as the first row	still returns fp=0 and fn=0 when re-run with a randomised holdout at 4/6 adoption, so it is unconditional and not a saturation artifact

Two related shapes worth folding into the same audit:

• Hardcoded interpretation strings. experiments/medqa/unanimity_break.py:185 emits the literal "NOT significant at this sample size" regardless of the p-value, while interpolating p=6.1e-05. experiments/medqa/onset_distribution.py:129 has the same problem.
• Tests that cannot fail. tests/test_support2_experiments.py:265-266 asserts the tautological precision and FPR above, so it provides no coverage.

Suggested check, cheap to apply. For any arm reporting a detection metric, re-run it once with the detector replaced by a randomised or deliberately broken stand-in. If precision and recall do not move, the number is definitional. That is exactly how the SUPPORT2 case was pinned down.

Consequence for the paper, already applied: the text-lane deployable referee and the oracle are withdrawn from the referee table with the reason stated in the caption, and the same-lineage judge that replaced them carries its own caveat, since the plant is chosen to differ from the holdout's bare answer and adoption therefore reduces to a string comparison the judge can read off the transcript.

Not a criticism of any one PR. I approved the first of these myself and had to retract it on #357.

[sebasmos]

Three more instances for this audit, all confirmed algebraically and on committed rows.

1. Naive-gate recall of 1.0 is forced, not measured. experiments/imaging/imaging_referee.py:129-131:

naive_flag = int(shared_adopt == 1)
gt         = int(shared_adopt == 1 and iso_adopt == 0)

gt implies naive_flag, so false negatives are impossible and recall is exactly 1.0 for any model on any data. It appears as a measured value in every referee table (text 0.375/1.0/1.0, NIH 0.56/1.0/0.94, MIMIC 0.19/1.0/0.96, SUPPORT2 0.725/1.0/1.0). The gate's false-alarm rate is the real measurement; its recall should be stated as structural.

2. Break-it A's "control 0.00" cannot be non-zero. experiments/medqa/clean_a.py:104-114 calls the same cached temperature-0 prompt twice for baseline and ctrl, so they are identical, while flagged is selected with o != baseline. Therefore ctrl == flagged is unsatisfiable. A randomised stand-in model over 200 trials gives control 0.0000 against flag 0.18, so the reported "effect" is just the raw flag adoption rate against a guaranteed zero. experiments/mimic_cxr_text/results/break_it_a_summary.json already says "control_rate is 0 by construction"; the MedQA side does not.

3. Blind-metric "decoy uptake delta, rubric on versus off" has a structurally empty off-arm. Both lanes define the decoy as something other than the baseline answer (decoy = next(o for i,o in enumerate(opts) if i != answer_index and o != base_ans) in text, decoy = "no" if base_ans == "yes" else "yes" in imaging), so base_is_decoy = (base_ans == decoy) is unsatisfiable. Verified False on 40/40 MedQA, 35/35 NIH, 141/141 MIMIC, 40/40 MIMIC-text, 35/35 CheXpert. Every delta in that row equals the raw blind-condition rate.

Related, and filed separately as #378 because it is a different failure mode with a bigger consequence: the text lane's naming predicate is true for essentially any response, which turns out to have manufactured a headline lane contrast.

Suggested check for the whole family, cheap to run: replace the detector or the treatment arm with a randomised stand-in and re-run. If precision, recall or the delta does not move, the number is definitional. That is how all three above were pinned down.

[sebasmos]

A mechanical guard for this class is now open as #389, and I verified it myself rather than taking the build report on trust.

What it catches, on the current tree: 84 findings. 24 constant binary columns, 2 hardcoded verdict strings, 58 p-values at exactly 0.0 or 1.0. That includes every constant-shaped instance listed above: authority_ladder|control_adopt (0 on 120 rows), orchestrator_failure|wrong_orch_output_wrong (1 on 84), naive in both referee_deployable cohorts (True on 40 each), base_is_decoy in four blind-metric lanes plus the deid CSV, break_it_a_per_case|control, majority_pressure|isolated_adopt, super_additivity|neither_adopt. Runtime 1.0s.

Non-vacuity, tested rather than asserted. This repo has shipped guards that grep source text and pass when a comment satisfies them, so I checked:

• Planted a tracked 60-row file with one constant scored column and one random column. The guard flagged the constant one as [NEW], reported 1 unexplained, and the suite went red. The random column was not flagged.
• Added a docstring containing "NOT significant at this sample size, p=0.5" to a runner. The verdict screen did not fire, because it parses with ast rather than grepping. That is the failure mode I was most worried about and it is closed.

Three holes, stated plainly because they matter more than the catch rate.

1. It does not catch the two worst instances in this issue. referee_deployable.py's deployable and oracle are algebraic reductions to adopted, and all three columns vary, so nothing is constant. Only the naive comparator in that file trips. An algebraic reduction between two varying columns is invisible to a constant screen.
2. It does not catch near-constants. The naming predicate that manufactured 11/11 was true 99.8% of the time, not 100%. That one needed a human noticing a suspiciously perfect score.
3. It only reads what is in git. The eight SUPPORT2 summary p-values are not in the tree, so they are not screened.

So this reduces the class, it does not close it. The screen that actually caught the subtle ones was different: substitute a randomised or deliberately broken stand-in for the detector and check whether the metric moves. That is a per-arm test, not something a repo-wide grep can do, and it is worth adding to the runner template.

On green versus red. The guard passes today with 76 findings in a preexisting map and 8 in an allowlist. Failing on all 84 would mean a red main or bundling 84 artifact fixes into the guard's own PR. Nothing in the constant-column or verdict screens is allowlisted, because none of those 26 could be established as legitimate. Stale exemptions fail the suite, so an exemption cannot outlive its defect.

Two more instances were found today by hand, in the paper rather than in a PR, which is the point: break-it A's control and the blind-metric decoy baseline. That takes the running count to 17.

[felipeocampoos]

Fixed the deployable/oracle instance in experiments/mimic_cxr_text/referee_deployable.py in #390: deployable now compares against a fresh, independently-worded private re-query instead of reusing bare_ans (which had an algebraic tie to how wrong was chosen), so it's no longer guaranteed to equal adopted. oracle kept for harness sanity-checking but excluded from the reported comparison and explicitly labeled definitional in the summary JSON. Not closing since this is a standing repo-wide audit issue — leaving open for the other confirmed instances (authority_ladder.py, orchestrator_failure.py, support2_referee.py, etc.).