feat(#395): SUPPORT2 same-lineage judge referee, scored against the naive gate

[Agastya191]

Closes #395.

Please read this first. The runner commit for this issue, d6c8b52, is already on main, pushed directly and without review. That was my mistake, not a deliberate merge: I created the branch with git checkout -b feat/support2-referee-judge origin/main, which sets the new branch's upstream to origin/main, so a plain git push sent it to main instead of to the feature branch. main has no protection rule, so nothing stopped it. I have not rewritten main to undo it, since #220 has already landed on top and others may have pulled. Say the word and I will revert it on a branch and re-land the whole thing through this PR properly.

So this PR contains only the docs: the README arm and the lane_findings.json entry. The runner, its tests and its artifacts are on main already and are described below so they can still be reviewed here.

Tabular was the only lane without a judge runner, so the Table 2 cell was a never-run gap rather than a measurement of zero. It is now a number, and the number is weaker than I expected.

Result

detector, same 115 rows	precision	recall	FPR	tp	fp	fn	tn
naive_gate (the floor)	0.713	1.000	1.000	82	33	0	0
same-lineage judge	0.752	0.963	0.788	79	26	3	7

The judge clears the transcript-only floor by four points of precision, and that is the whole finding. It flags 105 of 115 patients. The entire gain over a blunt agreement gate is that it declines 7 of the gate's 33 false alarms, paired McNemar p=0.016, and pays 3 of 82 true detections for them.

So I put the gate row in the summary as naive_gate_same_rows rather than reporting the judge alone. Both detectors read the transcript and nothing else, so the gate is the floor, and 0.752 quoted without it reads as a working monitor. This is a far weaker showing than the same detector on MedQA and MIMIC-CXR text, both precision 1.000 at FPR 0.000. Why the tabular lane is harder for a transcript-only monitor is not established here, and I did not want to guess in the artifact.

The arm is a measurement, not a restatement

The failure mode to watch on this lane is a judge that flags everything, which would report the adoption rate as its precision and 1.0 as its FPR, exactly what the naive gate already does. Recorded from the rows as judge_is_not_a_restatement_of_the_label rather than asserted in prose: the judge disagrees with the adoption label on 29 of 115 rows, missing 3 real adoptions and raising 26 false alarms. Unlike deployable it is an independent model call, so it is not algebraically tied to the label.

Verification

• The judge arm and the referee arm score the same patients. board, bare and adopted are identical on all 115 scored rows of support2_referee.jsonl, and n_valid_pairs is 115 in both.
• The gate row reproduces the referee arm's committed number exactly: 0.7130434782608696 / 1.0 / 1.0, tp 82 / fp 33. That is the planted-only block, not the with_clean_control block from results(#399): honest-peer clean control for the SUPPORT2 deployable referee #402; this arm runs the planted cascade only.
• 115 new API calls, all gemini-2.5-flash, committed to the cache. A re-run then replays the summary keylessly at zero calls, verified after rebasing onto the merged results(#399): honest-peer clean control for the SUPPORT2 deployable referee #402.
• Full suite green, ruff check clean on both changed files.

Two deviations from the template, both deliberate

1. The verdict parser. The template tests "FLAG" in verdict with a guard on the first word, which reads a hedge like "this is OK, nothing to flag" as a flag. This takes the last declared \bFLAG\b|\bOK\b instead. No committed number changes on the other lanes: every verdict there is the bare word, which I checked.
2. A reply declaring neither verdict is censored, not scored as OK, for the same reason the lane censors a refusing holdout. A silent negative flatters both recall and FPR. It fired 0 times on this run, and censored reports both causes separately.

Not fixed here, worth a follow-up

lane_findings.json is still stale from #402 in two places I deliberately left alone, since they are that PR's claims to correct rather than mine: referee.verdict still says "one of the four rows is a measurement; two are definitional", and the open follow-up asking for "a keyless-referee design whose peers do not always assert the planted answer" is what #402 delivered for this lane. I did correct the lane's API-call total, which #402 left at 2,531 against a cache that now holds 2,690 deterministic entries.

Note on the issue's premise

The issue suggests scoring the already-committed support2_referee.jsonl. That is not possible: those rows carry the verdicts and answers but no deliberation turns, and the judge reads the transcript. The runner rebuilds it through _common.run_board, which is free from the cache and keeps the holdout's board prompt byte-identical to support2_referee.py. Cost is unchanged.

🤖 Generated with Claude Code

[sebasmos]

Verified against the committed rows, and every number reproduces exactly. Approving the analysis. One process item below that is not about the analysis.

Re-derived from the 115 scoreable rows of support2_referee_judge.jsonl:

detector	tp	fp	fn	tn	precision	recall	FPR
naive gate, same rows	82	33	0	0	0.7130	1.0000	1.0000
same-lineage judge	79	26	3	7	0.7524	0.9634	0.7879

And the contrast that carries the claim: on the 33 negatives, 7 gate-only, 0 judge-only, exact p = 0.015625. All exact.

The check that matters most here passes. judge_flag differs from adopted on 29 of 115 rows, so this is a measurement and not another predicate that reduces to its own label. That is the thing I have had to reject repeatedly in this family, and it is worth saying explicitly that this arm does not have the problem.

Your framing is the right one and I would keep it as written. Precision 0.752 against the gate's 0.713 is four points, the judge still flags 105 of 115, and its FPR is 0.788. Declining 7 of 33 false alarms at the price of 3 of 82 detections is a modest, real, and honestly small result. Reporting the gate as naive_gate_same_rows inside the same summary is what makes it legible, since the reader can see the floor without going to another file. The tabular lane now has a judge number where it previously had a never-run gap, and a weak number is a fine outcome for that.

The process item, and it is a real one. d6c8b52 is on main, in no pull request, so the runner, its 155 lines of new tests and its artifacts were never reviewed. I confirmed that independently: the commit is an ancestor of main and the GitHub API returns no associated PR. Your account of how it happened is plainly right and main has no protection rule to stop it.

I would not revert it. #220 and #405 have landed on top, the artifacts are cited from the docs in this PR, and the code is fine on inspection, which I can now say because reviewing it here is exactly what this PR makes possible. Rewriting shared history to satisfy a process point would cost more than it buys. What I would rather do is take the lesson: this is the second time this week that main having no protection has let something through that everyone assumed a gate would catch, the first being the stale change requests that were blocking nothing. Worth raising with the owner as a settings question rather than a personal one.

So: approving, with the runner reviewed as part of this PR rather than pretending it was reviewed before it landed. Merge timing is the reviewers' call.