results(#399): honest-peer clean control for the SUPPORT2 deployable referee - #402
Conversation
Agastya191
left a comment
Agastya191
left a comment
There was a problem hiding this comment.
good work on support2_referee.py, the two-block summary is the right shape and the planted block reproduces main's committed numbers byte for byte. One problem is that the board == inferred_shortcut conjunct is vacuous on both arms, true on all 230 scored rows of support2_referee.jsonl, so deployable reduces to board != bare and referees_vs_adoption_with_clean_control is arithmetic in the same way the planted block was: precision 0.7130434782608696 is exactly the holdout's bare accuracy 82/115, fpr 0.22297297297297297 is exactly 33/148 out of n_holdout_wrong_bare, and scoring board != bare on its own reproduces the whole row. The new any(r["deployable"] != r["adopted"] for r in clean) assertion in test_deployable_referee_is_degenerate_by_construction fires whenever a single holdout was wrong bare, so it does not catch this, and the read string declares the row citable on that basis.
You have multiple options to fix this. For example, I would record the per-arm board == inferred_shortcut rate next to deployable_reduces_to_ground_truth and hold the with_clean_control deployable row back while it is 1.0, which would also keep this lane out of the three-lane agreement table, since MedQA and MedMCQA sit at 0.375 and 0.575 adoption and are the only two where 0.682 and 0.742 are not forced.
…referee (#402) Adds an arm where the peers assert the CORRECT answer, so shortcut adoption is false by construction and a deployable flag there is a genuine false positive rather than a restatement of the label. Planted-only stays reported and stays labelled degenerate: on that arm the peers are scripted to the shortcut, so `deployable` reduces to `adopted` exactly and 1.0/1.0/0.0 is forced at any adoption rate. With the clean control the deployable referee gives precision 0.713, recall 1.0 and FPR 0.223, costing 33 false alarms across 148 negatives. Two things the control does not fix, both stated in the summary rather than left for a reader to find: recall stays 1.0 because every positive is a planted row, and oracle_audit stays definitional in both blocks since its clean-arm verdict is assigned, not measured. Verified before merge: all four detectors re-derived from the 230 committed rows, arithmetic exact; `oracle == adopted` on 230/230; 0 new API calls, so the arm replays from the committed cache. 1113 passed, 7 skipped.
Adds arm 5 to the README with the gate row beside the judge row, since 0.752 quoted alone reads as a working detector and the floor on the same 115 rows is 0.713. Adds the judge block to lane_findings.json, which the README points at as the one home for the lane's numbers. Also corrects the lane's API-call total, which #402 left behind: 2,690 deterministic plus 120 noise draws is 2,810, counting that PR's 164 clean-control calls and this arm's 115 judge calls.
2 participants
Closes #399.
Third and last lane to get the honest-peer clean control, which completes the set.
Why the cell was a dash
SUPPORT2 is binary, so
plantedis the unique option other than the truth and the peers are forced to it. That makesdeployable,oracleandadoptedthe same expression, so the reported 1.0/1.0/0.0 was arithmetic.deployable == adoptedon 115 of 115 scored rows.The fix
An honest-peer arm, where both peers assert the truth. Shortcut adoption is then false by construction, so a deployable flag there is a genuine false positive: the referee has no key with which to tell honest peer correction from being lured onto a wrong answer.
Result, over planted plus clean
33 false alarms across 148 negatives. That is the highest false-positive rate of the three lanes, and it makes sense: this holdout follows peers to a shared adoption of 1.0, so it follows honest peers too, and a keyless monitor cannot tell those apart. Note that targeted and deployable now land identically, which they did not in the degenerate block.
The three lanes agree, independently
Three cohorts, three text/tabular lanes, produced separately, all landing near 0.7 precision. That consistency is worth more than any single number.
A test went red, and it was designed to
test_deployable_referee_is_degenerate_by_constructionsaid in its own docstring that if a future change ever gave the peers an answer other than the planted one it should fail, and that this would be "the point at which the deployable row may be cited again". This is that change.The test now pins both halves rather than one: the planted block is still asserted degenerate, and the clean block is asserted to be able to produce a false positive. So it goes red if either the degeneracy disclosure or the falsifiability is ever lost.
The degeneracy guard from #389 also stopped flagging this file, independently. Third time it has tracked a real fix rather than being told about one.
Verification
nandn_valid_pairsunchanged at 120 and 115. I did briefly changento the scored count while refactoring and caught it on the diff.readstring no longer claims the deployable row is not a measurement, since that is now true of only one block.1113 passed, 7 skipped.
ruffclean. No em dashes.