results(#297): SUPPORT2 tabular lane, real run on n=120 (solo null, cascade at ceiling)

[Agastya191]

Closes #297. The replication half of the SUPPORT2 tabular lane, run against the manifest #354 committed.

gemini-2.5-flash-lite, the pinned 120-case balanced set, 1,266 new API calls (solo 960, cascade 306, referee 0). Clean accuracy 0.683.

Headline: the lane does not reproduce the other two lanes' shape. Solo is a null, the cascade is pinned at the ceiling, and the referee ranking is entailed by that ceiling rather than measured against it. I have written it up that way rather than presenting three green arms.

1. Solo: a null

cue	flip rate	minus floor	McNemar p
field_order	0.258	+0.017	0.86
unit_rescale	0.308	+0.067	0.22
precision_inflation	0.183	-0.058	0.25
redundant_restatement	0.283	+0.042	0.49
missingness_recode	0.200	-0.042	0.36
administrative_hint	0.233	-0.008	1.00

Noise floor 0.242. Nothing separable in either direction. All six cues fired on all 120 patients, so this is not a skipped-record power problem.

Two caveats change how it should be read, and both are in the README:

The floor is not a matched control. Cue contrasts are temperature 0 against temperature 0, so both reads are deterministic and any flip is caused by the rewrite. The floor is temperature 0 against temperature 1.0, which measures sampling noise in a different decoding regime. Subtracting them is not like-for-like, and it biases toward zero: temperature-0 instability is 0 by construction. What the lane needs is a temperature-0 surface-form null cue, a rewrite that changes bytes without changing anything the hypothesis says should matter. It does not exist yet. Until it does, flip_above_noise is a conservative bound, not the effect.

At face value the raw rates are the finding. Information-identical rewrites move the prognosis on 18% to 31% of patients at temperature 0, and field_order, which only reorders lines and is the most meaning-free perturbation available, sits mid-range at 0.258. So the supportable claim is not "cue X beats cue Y" but "roughly a quarter of this model's tabular prognoses are arbitrary with respect to surface form, and no cue is special." Weaker than the imaging and text lanes, and a different claim.

Context for the accuracy number: the model says "Dies in hospital" 88 times against 27 "Survives" on a set balanced 60/60, so 0.683 is a mortality prior more than calibrated prognosis.

2. Cascade: saturated

arm	n	eligible	isolated	shared	contagion	McNemar
wrong_seed	120	87	0.275	1.000	0.725	87/0, p<1e-6
flip_seed	115	115	0.000	1.000	1.000	115/0, p<1e-6

235 of 235 board runs adopted the seed. Zero resistance.

An exact 1.0 is usually plumbing, so I checked it rather than reporting it. On the first case the holdout replies A bare (correct) and B with two peers asserting B, and the whole arm re-scores to 120 adopted / 0 resisted under the centralized parser. It is real.

It is also a dead end for measurement. The polarity contrast the second arm exists to provide cannot answer its own question: a confidently wrong peer pair recruits at 1.000 and a confidently right one also recruits at 1.000. Any effect size here is a floor on the manipulation's strength, not a measure of persuadability. Getting off the ceiling needs a weaker manipulation: one peer instead of two, lower stated confidence, or a stronger holdout.

Contributing design note: in _common.mcq_prompt the peers' answers land immediately before "Answer with only the single letter of the best option", so the seeded answer is the last thing read before the instruction to answer. Close to maximally leading for a lite model on a binary question.

3. Referee: the ranking reproduces, the ceiling forces it

referee	precision	recall	FPR	extra calls
naive_gate	0.725	1.000	1.000	0
targeted	0.725	1.000	1.000	0
deployable	1.000	1.000	0.000	120
oracle_audit	1.000	1.000	0.000	120

Qualitatively the other lanes' oversight result holds: the monitor that buys one counterfactual re-query separates adoption from independent error, and the ones that do not cannot, even holding the planted key.

But the arithmetic is forced, and this corrects a claim in #354's own description. Because adoption is 100%, board == planted for all 120, so targeted fires on all 120 and is identical to naive_gate; and deployable's predicate board == inferred and board != bare reduces to exactly the scoring definition, making it identical to oracle_audit. Four referees collapse into two tiers, both entailed by the ceiling. #354 argued targeted would be "a genuinely distinct referee here" giving "an informative ranking rather than one trivially perfect row." It is trivially determined after all, just for a different reason than the imaging lane's. The 0.725 is the 87/120 base rate of true adoption, nothing more.

Reproducibility

results/call_cache.jsonl holds all 1,146 deterministic responses; solo and cascade replay from it with zero calls and no key. Referee already reported new_api_calls_this_run: 0 on the original run, since its board prompts are byte-identical to the cascade's wrong_seed arm.

One number does not reproduce, by design. The noise floor is an uncached temperature-1.0 resample, so Cache.complete_uncached never persists it and a re-run recomputes a different 0.242. The per-case resampled answers in results/support2_solo.jsonl are the record. Persisting the raw resample text so the floor is auditable rather than merely recorded is a worthwhile follow-up; #365 flagged the same class of problem for MedQA.

Parser sensitivity

26% of cached replies are not bare letters: 13% carry a LaTeX answer box, 13% are prose, much of it hedging. Rather than assert the parser is fine, experiments/support2/parser_sensitivity.py re-scores the entire cache with benchmaxxing.extract.parse_mcq_choice, offline and at zero cost, and writes results/support2_parser_sensitivity.json.

Result: the parsers agree on 96.9%, clean accuracy is identical at 0.683, every flip rate moves down 0.04 to 0.08 (the centralized parser censors abstentions the lane parser scores as flips, so the null gets more null), and the cascade ceiling is unchanged.

So no conclusion turns on the parser. The lane should still migrate, because scoring a refusal as a prognosis is wrong on its own terms, but that touches Abstention handling in all three runners and belongs in its own PR rather than riding along with results.

status.py

solo, cascade, referee to done. plausibility to not applicable, not pending: the plausibility arm is the plausible-vs-implausible distractor contrast, and a binary outcome has exactly one wrong option, so there is no gradient to vary and that field could never have reached done. Imaging already carries not applicable for the same structural reason. Thanks @sebasmos for adding the entry in #360; this is the one field you could not have known about without the lane.

Reviewer attention

• The two ceiling caveats are the substance of this PR. If you read only the tables, this looks like two green arms and one null. It is closer to one null and two arms that cannot discriminate at this manipulation strength.
• No CI exists in this repo (no .github/workflows/), so "69 passed, ruff clean" below is a self-report, same as every other PR here. Worth fixing separately. Unrelated to this branch: main currently has a real ruff I001 in benchmaxxing/datasets/status.py that predates it.
• Follow-ups I did not do: the parser migration, the temperature-0 surface-form null cue, persisting noise resamples, and a weaker cascade manipulation to get off the ceiling.

Testing

69 passed: test_support2_adapter.py, test_tabular_cues.py, test_support2_experiments.py, plus test_datasets.py and test_staging.py for the registry/status contract. ruff check clean on every file this PR touches.

---

Amendments after @sebasmos's review (33e6f55)

Both items were claim-level, as you said, and neither moves a number. Both summaries regenerate from the committed cache at 0 new API calls and every figure is identical.

The deployable referee row, and I had the cause wrong. Section 3 above says the collapse is forced "because adoption is 100%". That is right for targeted into naive_gate and wrong for deployable into oracle_audit. The peers are scripted to assert the planted answer, so inferred_shortcut == planted identically and board == inferred and board != bare reduces algebraically to board == planted and bare != planted, which is the adopted label it is scored against. Precision 1.000 with FPR 0.000 is forced at every adoption rate, which is what your randomised holdout at 4/6 adoption demonstrated. The README now separates the two collapses and the reduction is recorded from the rows as deployable_reduces_to_ground_truth in the summary, so a reader can check it rather than take the note on trust. Given you have withdrawn the corresponding number from the text lane, the README also flags that the pattern needs checking wherever a keyless referee infers the shortcut from peers scripted to assert it.

The board shows bare answers. Confirmed. run_board computes the case-anchored rationale, attaches it to the transcript, and renders only - agent: answer into the holdout's prompt. The seed_style field claimed a case-anchored reasoned seed and was false; it now reads answer_only, and section 2's description says what the holdout actually sees. As you note this is the stronger finding, 235 of 235 holdouts moving a mortality prognosis on two unargued votes, but it is a different one, so the write-up says that rather than keeping the argued-endorsement reading. #369 fixes the renderer and runs the 2x3 strength ladder on top of it; the ladder does not lift the ceiling.

So the two tables in sections 2 and 3 stand numerically. What changed is what they are claims about.

[Agastya191]

Correction to this PR, found while doing its own follow-ups. Fixed in #369, which is stacked on this branch.

The seed_style label in the committed cascade summary is false. It says "reasoned (case-anchored rationale quoted from the patient's own record)". It was not. _common.run_board computed that rationale, attached it to the transcript, and then dropped it: only - agent: answer was ever rendered into the holdout's prompt. I confirmed it by diffing the stimuli, and a confident rationale and a hedged one produce byte-identical prompts.

So the cascade and referee arms here measure adoption of bare votes with no reasoning attached. That is a stronger result than the one I described, not a weaker one, but it is a different one, and #354's description made the same claim and quoted the anchored text. No number in this PR moves; the numbers were always valid and only the label was wrong.

Knock-on effects, all in #369:

• The stated-confidence knob could not have done anything, since it never reached the prompt.
• case_anchor and seed_rationale were dead code in the stimulus path, so dc57e02 (Cascade: shortcut-REASONING seed (a voiceable rationale, not a bare answer) #115, "cue-anchored reasoned seed style for the cascade") is a no-op for this lane. Other lanes that build their own holdout prompt from transcript turns may have the same defect. I have not checked and am not claiming they do.

Two other things in #369 that change how this PR reads, so reviewing them together is probably easier than reviewing this one on its own:

• The solo null was measured against the wrong comparator. The temperature-1.0 floor of 0.242 answers a different question from a temperature-0 cue contrast, and it was additionally inflated by counting refusals as flips. Against a proper temperature-0 whitespace control (0.140), unit_rescale and field_order clear it nominally. Nothing survives BH across the six-cue family, so the null stands at family level, but three of the six cues turn out to have been tested at power 0.03 to 0.09, meaning they were never evidence of absence. Also: I informally described field_order here as the most meaning-free perturbation available; the real control shows that was wrong.
• The "weaker manipulation" follow-up I proposed is answered, and it was the wrong lever. A full 2x3 over peer count and board style moves adoption only between 0.976 and 1.000, non-monotone, nothing significant. The ceiling is this model on this task, not the stimulus.

[sebasmos]

Edited in place after a deeper re-verification on 33e6f55. Three sentences of mine below were wrong and are corrected here rather than in a second comment. The two claim-level problems I raised are unchanged and have both been fixed by the author.

Verified this against the artifacts rather than the summary. On 33e6f55, with the committed cache copied to scratch and GEMINI_API_KEY unset: support2_cascade.jsonl, support2_cascade_summary.json, support2_referee.jsonl, support2_referee_summary.json and support2_parser_sensitivity.json all regenerate diff-identical at zero new calls, and the cache does not grow. All six solo p-values, all 16 referee cells, and the whole call ledger recompute from the raw rows: 1146 cached responses = 120 clean + 720 cue + 306 cascade, and 306 = 153 distinct boards x 2 rounds. Real run on real data, cache tracked, nothing gitignored.

Corrections to my own earlier text, all mine, none the author's:

1. I wrote that "weak-alone now holding on tabular data makes this a genuine third lane" and said I had put it in the full draft. That was wrong and I have removed it. The solo arm perturbs the record's surface form; the cascade arm plants peer votes on the clean record and injects no cue at all (_common.run_board builds the holdout prompt from case.question with only the peer board prepended, and Condition.CONTAMINATED is a label that injects nothing). The two arms manipulate different things, so the weak-alone / strong-in-committee conjunction was never measured here. The lane contributes a cue-design result and a peer-adoption result separately.
2. I credited "the BH adjustment" to this PR. There is none in it. The six p-values do recompute exactly; the Benjamini-Hochberg work is in #369.
3. I said "all four runners" regenerate byte-identical with no key. Three do. support2_solo.jsonl and its summary cannot, because the committed versions carry noise-floor fields that need a key. The PR body says this plainly and my review overstated it.

1. The deployable referee here is not a measurement. Unchanged, and now fixed. The peers are scripted to assert the planted answer, so inferred is that answer by construction and the flag reduces algebraically to the adopted label it is scored against. I re-ran the arm with a randomised holdout at 4/6 adoption, far off saturation, and it still reported no false positives and no misses, so the reduction is unconditional rather than a ceiling artifact. deployable_reduces_to_ground_truth now records this from the rows, and I confirm both flags on the real data: inferred_shortcut == planted on 120/120 and deployable == adopted on 120/120.

2. The committee arm was described as an argued endorsement when the board showed bare votes. Unchanged, and now fixed: seed_style reads answer_only, which is the honest label.

One new problem, and it is the same class this PR just corrected one arm for.

support2_cascade.py:91 picks the flip seed as next(o for o in options if o != bare). On a binary task that is the unique other option, so bare_adopt = int(bare == seed) is 0 for every case unconditionally, which I confirmed on all 115 rows. Four of the five numbers in README.md:219 are then restatements of one: isolated 0.000 is algebra, contagion equals shared identically, n_eligible == n is forced, and lose is 0 so mcnemar(gain, 0) can only ever point one way. At an adoption rate as low as 10/115 it would still return p about 0.002. The README discloses only the weakest of these ("eligible everywhere by construction"). Anyone quoting flip-seed's contagion or p-value as evidence of persuasion is quoting arithmetic.

Two smaller ones from the same check:

• The eligible pool is 87, not the 82 correct-bare that README.md:124 and the support2_cascade.py docstring both claim. The extra five are support2-12, -23, -29, -31 and -54, where the holdout refused in 465 to 2896 characters and parse_choice returned the raw response text, making "bare differs from seed" trivially true. Those five have no defined counterfactual. This file already censors the flip-seed arm for exactly this condition and does not censor this one.
• "235 of 235 board runs" is 153. Where the holdout is right bare, flip-seed's seed equals wrong-seed's seed, so the prompt is byte-identical and cache-served. 82 of the 115 flip-seed rows are the same case, same seed, same board answer as their wrong-seed row. That is what makes the 306 call count work.

Also worth a follow-up rather than a block: tests/test_support2_experiments.py:132 asserts deployable["fpr"] == 0.0 and precision == 1.0, pinning the degenerate outcome as expected behaviour, and deployable_reduces_to_ground_truth has zero test coverage. I replaced "deployable": adopted in the source and 1049 tests still passed with a byte-identical referee block, so nothing in the suite can tell the referee from the label.

The run stands and I could not break a single headline number. The flip-seed table needs labelling before it can be cited.

[Agastya191]

Thanks for verifying this against the artifacts rather than the summary, and for the randomised-holdout check in particular. Both corrections are in 33e6f55; neither moves a number and both summaries regenerate from the cache at 0 new calls.

You are right that I had the cause of the deployable collapse wrong, and the way I had it wrong made it look milder than it is. Attributing it to the 100% ceiling implies it would go away at a lower adoption rate. It does not: the peers are scripted to assert planted, so inferred_shortcut == planted identically and board == inferred and board != bare reduces to board == planted and bare != planted, which is the adopted label it is scored against. Your 4/6 run is the demonstration. The README now separates the two collapses, since targeted into naive_gate really is the ceiling and does reverse, while this one never does, and the reduction is recorded from the rows as deployable_reduces_to_ground_truth so it is checkable instead of asserted. Given you have withdrawn the text lane's number, I have flagged in the README that the pattern needs checking anywhere a keyless referee infers the shortcut from peers scripted to assert it, rather than treating it as a SUPPORT2 quirk.

The dropped rationale is confirmed and seed_style now reads answer_only instead of claiming a case-anchored reasoned seed. Your framing is the right one and I have used it: this is the stronger result, 235 of 235 holdouts moving a mortality prognosis on two unargued votes, but it is a different result, so the write-up says that rather than keeping the argued-endorsement reading and quietly hoping the renderer catches up. #369 fixes the renderer and answers the follow-up this arm suggested, which was a weaker manipulation: the 2x3 ladder does not lift the ceiling, so the lever is the holdout model rather than the stimulus.

One thing worth flagging since you are putting this in the paper draft. new_api_calls_this_run in the cascade summary now reads 0 rather than 306, because regenerating from cache is what the field measures. The 306 is still in the README's call attribution, but if you want the original cost auditable from the artifact rather than the prose, that is a real gap and I am happy to add a separate provenance field for it.