You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found while re-verifying #377 before merge. Merged anyway, because no committed number is affected: nothing in the repo has been run with --show-rationale. But the trap is set for the next real run, and #270, #271, #115 and #221 are all meant to go through that flag.
benchmaxxing/blackboard.py:434 gives a seeded turn with no supplied content a default:
f"[seeded] planted answer: {seed_answer!r}"
Before #377 that string was unrenderable, because the board only ever printed - agent: answer. #377's show_rationale=True path renders Turn.content, so it now reaches the holdout's prompt. Two live sites, both reproduced with a capturing backend:
benchmaxxing/runner.py:653, the CLI cascade stage, which takes the flag from config.show_rationale. With Config(show_rationale=True):
- m1: A
Reasoning line one.
Reasoning line two.
A
[seeded] planted answer: 'heparin'
experiments/medqa/reproduce.py:231, whose st = (1, seed_answer, seed_agent) carries no content:
So benchmaxxing run --stage cascade --show-rationale would not measure conformity. It would measure whether an agent copies an answer the transcript itself labels as planted, and any referee or judge that reads whole turns becomes cannot-fail by construction, since the shortcut it "infers" is printed on the board. That is the #374 family.
Suggested fix: never let the auto-generated marker render. Either keep it off Turn.content and put it on a separate non-rendered field, or have render_board skip the default marker specifically. A test should assert the flag does not leak it, since none does today.
Two adjacent ones found in the same pass, neither caused by #377:
experiments/cascade/multi_round.py:181 still renders f"- {t.answer}: {(t.content or '').strip()}" with no per-line indent, so the exact defect 7c165f4 fixed exists there again the moment that lane's board carries a live multi-line reply. It is the one renderer fix(#373): one board renderer, so a seeded peer's rationale can reach the holdout #377 deliberately exempted (tests/test_board_render.py:222).
The reintroduction guard cannot catch the 17th renderer.tests/test_board_render.py:54 matches only f"- {t.agent_id}: {t.answer}" and the answered: variant. Those two regexes hit 16 files on the pre-fix commit, and runner.py is not one of them, because its form lettered the vote: f"- {turn.agent_id}: {letters[options.index(turn.answer)]}". So the one renderer the audit nearly missed is the one the tripwire does not protect. Worth widening the pattern.
Pre-existing, same marker, different consumer.[seeded] planted answer: 'X' sits on Turn.content in every transcript, and benchmaxxing/blind_metric.py:176 (latch_rate) and benchmaxxing/referee.py:150 (_default_shortcut_detector) both read turn.content as stated reasoning. So a turn no model wrote can book a latch or a cue mention. Present before fix(#373): one board renderer, so a seeded peer's rationale can reach the holdout #377.
Found while re-verifying #377 before merge. Merged anyway, because no committed number is affected: nothing in the repo has been run with
--show-rationale. But the trap is set for the next real run, and #270, #271, #115 and #221 are all meant to go through that flag.benchmaxxing/blackboard.py:434gives a seeded turn with no supplied content a default:f"[seeded] planted answer: {seed_answer!r}"Before #377 that string was unrenderable, because the board only ever printed
- agent: answer. #377'sshow_rationale=Truepath rendersTurn.content, so it now reaches the holdout's prompt. Two live sites, both reproduced with a capturing backend:benchmaxxing/runner.py:653, the CLI cascade stage, which takes the flag fromconfig.show_rationale. WithConfig(show_rationale=True):experiments/medqa/reproduce.py:231, whosest = (1, seed_answer, seed_agent)carries no content:So
benchmaxxing run --stage cascade --show-rationalewould not measure conformity. It would measure whether an agent copies an answer the transcript itself labels as planted, and any referee or judge that reads whole turns becomes cannot-fail by construction, since the shortcut it "infers" is printed on the board. That is the #374 family.Suggested fix: never let the auto-generated marker render. Either keep it off
Turn.contentand put it on a separate non-rendered field, or haverender_boardskip the default marker specifically. A test should assert the flag does not leak it, since none does today.Two adjacent ones found in the same pass, neither caused by #377:
experiments/cascade/multi_round.py:181still rendersf"- {t.answer}: {(t.content or '').strip()}"with no per-line indent, so the exact defect7c165f4fixed exists there again the moment that lane's board carries a live multi-line reply. It is the one renderer fix(#373): one board renderer, so a seeded peer's rationale can reach the holdout #377 deliberately exempted (tests/test_board_render.py:222).tests/test_board_render.py:54matches onlyf"- {t.agent_id}: {t.answer}"and theanswered:variant. Those two regexes hit 16 files on the pre-fix commit, andrunner.pyis not one of them, because its form lettered the vote:f"- {turn.agent_id}: {letters[options.index(turn.answer)]}". So the one renderer the audit nearly missed is the one the tripwire does not protect. Worth widening the pattern.[seeded] planted answer: 'X'sits onTurn.contentin every transcript, andbenchmaxxing/blind_metric.py:176(latch_rate) andbenchmaxxing/referee.py:150(_default_shortcut_detector) both readturn.contentas stated reasoning. So a turn no model wrote can book a latch or a cue mention. Present before fix(#373): one board renderer, so a seeded peer's rationale can reach the holdout #377.Part of #374.