feat: benchmaxxing run entry point for the stage runners

[duckyquang]

Summary

Added benchmaxxing run, one command that turns a config plus a manifest into a run directory:

benchmaxxing run --stage {pilot,solo,overlap,cascade} --manifest m.csv --config cfg.json --out runs/x
benchmaxxing run --stage cascade --manifest m.csv --out runs/x --dry-run
benchmaxxing run --stage solo --manifest m.csv --out runs/x --backend mock

The stage logic stays in experiments.py. The new benchmaxxing/runner.py is just the wiring: resolve the config and roster, load cases, inject cues, call the matching runner, write results.json, summary.md, config.json and run_manifest.json. The cascade stage also saves every shared and isolated transcript under --out/transcripts/ through the existing RunStore.

Two small adapters are the only new behaviour, and they are both plumbing. SoloAgent renders a twin payload as a lettered MCQ and parses the reply back into an option string (solo_evaluate prefers a run(payload) backend, which is the hook that lets the parse see the twin's own options). CommitteeAgent does the same for a blackboard turn with the visible board appended. I deliberately did not touch prompt design, that's #107, so these render the same shape the first real MedQA runs used.

Result

All four stages run offline with --backend mock on a 4-case synthetic manifest:

• pilot: flip rate 0.33 over 12 twins, cues bite
• solo: clean accuracy 1.00, contaminated 0.67, shortcut reliance 0.33
• overlap: adds the within vs cross lineage overlap and its permutation p-value
• cascade (3-model roster): shared adoption 1.00 vs isolated 0.00, so the shared-vs-isolated contrast shows up end to end with no key
• --dry-run on the cascade prints models, lineages, 4 cases / 12 twins, 64 estimated calls, and writes nothing

ruff check . clean, pytest -q 594 passed. New tests drive each stage through the CLI and assert the four output files exist and parse.

Notes

Three things worth flagging:

• Lineage is inferred from the model id (gemini-2.5-flash to gemini, qwen2.5-72b-instruct to qwen) because Config has no lineage field. It's what the overlap arm splits on, so if you'd rather have it explicit in the config, say so and I'll add the field.
• I taught load_config to read .json as well as .yaml. YAML needs the optional config extra, and it felt wrong for the main entry point to require an extra just to pass a roster. YAML still works exactly as before.
• extract.parse_mcq_choice only takes the letter branch when the options themselves are single letters, so with real MedQA-style options a reply of "The answer is B" comes back unparseable. I work around it by parsing against the letters first and falling back to the option text. That's the unification the TODO in extract.py (Answer extraction and abstention normalization for real model outputs #102) is about, so I left the parser alone.

The imaging lane raises with a pointer to experiments/imaging rather than pretending: it needs the images loaded and injected as arrays, which is a bigger piece of work.

Closes #101

[sebasmos]

Approving for now, solid plumbing. But let's shift effort from CLI and infra to actual experiments on real data. Park further runner polish until the paper's runs are in.

[duckyquang]

Status update on this one, since a stack has grown on top of it.

A real-model run is still outstanding. Everything here has been exercised against --backend mock only. I don't have a GEMINI_API_KEY in this environment, so I can't produce the "unchanged against a real Gemini roster" half of the acceptance criteria. Whoever has a key can settle it cheaply:

export GEMINI_API_KEY=...
benchmaxxing run --stage pilot --manifest <medqa manifest> --limit 20 --out runs/first-real

That is roughly 40 calls on flash. If it produces a sane summary.md this PR is done; if the parse or the prompt shape is off, better to find out on 20 cases than on 200.

Follow-ups now stacked on this branch, in merge order:

• feat: bootstrap CIs and effect sizes in the run summary #252 (Analysis: report bootstrap CIs + effect sizes in the run summary #127) bootstrap CIs and effect sizes
• feat: correct the whole family of p-values in the run report #253 (Analysis: multiple-comparison correction across arms in the report #128) family-wide multiple-comparison correction
• feat: declare model lineage in the config instead of inferring it #257 (Config: declare model lineage instead of inferring it from the id #255) declared model lineage instead of the id heuristic
• feat: noise-floor control and floor-adjusted flip rate #258 (Analysis: measure the noise floor in the runner and report flip rate against it #256) noise-floor control
• feat: run the imaging lane through the run entry point #260 (Runner: imaging lane in benchmaxxing run (--image-root and array twins) #254) imaging lane
• feat: hard-case selector for the cascade arm #261 (Cascade: hard-case / uncertainty selector (where cascades can actually form) #116) hard-case selector
• feat: per-run reproducibility bundle and the report command #262 (Real-run results report and reproducibility bundle (per-arm) #105) reproducibility bundle and benchmaxxing report

Each is scoped to one issue with its base set to the previous branch, so they review bottom-up. If you'd rather have fewer PRs I can squash the later ones together, just say which grouping you want.

[sebasmos]

@duckyquang you have access to the API, could you please revise?

[duckyquang]

@duckyquang you have access to the API, could you please revise?

On it, I'm currently on a road trip right now. Will get back by tonight.

[maximinl]

Peer note on the stack base.

Wiring looks sound: stage logic stays in experiments.py, runner.py is adapters + I/O, cascade dumps shared/isolated transcripts under --out/transcripts/ (exactly what offline deference reanalysis needs).

Two things still blocking a clean merge from my side, both already flagged:

1. Seb’s real-model pilot (GEMINI_API_KEY, ~20 cases) — mock-only acceptance is not enough for the parse/prompt path.
2. Everything stacked on this (#252→#253→#257→#258→#260→#261→#262) should wait; review those bottom-up after this lands.

No new code objections beyond the outstanding real run.

[armaanvgrewal]

@duckyquang @maximinl

FYI, per @sebasmos, #251 should land before #239 since both touch benchmaxxing/cli.py. I’ll wait on #239 until this is merged.

[Agastya191]

Really clean wiring in runner.py, and driving all four stages through the CLI on the mock backend is exactly the test net this entry point needed. The one thing I'd settle before trusting the overlap arm is _lineage: it splits the family off the id at the first -/_//, so a bare gemini-2.5-flash resolves right but an org-prefixed meta-llama/llama-3.1-70b resolves to meta (and mistralai/... to mistralai), landing it in a different lineage than a plain llama-3.1-70b. Because run_holes_test splits the within- vs cross-lineage overlap on exactly that value, a roster that mixes id conventions computes a plausible but wrong cross-lineage number with nothing raising. You floated making it explicit in the config and I'd take that option: a per-model lineage on Config that falls back to _lineage when unset keeps the convenience without letting a string silently decide the split.

[duckyquang]

@Agastya191 took the option you floated. Rather than patch _lineage in place, the fix lives one PR up in #257: lineage is declared in the config and only falls back to the id heuristic when unset. On top of that I added a run-time warning (#257, 3ed12d9) so an org-prefixed id that gets inferred — the exact meta-llama/llama-3.1-70b → meta case you hit — is loud at run time instead of only turning up in the manifest afterwards. So a mixed-convention roster can't quietly compute the wrong within/cross-lineage split anymore.

@sebasmos the real-model pilot is in progress (the ~20-case Gemini run), so the parse/prompt path gets exercised on a real backend, not just mock. Will post once it's done.

@maximinl thanks. On your two blockers: the real pilot is running now, and I'm going through the stack bottom-up (#252 → #262) so nothing above this lands ahead of it.

[sebasmos]

Approving and merging. The pilot is done and committed: 20 MedQA cases, 60 twins, 120 real Gemini calls at temperature 0, and it reproduces the paper's solo rates through a different runner (longest_option 0.05 and option_order 0.05 exact, lexical_overlap 0.10 against 0.09, overall 0.0667 against 0.0633). That satisfies the real-data ask I had been holding this on. Three tests pin the artifact as a real run rather than a mock.