feat: run the imaging lane through the run entry point

[duckyquang]

Summary

benchmaxxing run now drives Lane A. Set cue_set: image-v1 in the config, pass --image-root, and the solo lane (pilot, solo, overlap) runs on chest X-rays the same way the text lane does.

• images load through pillow into the uint8 arrays the cue injectors want, one twin per (case, cue) over cable, corner_tag, watermark, laterality
• the question is the yes/no reading question experiments/imaging already established, so the numbers stay comparable with the earlier imaging runs
• twin payloads carry the array plus the question and options, because an array alone does not say what was asked. SoloAgent routes them to complete(prompt, image=...)
• a case with no resolvable image or no positive finding label is counted and reported, not silently dropped and not fatal: Skipped cases: missing_image=1, no_finding_label=1 shows up in the plan and in summary.md

run_pilot and run_solo_baselines take an optional twins= so the imaging lane can inject its own pairs instead of the text-cue builders running. Additive, default unchanged.

Result

On a generated 4-image fixture (2 readable, 1 missing file, 1 negative label), solo with --backend mock: 8 twins, flip rate 0.3750 [0.1250, 0.6250], shortcut reliance 0.2500, skips reported. The mock answers the imaging lane from a checksum of the pixels, so it is sensitive to an injected artifact and the offline run is non-degenerate rather than a wall of zeros.

ruff check . clean. 688 passed with the image extra installed; 615 passed, 13 skipped without it, so the core suite still runs on a bare install.

Notes

The cascade stage refuses the imaging lane on purpose, with a pointer. experiments._shortcut_answer builds the planted answer from case.options, which an imaging case does not have, so an imaging cascade would report adoption of an answer no agent could ever give: a confident zero. That needs the seed builder from #104 / #115 first, and I would rather refuse than ship a degenerate arm.

Stacked: #251 -> #252 -> #253 -> #257 -> #258 -> this.

Closes #254

[duckyquang]

Extra check against real data rather than the fixture: pointed the imaging lane at the committed experiments/imaging/results/nih_manifest.csv (35 real ChestX-ray14 rows) with an image root that doesn't exist, since the pngs aren't in the repo.

$ benchmaxxing run --stage solo --manifest experiments/imaging/results/nih_manifest.csv \
    --config image.json --image-root /nonexistent --backend mock --dry-run
  lane: image
  cases: 35, twins: 0
  skipped cases: missing_image=35
  estimated model calls: 0

and the run itself refuses rather than writing an empty result:

error: no imaging twins could be built from 35 case(s): {'missing_image': 35, ...}.
Check --image-root and that the manifest carries a positive finding label.

So the skip accounting reads correctly off a real manifest, and a wrong --image-root fails loudly instead of producing a confident zero.

[sebasmos]

Wiring looks right, but the check used a nonexistent image root, so the comparability claim is still unproven. It needs a real NIH-image run posting per-cue flip-above-noise to match experiments/imaging. Requesting changes until then.

Superseded by the later Changes-requested review on this PR; dismissing the stale approve to avoid a mixed signal.

[Agastya191]

good work on this: runner.py threading the imaging twins through pilot/solo/overlap, the cues/image.py injection wiring, and the cascade refusal in _refuse_imaging_cascade are all clean, and the skip accounting is the right instinct. One problem is that in runner.py _finding does case.label.split("|")[0], so a multi-finding NIH case (nih_cxr14 stores the full pipe-separated Finding Labels, e.g. cardiomegaly|effusion) is only ever asked about its first finding. This drops the remaining findings without an entry in skipped_cases, so the Skipped cases: line reports full coverage the run never had, which is the exact failure the skip accounting was added to prevent. You have multiple options here; for example, I would either loop _finding over every positive label and build a twin set per finding, or, if one-question-per-case is the intended v1 policy, count the dropped findings so the skip report stays honest.

[duckyquang]

@Agastya191 kept the one-question-per-case v1 policy but made the accounting honest: _positive_findings + a new dropped_findings skip bucket, so a cardiomegaly|effusion case counts the finding it never asked about instead of the skip line claiming full coverage. mind re-reviewing?

[sebasmos]

The dropped_findings bucket is the right fix, honest accounting beats a silently wrong denominator. Still holding this one behind the priority-3 infra call rather than any objection to the code, and it needs a real API run to count under the real-data rule.

[duckyquang]

@Agastya191 fixed in 47d1133. v1 stays one question per case, but it's honest now: _positive_findings lists the real findings, image_twins asks about the first and adds the rest to a new dropped_findings bucket, which the skip line already renders. So a multi-finding NIH case like cardiomegaly|effusion no longer reports coverage the run never had.

@sebasmos agreed the dropped_findings bucket is the right call over a silently wrong denominator (done, 47d1133). And yes, this needs a real NIH-image run to count under the real-data rule — it's in the imaging battery I'm running, will post per-cue flip-above-noise to match experiments/imaging.

@maximinl thanks — the dropped-finding accounting is in (47d1133).

[sebasmos]

dropped_findings accounting is right (runner.py:316 and :389). Note it is clean against its base but conflicts with main in benchmaxxing/experiments.py; the resolution is keep-both, main's _build_twins = build_twins alias plus your twins=None on run_pilot. Still needs the real NIH-image run, since the --image-root check exercised the failure path rather than a run.

[sebasmos]

Approving. _positive_findings at runner.py:316 plus the dropped_findings skip bucket give honest accounting for multi-finding NIH cases instead of silently double-counting. I merged current main in; experiments.py needed the union resolution rather than a mechanical one, keeping main's _build_twins alias and progress parameters alongside your twins= path. 997 passed, ruff clean. The real NIH-image run through this path is still outstanding, the --image-root check exercised the failure path rather than a run, so #254 stays open.