feat(#134): cross-lineage open-weights VLM end-to-end smoke

[Agastya191]

What

Unblocks the cross-lineage arm (#134): proves a real open-weights vision-language model can drive a committee turn alongside Gemini, so oversight is not all one backbone. A Gemini-only committee can only ever produce same-lineage panels; this wires the other half.

• benchmaxxing/cross_lineage_smoke.py GatewayAgent is the missing bridge between the gateway (Backend.complete(prompt, image, decoding) -> str) and the blackboard (respond(view) -> AgentResponse): it renders the committee-member prompt plus whatever is on the board, calls the backend, and parses the reply into a committed option. Until now every experiment reinvented this inline (referee_judge.py, the cli smoke); this is the reusable form, so one wrapper drives Gemini and the open-weights VLM alike.
• benchmaxxing/gateway.py TransformersVLMBackend runs a local Qwen-VL / Llama-Vision through the transformers image-text-to-text pipeline with no API key. transformers/torch/accelerate (the models extra) are imported lazily and guarded, so the core installs and tests without them; a pipe can be injected for offline tests.
• run_cross_lineage_smoke builds a size-2 cross-lineage committee (open-weights + Gemini), drives one shared committee turn on one image, and returns a report. experiments/cross_lineage/ has the runnable script and the VRAM/setup notes.

Skip path

With the weights, the Gemini key, or accelerate unavailable the runner returns a skip report rather than failing, so the script and tests are safe to run anywhere. The offline wiring and skip path land here; the real open-weights completion needs a GPU (Qwen2.5-VL-7B ~18-20 GB; see hardware_notes()), same deferral as #93's numbers.

Adversarial review

A Codex pass flagged the real GPU-only pipeline call: the prompt was passed positionally into the pipeline's images slot (would TypeError before generating), device_map="auto" needed accelerate that availability did not check, and the test fake mirrored the broken shape. Fixed: the call is now a keyword text= chat message carrying the image, open_weights_available gates on accelerate (added to the models extra), and the fake matches the real signature. Caveat worth flagging: the transformers pipeline path is still exercised only through injected fakes offline, so the GPU integrator should confirm the exact call for their transformers version.

Testing

tests/test_cross_lineage_smoke.py plus additions to tests/test_gateway_backends.py, offline only: the bridge parses options and keeps content, abstention commits nothing, the smoke drives one turn per member with the board propagating to the second, the image reaches every backend, the guards (same-lineage, non-open-weights, missing image) raise, and the skip path returns cleanly / raises under require. Suite 681 passed / 3 skipped, ruff clean, benchmaxxing smoke composes.

Closes #134

[sebasmos]

This is the right unblock for #134, and folding the Codex fixes (keyword text=, accelerate gate) in is good. Approving the wiring. It only becomes real once it runs on a GPU: please post one real cross-lineage turn (Qwen2.5-VL + Gemini on one image) with the committed answers, so we know the pipeline call matches your transformers version. That single real run is what turns this from a skip-path into #95.

[Agastya191]

Thanks @sebasmos. I can't post the real Qwen2.5-VL GPU turn yet (no GPU on hand), but here is what I verified and ran.

Verified the pipeline call against the real transformers source (>=4.50, the pinned floor), so it should run first try:

• chat inputs forbid a separate images= arg (the pipeline raises), so the inline-image + no-images= shape is what transformers actually wants;
• top-level max_new_tokens is the documented form (not generate_kwargs=);
• {"type":"image","image":<PIL>} inline is exactly Qwen2.5-VL's documented usage;
• the chat return shape (last assistant message content) is what _generated_text reads.

All of that was already correct, so I left it untouched.

Real end-to-end run (Gemini key, real image): I drove the actual run_cross_lineage_smoke on a real PA chest X-ray with a real GeminiBackend behind both committee slots (the open-weights slot stood in by Gemini, since no GPU). Both members committed No (ground truth: no pneumothorax), image-grounded ("no visceral pleural line separated from the chest wall; lung markings extend to the periphery"), cross_lineage: true, 2 turns with the board propagating to the second member. That validates the GatewayAgent bridge, the shared committee turn, board propagation, and answer parsing against a live VLM API on a real image. The Qwen half still needs the GPU box, same deferral as before.

I also pushed a1c3c50 hardening the image loading (require Pillow for the path form) and the pipeline reply parsing, with two regression tests; suite 683 passed / 3 skipped.

[sebasmos]

Superseded by my later review on this PR, which re-checks it at a1c3c50e as of 31 Jul. The position is unchanged; the current statement of it is in the newer review rather than repeated here.

[sebasmos]

Checked again 31 Jul. Nothing has changed, and the ask is unchanged with it.

Last commit is a1c3c50e, 2026-07-24T02:22:50Z. The only activity since is your comment that you have no GPU on hand. experiments/cross_lineage/ still contains one file, the script, and there is no results/ directory and no artifact of any kind among the six changed files.

So the position is the same as a week ago: 906 lines of wiring and tests with zero executed output. It merges clean and the suite is green at 1099 passed, so nothing is wrong with the code as code. The real-data rule is what is unmet, and it is the rule that matters most for this project, since a cross-lineage referee is the one claim we cannot make from same-lineage Gemini runs alone.

This is the smallest gap-to-value ratio of any open PR. One real Qwen2.5-VL turn plus one Gemini turn on a single image, with the transcript committed, and this lands. It does not need the full battery, or a cohort, or a manifest. One image is enough to make the wiring evidence instead of intent.

If GPU access is the blocker rather than time, say so on the issue and I will look at whether the AWS grant can cover a single turn, since that is a cheaper unblock than waiting.

Holding, but on a run rather than on the code.

[EmmanuelpaulKwesiga]

The is_cross_lineage field at schema.py:79 derives from declared ModelSpec.lineage, so the Gemini-only run reports is_cross_lineage=True by spec declaration. The field should read from actual backend types at runtime before merge.

[duckyquang]

Reviewed the wiring and ran the tests — 37 pass, and I sabotage-checked the six guards (board propagation, image forwarding, abstention-commits-nothing, same-lineage reject, open-weights reject, and the keyword-only _pipe(text=...) call). Each one goes red when I break the thing it protects, so they're real guards, not vacuous. The _FakePipe keyword-only signature does catch the exact positional-collision regression from the Codex pass. Nice.

Two things holding me.

cross_lineage / open_weights_completion are construct-true, not observed (cross_lineage_smoke.py:304-305). is_cross_lineage is len({m.lineage for m in members}) > 1 over the declared ModelSpecs (schema.py:69-80), and the runner already guards that the two specs differ in lineage and one is is_open_weights — so cross_lineage: true can't fail, and when gateway_backends is injected nothing checks the backend serving the open-weights slot is actually an open-weights VLM. I bound one non-open-weights backend to both slot names and still got cross_lineage: True with the "qwen" turn served by Gemini. That's exactly what Emmanuel flagged at schema.py:79. Since the whole point of this PR is trustworthy cross-lineage evidence, the report should read the actual backend identity, not the spec — that fix is cheap and it's what makes the eventual real turn count as evidence.

"End-to-end" isn't met yet. The real TransformersVLMBackend path — the pipeline("image-text-to-text", …) build and the self._pipe(text=messages, …) call at gateway.py:492-513 — only runs through the injected fakes; with torch/transformers absent it's the skip path, so no real Qwen turn exists (you've said no GPU on your end). That's the real-data rule, same hold Seb has. One real Qwen+Gemini turn closes #134.

The wiring itself is mergeable-quality — it's these two that block the value claim.

[sebasmos]

Pushed a fix on this branch (71bb40c): cross_lineage was construct-true from the declared ModelSpecs, so binding one backend to both slots still reported True. Report now also carries same_backend_served_both_members, tracking the actual backend object per turn. Also merged main to clear the dirty mergeable state. GPU-gated real Qwen turn still open, same as before. Can you verify and merge if good?

[sebasmos]

Closing as part of the pre-release cleanup, ahead of a git history rewrite that changes every SHA in this repository and would strand this branch regardless.

Not a rejection of the work. The branch is archived to a git bundle and can be restored in full; #265's cross-lineage smoke needs a GPU turn that is not being pursued, and #387's CheXpert interim analysis is not depended on by either paper draft since the CheXpert paragraph now reports raw shared adoption from natural_independent on main.

If either lane is picked up again, restore from the archive and open a fresh PR against the rewritten main.