feat(host-core): let Read return an image the model can be shown

[zhangqingkun976]

Fixes #711.

The report is exact: with an image-capable model, asking the agent to read an image from a path fails while attaching the same file to the prompt works. The reason is in the tool, not in the transport — and the transport half of this already existed, untested.

What was actually missing

Read refuses any file whose extension is in BINARY_EXTENSIONS (png, webp, ico …) and then refuses anything the byte sniff calls binary, both as TOOL_BINARY_CONTENT ("has no text to read"). So the model was told the file had no content, which is true of text and false of an image.

The other half was already wired: a host tool result shaped {text, images: [{data, mimeType}]} becomes [{type: "text"}, {type: "image"}…] in the runtime, gated on the model declaring image input (visionFromModelConfig). Nothing produced that shape, and the path had no test. pi's own AgentToolResult.content is (TextContent | ImageContent)[], so this needed no new plumbing.

What the change does

input	result
shot.png (a real PNG, model declares image input)	one text block plus an image block with image/png and standard base64
the same, model declares text only	the text block alone, saying an image is there and to say so rather than guess
notes.png (text renamed)	TOOL_BINARY_CONTENT — a request carrying a broken image fails whole, so the magic number has to agree with the extension
huge.png (4 MB)	TOOL_IMAGE_TOO_LARGE, naming the size and the alternative
.pdf, .zip, …	unchanged TOOL_BINARY_CONTENT

The 3 MB raw bound exists because base64 inflates by four thirds and the strictest per-image ceiling this build talks to is 5 MB encoded. The base64 payload never reaches a persisted UI message or transcript record — the stored detail keeps imageCount, which the existing attachment rule already required.

Verification

• host-core: read_returns_an_image_for_the_model_to_view, read_refuses_an_image_extension_whose_bytes_are_not_an_image, read_refuses_an_image_too_large_to_inline (the fixture is a real 1×1 PNG, decoded from base64 in the test); cargo test -p host-core --locked → 552 passed, 4 failed, the four being the pre-existing Windows path-separator assertions in mcp_servers, user_skills, scheduled_rpc and scheduled_tools.
• agent-runtime: two new tests cover the model-side bridge for the first time (attached with vision; dropped without, text kept). On the rebased head d71d97ed the suite is 751 passed / 1 failed, the failure being the pre-existing native-pi-session case, and src/runtime.test.ts — 215 tests, the new bridge among them — passes in full.
• cargo fmt --all --check 0; cargo clippy --workspace --all-targets --all-features exit 0; shared 78/78 test files (the new code in errors.ts); pnpm lint clean; node scripts/check-architecture.mjs passed; pnpm docs:check (78 EN/zh pairs, 495 pages) exit 0.

E2E: partial. pnpm test:e2e (scripts/e2e-smoke.mjs, the headless protocol suite) → 23/23 passed, 2 skipped on this branch, and it does exercise the changed code: E2E-013-read-tool runs the real Read tool through the host binary built from this branch, across the RPC boundary. What is not run is the scenario written below — E2E-TOOL-read-returns-an-image-the-model-can-see needs a live provider request with a fixture image, and this repository's E2E set does not drive a real image read. Alternative validation: the host tests exercise the real Read tool through execute_tool, and the runtime tests exercise the real tool callback and assert on the exact content blocks pi-ai would send. Remaining risk: how a specific provider renders the image (and whether a relay strips image blocks) is not observed here.

Docs

03-runtime/16-tool-result-limits.md §4/§6 and its acceptance list, 03-runtime/08-error-codes.md (the new code plus the widened meaning of the old one), 08-meta/decisions-log.md D609, and the E2E scenario above — all mirrored to docs/zh-CN/….

D609 was free when this was opened and still is: D606 (calibration), D607
(plugin crash) and D608 (tool-call ids) have since landed in main, and
#748/#756/#760 are closed. D604/D605 still belong to the open #721, so nothing
needs renumbering unless that one lands first.

Base and head

base 43a37373 · head c65b1177 · 11 files, +378 / −7. Rebased onto current main (24 commits further on);
it applied cleanly, including the convertToLlm seam main has since rewritten.

[zhangqingkun976]

Rebased onto main (ab39a9b4) so the new Head contains latest base gate passes. Head is now 7b5ffe4e.

The only conflicts were append-vs-append — the E2E scenario catalog and the decisions log, where main and this branch each added an entry — and both sides are kept. The identifier bands are still free: upstream's newest ADR is 0299 and its newest decisions-log entry is D607 (the plugin crash report, which is this repository's own fix for issue #747 built on my #756).

[zhangqingkun976]

Rebased onto main (b71fcf05, 39 commits further on); the code and both docs applied cleanly. New head d71d97ed.

Two things changed in the description, both about evidence:

• E2E is no longer "NOT RUN". pnpm test:e2e (scripts/e2e-smoke.mjs, the headless protocol suite) is 23/23 passed, 2 skipped on this branch, and it exercises the changed code — E2E-013-read-tool runs the real Read tool through the host binary built from this branch, across the RPC boundary. What remains unrun is the written scenario (E2E-TOOL-read-returns-an-image-the-model-can-see), which needs a live provider request with a fixture image.
• The D609 note is updated: D606/D607/D608 have since landed in main and fix(agent-runtime): calibrate the context estimate against what requests cost #748/fix(desktop): name the plugin host exit code in a crash report #756/fix(agent-runtime): keep tool-call ids unique in every request #760 are closed, so nothing needs renumbering unless feat(agent-runtime): recall, a reversible compaction boundary, and per-model context settings #721 lands first.

Local gates on the rebased head: cargo fmt --all -- --check 0; cargo clippy -p host-core --all-targets --all-features exit 0; cargo test -p host-core --locked 552 passed / 4 failed (the four pre-existing Windows path-separator asserts); agent-runtime typecheck 0 with the suite at 751 passed / 1 failed (the same pre-existing native-pi-session case, and src/runtime.test.ts — 215 tests, the new bridge among them — passes in full); shared typecheck 0; pnpm docs:check exit 0; node scripts/check-pr-base-main.mjs passed.

[zhangqingkun976]

Rebased onto main (43a37373, 24 commits further on). New head c65b1177, one commit.

It applied cleanly, including the convertToLlm seam in runtime.ts that main has since rewritten: your tool-call-dedupe.ts refactor of my #780 now sits there, and git's three-way merge composed it with this change instead of dropping either side. That composition is verified rather than assumed — on the two branches that add a pass at that seam (#705 and #721, a superset of it) the seam now reads this.narrowToolResultsUnderPressure(this.dropDuplicateToolCalls(messages)), so the dedupe runs first and the tiering second. Both passes are identity functions when they have nothing to do, which keeps this PR's original promise that an ordinary request is byte-identical.

Local gates on the rebased head: node scripts/check-pr-base-main.mjs passed; cargo fmt --all -- --check 0 where the branch touches Rust; pnpm docs:check 79 EN/zh pairs, 499 pages on the branches that touch docs. mergeable=true, behind=0.

One thing to know when reading the JS job: main has landed image generation and its new suite is red on main itself. Pristine 43a37373 fails 8 tests across 4 files (native-pi-session, parent-host-proxy, image-generation, openai-images-contract); this branch fails exactly the same 8 and adds its own on top. I have not touched those files.