Decode HEIC chat attachments to JPEG via the OS decoder

[TeleporterGuy]

Summary

• Decode iPhone HEIC/HEIF photos to JPEG on ingest using the OS decoder (sips on macOS, optional heif-convert/magick on PATH) so Stem never ships HEVC.
• Convert on the desktop before upload so a Mac client talking to a Linux server still works; thumbnails use JPEG because Chromium cannot paint image/heic.

Scope (what this does and does not do)

This is chat attachments only: paperclip, drop onto the conversation, clipboard paste. Those files are decoded to JPEG and sent to the model as native images.

Connected / shared folders are unchanged. The folder index still catalogs Markdown, .txt, PDF text layers, and Word. HEIC (like PNG/JPEG) is skipped as an unsupported binary — Stem will not search, memorize, or automatically vision-read photos sitting in a connected folder. Dropping a .heic from that folder onto the chat still works, because that is the attachment path. The agent can still open other files in the folder as ordinary files while working, same as today.

Licensing

Stem does not distribute HEVC. Decode is delegated to the OS (sips / ImageIO on macOS) or a user-installed tool on PATH (heif-convert, magick). The official Docker image does not install libheif. Linux/Windows without such a tool skip the file with a named note.

Why the tests look like this

HEIC decode is a patented codec we refuse to ship. The unit suite therefore cannot assume a real decoder exists on Linux/Windows CI, and must not depend on checking in a copyrighted iPhone .heic fixture.

• Mock the decoder (setHeicDecoderForTests) for ingest, skip notes, client-already-converted JPEG, and Chromium preview. That covers the Stem contract: HEIC becomes image/jpeg for pi, garbage is named in rejected, converted attachments are not decoded twice.
• A 16-byte ftypheic stub is enough to assert brand detection. A real photo is unnecessary for classification tests.
• One macOS-only sips smoke builds a 1×1 HEIC from a PNG and decodes it back. That is the path Stem actually ships; a mock cannot catch a wrong sips flag. It is skipped on Linux/Windows, and if encode is blocked (sandbox) the test returns rather than failing CI.
• Renderer tests assert pasted HEIC stays a file chip until previewImageData returns a JPEG data URL — Chromium cannot paint image/heic, so an optimistic HEIC thumbnail would be a broken <img>.

Swallowing decode/preview failures is intentional (preview is not the send; a missing PATH tool is the Linux default). Those catches carry // quiet: so quiet-failures.test.ts agrees.

CI note

Unit tests, typecheck, and lint are green. The remaining red jobs (test on macOS and Ubuntu) fail one Playwright E2E:

tests/e2e/external-server.spec.ts — a turn killed along with the server stops pretending to run

After SIGKILL + restart it expects Echo: after the second crash and never finds an assistant bubble. That spec exercises reconnect/resync after a killed stem-server. It does not attach images or call the HEIC decoder. 100+ other E2E tests in the same run passed; unit-windows passed. Treat this as an unrelated flake of the replay/resync path unless a re-run reproduces it.

Test plan

• Attach a .heic from Downloads or paperclip on macOS — bubble shows thumbnail and model sees the photo.
• Paste a HEIC from clipboard — chip upgrades to JPEG thumbnail after preview IPC.
• Remote server: Mac desktop + Linux Docker — JPEG reaches the server without libheif in the image.
• Linux without heif-convert: skip note names the file and mentions decode failure.
• Confirm a .heic in a connected folder is still not indexed; dropping it onto the chat still works.