Skip to content

feat: add native compression models with image-aware summaries - #656

Open
Neko-Yukari wants to merge 2 commits into
Tarquinen:masterfrom
Neko-Yukari:feat/model-aware-image-compression
Open

Neko-Yukari wants to merge 2 commits into
Tarquinen:masterfrom
Neko-Yukari:feat/model-aware-image-compression

Conversation

@Neko-Yukari

Copy link
Copy Markdown

Image-bearing tool results can contain megabytes of Base64. Treating that payload as summary text inflates token estimates and can reject compression before inference. This adds an optional native compression model that receives selected images when its metadata declares image input; text-only models receive retrievable references. Summaries retain those references in either case.

With compress.model set, the main model selects the topic and closed range boundaries. DCP sends only the selected observable source and existing compressed blocks to the configured model. Without this setting, existing main-authored summaries remain unchanged.

  • Extract media from user attachments, native/MCP tool results and embedded JSON/data URLs before text counting or summarization. Save inline bytes with content-derived filenames and preserve references after protected content is restored.
  • Use a tool-free isolated native session for image requests and generate.text for text-only requests. Check model capabilities, context/output budgets and image dimensions; forward cancellation and clean up the temporary session.
  • Validate all summaries before committing any compression blocks. Explicit fallback models are tried only for classified provider failures; invalid output, cancellation and ambiguous native errors retain the original context.
  • Align V2 SDK/theme types with OpenCode 2.0.30 and their OpenTUI peers; preserve compatibility with the earlier V2 media projection shape.

Validation:

  • npm run typecheck and npm run format:check pass in a clean LF checkout (the working Windows checkout uses CRLF for untouched files).
  • 168 root tests and 8 request-logger tests pass. On Windows, the logger's existing rm -rf build script was replaced by the equivalent direct TypeScript build for the check. One initial file-lock-related logger failure passed on isolated and full rerun.
  • The production bundle, declarations and package verification pass. The existing verifier's execFileSync("npm", ...) was adapted at invocation time to node npm-cli.js on Windows, without changing the repository script.
  • Independent media boundary tests reproduced four leaks/duplication cases; all are fixed and covered. Additional regressions cover unmatched-brace scanning, non-image bytes labelled as images, and native interrupted responses. Interrupted sessions are rejected even when their text is otherwise valid summary JSON.

Native end-to-end validation used the built plugin with OpenCode 2.0.30 and a controlled local provider: the vision branch made one summary request with one native image and zero tools, while the text-only branch made one request with zero images. Both committed a summary and retained an image file whose bytes matched the source. A third run with non-image bytes labelled as PNG completed with a reference and no image attachment. A separate real-provider run also generated a valid image summary and retained the original reference. All native checks used fresh V2 sessions; V1 native host integration and historical session migration were not exercised.

Known existing check limitation: npm audit reports 6 vulnerabilities (5 critical, 1 high), rooted in the same seroval, solid-js and brace-expansion versions as the upstream baseline. Auditing the baseline lockfile separately reports 17 vulnerabilities. This change does not apply unrelated forced dependency upgrades.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant