Conversation
stikves
force-pushed
the
sukru/vlm-server-image-input
branch
3 times, most recently
from
September 23, 2026 20:17
111c4a0 to
b957b43
Compare
stikves
marked this pull request as ready for review
September 23, 2026 20:59
stikves
requested review from
alejandro-isaza,
pkmandke,
srjoglekar246 and
tjia1818
September 24, 2026 22:20
| public struct ReplayRequest: Decodable, Sendable { | ||
| /// Caller-supplied id, echoed on the result. Optional. | ||
| public let id: String? | ||
| /// Session identity — maps to `X-Session-ID`. Defaults to "default" when absent. |
Contributor
There was a problem hiding this comment.
Defaults to "default" when absent.
Is this true? If so, seems like it happens very downstream. Might be more clear to set the default in init
Contributor
Author
There was a problem hiding this comment.
This was from the stacked PR, they should be merged now
| if let outputPath { | ||
| try output.write(toFile: outputPath, atomically: true, encoding: .utf8) | ||
| } else { | ||
| print(output, terminator: "") |
Contributor
There was a problem hiding this comment.
this will always print if outputPath is not defined. Desired? Or switch to CLILogger?
Contributor
Author
There was a problem hiding this comment.
This is desired, the parent process will grab the stdout. CLILogger is gated and not built for this purpose.
Add a vision-language path to llm-server so a kind=vlm bundle (e.g. muse-glimmer-30b) can be served over the OpenAI-compatible chat endpoint with image input. - LLMServerMain builds CoreAISequentialVLMEngine directly for kind=vlm bundles (EngineFactory only produces text engines) and skips the text-shaped warmup. - ServerState carries the VisionConfig and exposes the engine as a MultimodalInferenceEngine. - runVLMCompletion decodes one image (base64 data URL or local path), runs the vision encoder, expands the image placeholder in the prompt, and generates through the multimodal engine. It is routed from the shared runChatCompletion core, so both the HTTP non-streaming handler and --replay exercise it. - Output reuses the existing think-tag / tool-call parsing, so reasoning models return a clean final answer with the analysis separated into reasoning_content. - /v1/models reports supports_vision for VLM bundles. Validated with --replay on the muse-glimmer-30b VLM bundle: an image referenced by path is transcribed accurately, with the harmony analysis stripped from the answer. Stacks on the seed (apple#265) and --replay (apple#266) branches since the VLM core lives in the shared runChatCompletion; the diff includes those commits until they merge. Deferred: multiple images, video, http(s) URL fetch, guided/constrained decoding on the multimodal path, VLM prefix reuse, and the HTTP streaming VLM path.
Add a --file-access flag (off by default, subdirs) controlling whether image_url content parts may read local files. off rejects all bare paths and file:// URLs; subdirs allows only canonicalized real paths inside the server working directory, rejecting .. traversal, absolute-outside paths, and symlink escapes. Thread the policy through ServerConfig into decodeImage. Reject stream:true requests carrying an image with a 400 instead of silently answering on the text path without the image.
Replace the inline kind == .vlm CoreAISequentialVLMEngine build in LLMServerMain with the shared EngineFactory bundle entry point, which routes VLM bundles to the sequential VLM engine and text bundles to the text engines. Chunking overrides pass through engineOptions; the VLM warmup skip is unchanged.
stikves
force-pushed
the
sukru/vlm-server-image-input
branch
from
September 26, 2026 02:51
96962e1 to
fc5e0ca
Compare
Co-authored-by: kevchengcodes <59463423+kevchengcodes@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a vision-language path to
llm-serverso akind=vlmbundle (e.g. muse-glimmer-30b) can be served over the OpenAI-compatible chat endpoint with image input.What it does
LLMServerMainbuildsCoreAISequentialVLMEnginedirectly forkind=vlmbundles (EngineFactoryonly produces text engines) and skips the text-shaped warmup.ServerStatecarries theVisionConfigand exposes the engine as aMultimodalInferenceEngine.runVLMCompletiondecodes one image (base64 data URL or local path), runs the vision encoder, expands the image placeholder in the prompt, and generates through the multimodal engine. It is routed from the sharedrunChatCompletioncore, so the HTTP non-streaming handler and--replayboth exercise it.reasoning_content./v1/modelsreportssupports_visionfor VLM bundles.File access & streaming
--file-accessflag controls whether animage_urlmay read local files. Defaultoffrejects all bare paths andfile://URLs (onlydata:andhttp(s)://are accepted);subdirsallows local files only when the canonicalized real path resolves inside the server working directory subtree, rejecting..traversal, absolute-outside paths, and symlink escapes. This closes an unrestricted local-file-read hole (a client could previously pass/etc/hosts).stream:truerequest that carries an image now returns a 400 ("Streaming is not supported for image (VLM) requests") instead of silently falling through to the text path and answering without the image.Testing
LLMServerTeststarget:decodeImageon a data URL and on a remote URL,buildPromptTokensplaceholder expansion, and file-access policy coverage (offrejects/etc/hostsandfile:///etc/hosts;subdirsallows inside-CWD paths and rejects traversal, absolute-outside, base-dir, and symlink-escape).--replayon the muse-glimmer-30b VLM bundle: an image referenced by path is transcribed accurately, with the harmony analysis stripped from the answer.Stacking
Stacks on the seed (#265) and
--replay(#266) branches since the VLM core lives in the sharedrunChatCompletion; the diff includes those commits until they merge.Deferred
Multiple images, video, http(s) URL fetch, guided/constrained decoding on the multimodal path, VLM prefix reuse, and the HTTP streaming VLM path (currently rejected with a 400).