Repository navigation
Unify stop-token resolution across text and VLM adapters - #269
Merged
Merged
Conversation
The Foundation Models VLM adapter stopped generation only on the main EOS and a hardcoded <|im_end|> lookup, so a VLM whose turn terminator is anything else (Gemma's <end_of_turn>, Phi's <|end|>) never stopped and ran to the response-token cap. The text server already resolves these via LanguageConfig.additionalStopTokenIds; this reads the same set once at load and unions it into the stop set. Models that use <|im_end|> are unaffected.
Extract the stop-token union in CoreAIVLMExecutor.respond() into a testable stopTokenSet(tokenizer:additionalStopTokenIds:) helper, since the executor itself needs a real CoreAISequentialVLMEngine (compiled .aimodel assets) to construct and can't be exercised at unit level. Tests prove additional stop tokens resolved at load are actually unioned into the set the executor checks per generated token, not just stored.
stikves
force-pushed
the
sukru/vlm-additional-stop-tokens
branch
from
September 21, 2026 22:20
f7edea8 to
53dd712
Compare
Text path now also stops on a base-vocab <|im_end|> via the shared additionalIds im_end check, matching the VLM adapter.
stikves
changed the title
Fold <|im_end|> into the shared LanguageConfig.additionalStopTokenIds resolver so it is universal, and route the server and CLI through it (they previously hand-rolled resolution and missed the fold). Dissolve the StopTokens enum: runtime union moves to Tokenizer.runtimeStopTokens, the agentic <|eot|> fold moves into the text adapter. Sort to Array only at IO/ordering boundaries (StopSequences fold, CLI log).
Move the base-vocab <|im_end|> fold out of additionalStopTokenIds (only called when a bundle ships a tokenizer directory) and into runtimeStopTokens, which every adapter runs regardless of tokenizer dir. Bundles using the HF-runtime tokenizer fallback now stop on <|im_end|> again.
Server's agentic-eot fold called convertTokenToId without checking vocabContains first, unlike the text-adapter reference pattern, so an unk-token fallback could be folded as a false stop. Add the guard. The CLI never folded the agentic <|eot|> into its stop set at all. Add the same detect-thinking-format + guarded fold used by the server and text adapter. Requires making Tokenizer.vocabContains public since llm-runner/llm-server are separate targets from CoreAILanguageModels.
stikves
marked this pull request as ready for review
September 22, 2026 13:49
stikves
requested review from
Lewis300,
carinapeng and
kevchengcodes
and removed request for
Lewis300 and
carinapeng
September 22, 2026 19:37
Lewis300
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unifies stop-token resolution across the text (
CoreAILanguageModel) and VLM (CoreAIVisionLanguageModel) adapters, the server (LLMServerMain), and the CLI (LLMRunnerMain), and makes stop/EOS collectionsSet<Int32>end-to-end, sorting toArrayonly at IO/ordering boundaries.Resolution splits into two functions:
LanguageConfig.additionalStopTokenIds(from:tokenizer:) -> Set<Int32>scans tokenizer_config.json / tokenizer.json for turn-end special tokens. Called only when a bundle ships a tokenizer directory.Tokenizer.runtimeStopTokens(additional:) -> Set<Int32>(Tokenizer+StopTokens.swift) builds the runtime terminating set:{eos-if-any} ∪ {base-vocab <|im_end| if present} ∪ additional. Both adapters call it in their generation loop.The base-vocab
<|im_end|>fold lives inruntimeStopTokens, so it applies whether or not the bundle ships a tokenizer directory (a bundle using the HF-runtime tokenizer fallback previously lost it).The agentic
<|eot|>fold is text-adapter-specific and is folded inline inCoreAILanguageModel.init.Types:
CoreAILanguageModel.additionalEosTokenIds,CoreAIVLMExecutor.Configuration.additionalStopTokenIds,StopSequences.init(additionalEosTokenIds:),ServerConfig.additionalEosTokenIds, and the CLI locals/helpers areSet<Int32>.StopSequences.sequences: [[Int32]]and the xgrammar-facingstopTokenIds: [Int32]?stay arrays (positional / C-facing). Additional IDs are sorted toArraywhen folded intoStopSequences.sequencesand in the CLI stop-token log line.Per-site agentic-EOT behavior is unchanged: the server folds an eot explicitly, the CLI does not.
Tests:
RuntimeStopTokensTestscoversTokenizer.runtimeStopTokens, including the base-vocab<|im_end|>fold.AdditionalStopTokensTestscovers the config scan and asserts the resolver does not fold base-vocab<|im_end|>.