Add voice dictation and streaming transcription to the inference service - #760
Merged
Merged
Conversation
Two new tiers on top of the Whisper transcriber: push-to-talk dictation (/dictate) with an optional LLM cleanup pass, and streaming transcription sessions (/transcription/sessions) with endpointing, partials, an SSE event stream, and a duplex raw-audio upload that works over HTTP/2 and over HTTP/1.1 chunked bodies. Transcription pipeline: KV-cached native Whisper decoder, timestamp tokens with word-level timing, prompt conditioning (<|startofprev|>), hallucination guards with temperature fallback, dynamic audio context, language lock across windows, long clips cut at quiet points, and a native Silero VAD next to the energy VAD. The GGUF exporter now writes BPE merges so exported Whisper bundles load; the registry accepts a `vad` pull task; the prompt KV cache is on by default. Metal: each decoder token runs as one command buffer with one compute encoder and no blits. Weights are resident handles, K/V cache slabs are preallocated and projected in place, Q/K/V share one dispatch, every residual add is fused into the following layer norm, the token choice (suppression lists and timestamp grammar) runs on the device, the single-query attention kernel is enabled for all callers with a split-K form for long K/V, and the projected encoder keys stay resident. On an M-series laptop whisper-tiny decodes at about 2.3 ms per token, down from about 100 ms. Also: Metal shared-provider lease and runtime-cache allocator fixes that made the Gemma executor fail under concurrent requests, 503 mapping for capacity refusals, node-wide session buffer cap, e2e suites for both tiers, and a guide under docs/guides/voice-dictation.mdx.
Compute the log-mel spectrogram with BLAS matrix products, run the encoder as one Metal frame with im2col convolutions, the hd64 flash attention kernel, and the fused add+layernorm rows kernel, upload the encoder output once before the cross projections, project the prompt block's K/V in place, and loop the split-K attention kernel per query row for short prompt blocks.
The logits kernel can now choose the token on the device, keep the timestamp grammar in a device buffer, publish the token to a slot the next step embeds from, and write its statistics into a two-slot ring, so the transcription loop can encode step p+1 while step p runs and submit it once p is awaited. Transcripts are unchanged, but with one frame in flight the host encode was never on the critical path and decode time does not move, so the mode stays behind TERMITE_WHISPER_ENABLE_PIPELINED_DECODE.
…ffer accounting A closed or committed utterance longer than max_segment_ms was handed to the decoder whole, which decodes at most one 30 s window, and the rest of the buffer was then dropped; settled speech is now cut at the quietest point of the window like open speech. Session creation took its response snapshot after inserting the entry, so a snapshot allocation failure freed an entry the map still pointed at; the snapshot is taken first. The node-wide audio buffer check counted other sessions at their last released size, letting two concurrent uploads each take the whole allowance; a passing check now reserves the room in the entry's published size so a concurrent session sees it at once.
The event queue grew one event at a time, so an allocation failure partway through left the earlier events queued while the caller freed the whole batch. Capacity for the batch is now reserved before any event changes hands, so on error the caller still owns every event.
Contributor
Author
|
/ci run ebf95ec |
Contributor
Author
|
/ci run 433e799 |
Contributor
Author
|
/ci run 9ac144a |
Contributor
Author
|
/ci run 93d8a1b |
1 similar comment
Contributor
Author
|
/ci run 93d8a1b |
# Conflicts: # go/pkg/sdk/oapi/client.gen.go
Contributor
Author
|
/ci run c5fd457 |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two new tiers on top of the Whisper transcriber, plus the Metal work that makes them fast enough to use interactively.
Dictation and streaming sessions
/dictate: push-to-talk dictation with an optional LLM cleanup pass (raw/cleanstyles)./transcription/sessions: streaming sessions with endpointing, partials, an SSE event stream, and a duplex raw-audio upload that works over HTTP/2 and over HTTP/1.1 chunked bodies. Audio is batched in 100 ms chunks before endpointing, so clients keep streaming silence like a microphone.<|startofprev|>), hallucination guards with temperature fallback, dynamic audio context, language lock across windows, long clips cut at quiet points, and a native Silero VAD next to the energy VAD.vadpull task; the prompt KV cache is on by default.docs/guides/voice-dictation.mdx.Metal performance (whisper-tiny, M-series laptop, warm server)
TERMITE_WHISPER_ENABLE_PIPELINED_DECODE=1: transcripts are identical, but with one frame in flight the host encode was never on the critical path, so decode time does not move.All Metal paths keep transcripts bit-for-bit identical to the CPU decoder; each has a kill switch documented in the guide.
Test plan
test_transcribe.py,test_dictate.py,test_transcription_sessions.pyagainst a ReleaseFast server: 24 passed