Run quantized Whisper on the fused Metal decode path

[ajroetker]

Summary

Follow-up to #760: quantized Whisper GGUF bundles now run on the same fused Metal decode path as f32, the export command applies a Whisper-aware quantization policy, the 128-mel large-v3 checkpoints load, and a ready-made Q8_0 large-v3-turbo is published at antflydb/whisper-large-v3-turbo.

• Fused decode for quantized slots. The in-place K/V and fused Q/K/V projections were dense-only, so a quantized bundle fell back to separate projections plus blit copies into the cache (17 compute encoders and 8 blits per token). Quantized slots now write straight into the slab rows through the existing quantized single-row and fused Q/K/V kernels, with the slot bias applied on device. No new kernels; q8_0 and q4_0 decode on one encoder with zero blits.
• Export policy. antfly inference export <dir> --target gguf --format q8_0|q4_0 already produced Whisper bundles. Under the requested format, position tables now stay dense and the token table (tied output head) and the whole encoder never go below q8_0 (q6_k included); decoder layers take the requested format. Prefix include and exclude filters still apply on top.
• 128 mel bins. large-v3 and large-v3-turbo failed with InvalidInputShape because the mel front end assumed 80 bins. The bin count now comes from the checkpoint config through the server, the dictation runtime, and the CLI.
• Published bundle. antfly inference pull antflydb/whisper-large-v3-turbo:gguf:Q8_0 --tasks transcribe installs the Q8_0 export (829 MB, MIT, model card with the reproduce command). Verified end to end: pulled through the registry and transcribed on Metal.
• Guide section with the pull and export commands, the policy, and the measurements below.

Why the policy

• A 4-bit token table breaks language detection on tiny (the Spanish clip decodes to nonsense); q8_0 fixes it.
• q4_0 across all 32 encoder layers of large-v3-turbo yields nonsense on both the Metal and CPU backends and with quantized linears computed on the host reference path, while either 16-layer half alone, q4_1, q5_0, and q4_k are all correct. That is accumulated q4_0 error, not a kernel bug.
• q8_0 is also the fastest quantized encoder on Metal; the q4_1, q5_0, and q4_k encoders are two to three times slower because they lack the fast multi-row kernels.

Measurements

Metal, warm CLI run, 2.5 s English clip, M-series laptop. The published large-v3-turbo safetensors are f16.

Checkpoint	Weights	Size	Encoder	Prompt block	Decode per token	Transcript
tiny	f32	144 MB	62 ms	28 ms	2.2 ms	reference
tiny	q8_0	42 MB	47 ms	10 ms	2.5 ms	identical
tiny	q4_0	39 MB	45 ms	9 ms	2.2 ms	drops a word
small	f32	922 MB	290 ms	86 ms	9.6 ms	reference
small	q8_0	257 MB	106 ms	28 ms	9.0 ms	identical
small	q4_0	203 MB	115 ms	29 ms	8.9 ms	identical
large-v3-turbo	f16	1543 MB	600 ms	52 ms	6.0 ms	reference
large-v3-turbo	q8_0	829 MB	406 ms	14 ms	5.4 ms	identical
large-v3-turbo	q4_0	779 MB	415 ms	14 ms	5.4 ms	identical

Decode per token barely moves because it is bound by kernel dispatch at these sizes, not weight bandwidth; the gains land in size, encoder time, and the prompt block. q8_0 is the recommended format. The f16 checkpoints run their decoder on the unfused path (the dense fused Q/K/V entry only takes f32 weights), which is another reason to export large-v3-turbo to q8_0.

Test plan

• Filtered inference unit tests (210 selected, all passing), including: exported tensor types under the policy with include and exclude filters; 128-bin mel shape plus BLAS and FFT parity; quantized projections written in place into slab rows on Metal with nonzero biases, the fused q8_0 kernel, the mixed-format fallback, and untouched neighboring rows
• Transcription e2e (test_transcribe.py, test_dictate.py, test_transcription_sessions.py): 24 passed against a server from this tree
• /transcribe and /dictate over HTTP against a q8_0 tiny bundle installed as a model directory (Spanish clip, correct transcript and language)
• CLI parity of q8_0 and q4_0 bundles against f32 on the English and Spanish clips for tiny, small, and large-v3-turbo; frame trace shows one encoder and zero blits per token
• antfly inference pull antflydb/whisper-large-v3-turbo:gguf:Q8_0 round trip, then transcription of both clips from the pulled directory

Deferred to a follow-up: listing the bundle on the supported-models page as the accuracy recommendation, and whether the e2e suite should pull it.

[ajroetker]

/ci run b2abb22