community_models: VibeASR INT8/ternary ASR pipeline (2/2)

[XsquirrelC]

Per your request in microsoft/VibeASR.cpp#10, split into two PRs: #447 is the additive ggml side, this one is the model integration. Stacked on #447 — this branch contains it, so review #447 first; the three commits from community_models: VibeASR I8_S VAE encoder onward are what's new here.

This replaces #440 and #446, which are now closed — same code, reorganized so the ggml and model halves are cleanly separated.

Result: --task asr --family vibeasr transcribes end to end on CPU.

$ ./build/bin/audiocpp_cli --task asr --family vibeasr \
    --model models/vibeasr/vibeasr-lm-i2_s-embed-q6_k.gguf \
    --model-spec-override model_specs --backend cpu --threads 8 \
    --audio assets/asr_validation/librispeech/librispeech_test_clean_6930-75918-0000.wav --metrics
text_output=Concord returned to its place amidst the tents.
metrics.wall_ms=1284.65
metrics.rtf=0.36652

Why this is a separate family and not a weight path in vibevoice_asr

Worth addressing head on, because audio.cpp already ships this model: vibevoice_asr in the core tree is the same architecture — same acoustic/semantic causal ConvNeXt tokenizers, same connectors, same Qwen2 decoder. What VibeASR.cpp adds is a different numeric pipeline, not a different model:

	vibevoice_asr (core)	this entry
Encoder weights	F32 / Q8_0	GGML_TYPE_I8_S, one F32 scale per tensor
Encoder activations	F32	INT8 throughout; every stage requantizes
Ops	generic ggml	the five fused I8_S ops from #447
Decoder weights	Q8_0 Qwen2	ternary GGML_TYPE_I2_S, 993 MB for a 1.5B decoder
Backends	CPU, CUDA, Metal	CPU only
Decode	greedy, sampling, beam search	greedy

It stays separate because the two share no encoder graph code: every activation there is I8_S and every node is one of the CPU-only fused ops, so folding it in would put a second, mutually exclusive graph builder and a second backend policy behind one family's loader. The reuse that is worth having — tokenizer vocabulary, prompt layout, feature-injection order, audio normalization — is data and conventions, and this follows vibevoice_asr on all of it, including using its exact audio front end rather than upstream's. Happy to fold it in as a weight path instead if you'd rather; the graph code is the only obstacle and it is contained in two files.

The decoder reuses the framework outright

modules::QwenCausalDecoderModule builds it unchanged. Every projection in it goes through LinearModule, which is a bare ggml_mul_mat on a 2-D flattened activation, so a ternary weight dispatches on type with nothing to opt into — that is what the ggml_mul_mat I2_S support in #447 buys. The published checkpoint leaves the embedding table at Q6_K and the output projection at F16 and both load through the framework's normal path; only the 196 blk.N.* projection weights need a dtype/shape check plus store.make_tensor(..., GGML_TYPE_I2_S, ...).

Geometry comes from the LM GGUF's KV block (28 layers, hidden 1536, intermediate 8960, 12 heads over 2 KV heads, RMSNorm eps 1e-6, RoPE theta 1e6). The checkpoint has no qwen2.attention.key_length, so head_dim falls back to qwen2.rope.dimension_count, cross-checked against head_dim * head_count == embedding_length and against token_embd.weight's actual shape rather than trusted. The encoder's geometry is derived from the tensor table instead — which block tensors exist, what shape each weight has — the same way upstream does it, since the checkpoint's KV block does not describe the VAE.

Two graphs per session: a prefill graph sized to the prompt, and a single-step decode graph over a static KV cache. Speech features are injected with ggml_set_rows over the EmbeddingModule output, so the embedding lookup and the <|speech_pad|> overwrite are one pass instead of a host-side gather. Both encoder branches emit 1536-wide features — decoder hidden size — and the reference sums them element-wise.

Faithful-to-upstream details that are easy to get wrong

• The six special tokens (151643–151648) are inserted by numeric id, and every text segment is tokenized with parse_special = false. The GGUF vocabulary still carries Qwen2.5's original text for those slots while the embedding rows are the ones VibeVoice trained, so tokenizing the literal text lands on the wrong rows.
• There is deliberately no generation prompt. The model emits its own <|im_start|>assistant\n header and the session strips that leading triple, as asr_server.cpp does. Without the strip the transcript is assistant\nConcord returned….
• Upstream builds ceil(samples / 3200) pad tokens but prefills only min(pads, frames), so emitting exactly frames pads yields the same sequence.
• RMSNorm epsilon is 1e-5 everywhere in the encoder, including norms the checkpoint metadata labels 1e-6, because upstream hardcodes it and the published weights were validated that way.
• The converter left-pads the 7-tap depthwise kernels to a SIMD-friendly width; convolving with the padded width and a matching causal left pad is bit-exact with the unpadded kernel, so geometry is read from weight shapes rather than metadata.

Packaging

The published package is two GGUFs — 703 MB encoder, 993 MB decoder — plus the tokenizer. A --model directory with more than one GGUF is rejected by require_selected_source, and a single component GGUF has no embedded spec, so this uses the convention minimax_h3 already documents: --model <one component>.gguf --model-spec-override model_specs. No framework change, and no merged GGUF that would diverge from what upstream publishes. Say the word if you'd rather have a single merged model.gguf with an embedded spec and I'll add the packaging step.

tools/community_models/convert_vibeasr_gguf.py (renamed from convert_vibeasr_vae.py, since the same fix serves both halves) rewrites the 4-byte type field per tensor info and copies everything else byte for byte — data offsets, data section, and KV block untouched. Flags: --in-place, --output, --list, --check.

Parity

Four LibriSpeech clips, greedy on both sides, against VibeASR.cpp's own asr_infer --greedy on the same two GGUFs:

Clip	VibeASR.cpp	this port
test-clean 6930-75918-0000	Concord returned to its place amidst the tents.	identical
test-clean 6930-75918-0001	The english forwarded to the french baskets of flowers, … (43 tokens)	identical
test-other 7902-96591-0001	Don't cry, he said. I was obliged to come.	identical
test-other 7902-96591-0000	I'm from the cut or lying off the coast.	I'm from the cutter lying off the coast.

Three of four match token for token. The fourth diverges because these clips are 16 kHz and the resamplers differ: this port uses audio.cpp's vibevoice_asr front end (soxr, plus the max_abs > 1 clamp), upstream uses naive linear interpolation and no clamp. The reference transcript is I AM FROM THE CUTTER LYING OFF THE COAST, so this port is the one that gets it right, but the point is that the difference is the front end and not the graph. A clip already at 24 kHz skips resampling and does not have this failure mode. Both divergences are commented at the point they happen.

Encoder-level, against reference F32 dumps from VibeASR.cpp's own vae_encode_acoustic / vae_encode_semantic, 3.505 s clip at its native 16 kHz, 17 frames × 1536:

Branch	max abs	mean abs	cosine
acoustic	1.478 (12.1% of range)	0.0930 (0.76% of range)	0.99238739
semantic	2.526 (9.5% of range)	0.1804 (0.68% of range)	0.98475210

Stage 0 is bit-exact, every int8 byte and every scale — that is what pins the layouts, the causal padding, the kernel padding, and the weight mapping. The first divergence is 5 of 1,794,560 elements one int8 step apart at an identical scale, entering stage 1, and it grows because each remaining stage requantizes.

Bit-exactness is not reachable and the tolerances say so, which I'd rather state than paper over. The scale convention differs in the last float bit (multiplier vs reciprocal, see #447), and upstream rounds ties to even in its vector body but away from zero in its scalar tail, so no single convention reproduces it. To calibrate: nudging one input sample by one int8 step and re-running VibeASR.cpp against itself moves its own output by cosine 0.99592 / 0.98700 — the graph amplifies a single LSB about as far as the two implementations differ. The probe gates on mean-abs-relative ≤ 2% and cosine ≥ 0.98; anything tighter would test rounding luck.

One more: output_format=json returns an empty transcript on short single-speaker clips, because the model emits an immediate end-of-turn. I checked upstream on the same input before assuming it was my bug — asr_infer --prompt-format json is also empty. Documented, not worked around.

Performance

Release, gcc, x86-64 AVX2, 24 vCPU EPYC 7V13. 3.505 s clip resampled to 24 kHz: 26 speech frames, 72-token prompt, 13 generated tokens.

Threads	encoder (both branches)	prefill	decode	wall	RTF
8	758 ms	214 ms	239 ms	1285 ms	0.367
1	4652 ms	1415 ms	941 ms	7081 ms	2.020

RTF 0.367 at 8 threads, so within the community-model bar. The encoder dominates — it runs twice, once per branch, over raw samples rather than tokens. Decode is ~18 ms/token at 8 threads.

Peak RSS 2.20 GB against 1.70 GB of weights, because BackendWeightStore stages each tensor before upload. Graph arenas are 64 MB (encoder) + 256 MB (prefill) + 256 MB (decode) and all three are session options (vibeasr.encoder_graph_arena_mb etc.). Memory is stable across requests — the graphs are built once per session and reused.

Tests

tests/vibeasr/test_vibeasr_asr.cpp (test_vibeasr_asr) loads through the real registry, creates an {Asr, Offline} session, and requires raw equality with the reference transcript, punctuation and casing included; a normalized compare runs only to localize a failure.

tests/vibeasr/test_vibeasr_vae_encoder.cpp (test_vibeasr_vae_encoder) checks shape, finiteness, and frame count, and with --reference-acoustic / --reference-semantic reports the parity numbers above and fails outside the tolerances.

Both exit 125 (SKIP) when the checkpoint is not installed, so neither needs weights in CI. Full ctest with the checkpoints present: 70/70 pass, 0 failures (2 pre-existing parakeet skips for missing assets). python3 tools/check_loader_catalog_sync.py reports runtime loaders, model_specs, and model_manager_v2 in sync.

Reproduce:

cmake -B build -DCMAKE_BUILD_TYPE=Release -DENGINE_BUILD_MODEL_TESTS=ON
cmake --build build -j
python3 tools/community_models/convert_vibeasr_gguf.py --input models/vibeasr/vibeasr-vae-encoder-i8_s.gguf --in-place
python3 tools/community_models/convert_vibeasr_gguf.py --input models/vibeasr/vibeasr-lm-i2_s-embed-q6_k.gguf --in-place
ctest --test-dir build -R vibeasr --output-on-failure

Docs

docs/community_models/vibeasr.md covers the architecture, both parity sections, performance, the packaging convention, and the limitations (CPU only, offline only, greedy only, text only). README.md, docs/community_models/models.md, and docs/asr.md get the family row and the cross-reference to vibevoice_asr.

Upstream

• Model port: https://github.com/microsoft/VibeASR.cpp (src/vae.cpp, src/lm.cpp, src/asr_server.cpp, utils/prompt_builder.h)
• Base model: VibeVoice ASR, also in tree as vibevoice_asr
• Weights: https://huggingface.co/XsquirrelC/VibeVoice-ASR-BitNet

[0xShug0]

@XsquirrelC Thanks for the update! I review PRs in a queue so #447 has been merged early. Could you rebase and upload the missing model? We’re good to go after that.