Integrate VibeASR.cpp into audio.cpp

[0xShug0]

Hi! I’m the maintainer of audio.cpp, a unified C++/GGML runtime supporting 50+ audio models across ASR, TTS, music generation, voice conversion, and more.

I wondering if you’d be interested in bringing it into audio.cpp. I think it would be a great fit, and having the original author involved is especially valuable since you understand the implementation and model behavior best.

Would be great to have VibeASR.cpp as part of audio.cpp!

[XsquirrelC]

Hi, thanks for the interest! Could you share more about how the integration would work — submodule, code port, or something else?

[0xShug0]

Hi, thanks for the interest! Could you share more about how the integration would work — submodule, code port, or something else?

I think the right integration path is to port it as an audio.cpp model or community model. One blocker is that VibeASR.cpp depends on custom ggml changes, and one of those changes modifies existing ggml_conv_1d behavior which affects almost all models in audio.cpp. It would be less risky if the ggml changes could be made purely additive, for example by adding new ops/types instead of modifying the behavior of existing vendored ggml paths. That’s why I reached out to your team first: you’ll have a better view of how to separate the model-specific speed path from broader ggml behavior changes. I’d appreciate two separate PRs if you’re interested: one focused on the additive ggml changes, and another for the model integration.

[XsquirrelC]

Both PRs are up, split as you asked:

• #447 — additive ggml types, fused CPU ops, kernels (1/2)
• #448 — model integration (2/2), stacked on #447

The ggml_conv_1d blocker is gone

ggml_conv_1d and ggml_im2col are untouched — the causal padding now comes from a new op, ggml_im2col_asym. No existing model's code path changes.

#447 adds GGML_TYPE_I8_S (id 42), GGML_TYPE_I2_S (id 43), and five fused ops (ggml_mul_mat_add, _relu, ggml_add_scaled, ggml_rms_norm_scaled, ggml_im2col_asym) with AVX2/NEON + scalar kernels. Only two existing functions are touched, both behind a type guard: an I2_S branch in ggml_compute_forward_mul_mat / ggml_graph_plan (8 + 10 lines), and dup/_cont carrying the I8_S in-band scale. The diff is confined to external/ggml/, two new unit tests, and their CMake registrations.

Results

ctest, #447 standalone	42/42 pass, no model code, no checkpoints
ctest, #448	70/70 pass (2 pre-existing parakeet skips)
End-to-end vs asr_infer --greedy	3 of 4 LibriSpeech clips identical token for token
Encoder cosine	0.9924 acoustic / 0.9848 semantic, stage 0 bit-exact
RTF, 8 threads	0.367 (1285 ms wall on a 3.5 s clip, 24 vCPU EPYC 7V13)
Peak RSS	2.20 GB against 1.70 GB of weights

The 4th clip differs by one word (cutter vs cut or): the clips are 16 kHz and audio.cpp resamples with soxr where upstream uses linear interpolation, enough to flip one greedy argmax. Bit-exactness isn't reachable — the two forks store the per-tensor scale as reciprocals of each other. For scale, perturbing one input sample by one int8 step moves VibeASR.cpp against itself by cosine 0.9959 / 0.9870.

Reproducing

Kernels only, no checkpoint needed — this covers all of #447:

cmake -B build -DCMAKE_BUILD_TYPE=Release -DENGINE_BUILD_TESTS=ON
cmake --build build -j --target i8_s_fused_ops_test i2_s_mul_mat_test
ctest --test-dir build -R "i8_s_fused_ops_test|i2_s_mul_mat_test" -V

i2_s_mul_mat_test compares the ternary matmul bit-exact against a plain loop; i8_s_fused_ops_test checks each fused op at 1 and 4 threads.

Then #448. VibeASR.cpp already ships both halves quantized — the converter only rewrites the type ids (that fork used 36/37, which upstream ggml had spent on the retired IQ4_NL_4_4 / IQ4_NL_4_8) and copies the rest byte for byte:

# models/vibeasr/ = the two GGUFs + tokenizer.json + tokenizer_config.json
python3 tools/community_models/convert_vibeasr_gguf.py \
    --input models/vibeasr/vibeasr-vae-encoder-i8_s.gguf --in-place
python3 tools/community_models/convert_vibeasr_gguf.py \
    --input models/vibeasr/vibeasr-lm-i2_s-embed-q6_k.gguf --in-place

cmake --build build -j --target audiocpp_cli
./build/bin/audiocpp_cli \
    --task asr --family vibeasr \
    --model models/vibeasr/vibeasr-lm-i2_s-embed-q6_k.gguf \
    --model-spec-override model_specs \
    --backend cpu --threads 8 --metrics \
    --audio assets/asr_validation/librispeech/librispeech_test_clean_6930-75918-0000.wav

text_output=Concord returned to its place amidst the tents.
metrics.rtf=0.36652

--model points at the LM GGUF because the package is two GGUFs — same shape as minimax_h3.

Limits

CPU only (no CUDA/Metal I8_S or I2_S kernels), offline and greedy only, registered as a separate vibeasr family since it shares the weights but not the numeric pipeline.

Full parity tables and the layer-by-layer divergence analysis are in docs/community_models/vibeasr.md.