feat: add native MiraTTS community model

[mirek190]

Summary

Adds MiraTTS as an experimental native community TTS/voice-cloning model.

The port runs the full pipeline in C++:

• Qwen2 autoregressive speech-token generator
• ECAPA-TDNN + Perceiver reference speaker encoder
• finite-scalar speaker quantizer
• conditional acoustic processor
• DAC waveform decoder
• learned FlashSR 16 kHz to 48 kHz upsampler

It also adds a local checkpoint converter, model-spec v1 metadata, documentation, and an upstream-reference parity harness.

Implementation notes

• Mirrors the official MiraTTS defaults: temperature 0.8, top-k 50, top-p 0.95, min-p 0.05, repetition penalty 1.2, and 1024 maximum speech tokens.
• Extends the shared Hugging Face sampler with min_p, covered by unit tests.
• Allows FlashSR to load from an existing tensor source so the converted model remains a single GGUF package.
• Preserves the upstream 8-second reference-audio limit and periodic Hann preprocessing.
• Supports BF16 parity-oriented and Q8 local GGUF conversions.
• Registers tts and clon offline routes through the normal loader/session framework.

Validation

Builds:

cmake --build build\windows-cuda-release --config Release --parallel 12
cmake --build build\windows-vulkan-release --config Release --parallel 12

Focused tests:

ctest --test-dir build\windows-cuda-release --output-on-failure -R "hf_sampler|model_spec_system|flashsr"
python tools\check_loader_catalog_sync.py --self-test
python tools\check_loader_catalog_sync.py

Results:

• CUDA Release build: passed
• Vulkan Release build: passed
• FlashSR, HF sampler, and model-spec tests: 3/3 passed
• Loader/catalog sync: passed
• Loader registration: mira_tts: tts (offline), clon (offline)
• CUDA BF16 end-to-end generation: passed
• CUDA Q8 smoke: passed
• Vulkan Q8 smoke: passed

Representative CUDA BF16 run:

.\build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task clon --family mira_tts `
  --model ..\models_v3_test\MiraTTS-GGUF\mira-tts-bf16.gguf `
  --backend cuda `
  --text "Good morning. This is a longer Mira TTS comparison generated with the original implementation and the native audio.cpp port." `
  --voice-ref reference.wav `
  --out mira-native.wav `
  --max-tokens 1024 --temperature 0.8 --top-k 50 --top-p 0.95 `
  --repetition-penalty 1.2 --request-option min_p=0.05 `
  --seed 1234 --metrics

Measured on RTX 3090:

• output: 48 kHz mono
• audio duration: 20.48 s
• wall time: 6.58 s
• RTF: 0.321
• throughput: 3.11x real-time

Reference parity using identical original speech/context tokens:

• waveform correlation: 0.9999646
• SNR: 41.15 dB
• MAE: 0.0004608
• reference preprocessing and all 32 speaker codes match

Conversion

python tools/community_models/convert_mira_tts.py /path/to/MiraTTS /path/to/mira-native
audiocpp_gguf \
  --input language_model=/path/to/mira-native/language_model.safetensors \
  --input speaker_encoder=/path/to/mira-native/speaker_encoder.safetensors \
  --input processor=/path/to/mira-native/processor.safetensors \
  --input decoder=/path/to/mira-native/decoder.safetensors \
  --input upsampler=/path/to/mira-native/upsampler.safetensors \
  --output /path/to/mira-native/mira-tts.gguf --type bf16 \
  --family mira_tts --root /path/to/mira-native

GGUF for testing
https://huggingface.co/mirek190/audio.cpp/blob/main/Text%20to%20audio%20(TTS)/mira-tts-q8.gguf

Current limitations

• Reference audio is required.

[mirek190]

Added native MiraTTS segment-level streaming in afa1b64.

The streaming session now:

• registers both tts and clon as offline|streaming
• splits input at framework text boundaries (text_chunk_size, default 160)
• encodes the reference speaker once and reuses its 32 context codes
• synthesizes and emits each completed 48 kHz segment immediately
• returns the concatenated final result after all events
• supports the normal OpenAI-style SSE speech streaming route

Validation performed:

• CUDA Release build: passed
• Vulkan Release build: passed
• focused FlashSR, HF sampler, model-spec, and CLI request tests: 4/4 passed
• loader/catalog synchronization: passed
• CLI three-sentence stream: five progressive model chunks, 22.66 s final 48 kHz mono audio
• HTTP /v1/audio/speech with stream_format=sse: eight speech.audio.delta transport events, one speech.audio.done, and [DONE]
• single-chunk offline/streaming equivalence: byte-identical WAVs (SHA256 7313AEF29FB91C8E1EFB2BB8D59AE5FEBF5C966560E6F0EBABAF7EE48D9548AF)

This is genuine progressive segment streaming. Each segment still runs the acoustic processor, DAC decoder, and FlashSR as a complete unit; token-level neural-codec streaming remains a possible future optimization.

[mirek190]

Added commit fb2150b with the MiraTTS validation and performance baseline.

Included

• long-lived native CUDA warm benchmark covering cold, repeat, changed-prompt, long-form, and post-long-form repeat requests
• separate streaming benchmark with first-event latency and progressive event checks
• deterministic hash, duration, RTF, WAV readability, and lifecycle validation
• revision-pinned upstream Python benchmark using the same request sequence
• native/upstream waveform and log-mel parity comparison tool
• documented CUDA, streaming, memory, decoder-parity, backend, and performance results

Original MiraTTS versus audio.cpp Q8

Test	Original MiraTTS	audio.cpp Q8	Difference
Cold request, 5.12 s audio	2.227 s	1.707 s	audio.cpp 1.30× faster*
Repeated request, 5.12 s	0.911 s	1.561 s	Original 1.71× faster
Second short prompt, 5.12 s	0.895 s	1.566 s	Original 1.75× faster
Long-form, 15.36 s	2.976 s	4.771 s	Original 1.60× faster
Repeat after long-form, 5.12 s	0.918 s	1.584 s	Original 1.73× faster

* The original cold request paid a one-time ONNX Runtime CUDA fallback cost; warm measurements are the representative steady-state comparison.

The three warm short requests average 0.908 s upstream and 1.570 s in native Q8, making the original implementation about 1.73× faster in this measurement. For the 15.36-second long-form case, the original achieved RTF 0.1938 (5.16× real time), while native Q8 achieved RTF 0.3106 (3.22× real time).

This is not precision-matched: the original uses BF16 while native uses GGUF Q8. The original also pre-encodes and retains the reference context, whereas native currently processes the reference input on each request.

CUDA offline — long-lived session

Request	Wall ms	Audio s	RTF	FNV-1a audio hash
clone_cold	1707.41	5.12	0.3335	426924e4695337c6
clone_repeat	1560.66	5.12	0.3048	426924e4695337c6
short_second_prompt	1566.03	5.12	0.3059	8255f4342afeb014
longform	4771.24	15.36	0.3106	5708fcccf7c25080
clone_repeat_after_longform	1583.74	5.12	0.3093	426924e4695337c6

The three identical requests remain bit-deterministic before and after the changed-prompt and long-form requests.

CUDA streaming

Request	Wall ms	First event ms	Events	Audio s	RTF	FNV-1a audio hash
stream_cold	6639.50	2489.93	4	21.14	0.3141	26eb3542a073a8db
stream_repeat	6498.52	2368.89	4	21.14	0.3074	26eb3542a073a8db

Streaming emits multiple independently consumable audio events, and the merged repeat is bit-deterministic.

Python/native decoder parity

Metric	Python	Native	Result
Sample rate	48000 Hz	48000 Hz	Match
Frame count	122880	122880	Exact match
Waveform cosine	—	—	0.99996735
Log-mel cosine	—	—	0.99943239

This exact-token comparison uses identical upstream speech and context tokens, isolating the native processor/decoder path from autoregressive sampling.

Resident memory — Q8 CUDA

Measurement	Resident usage
Process working set	1429.54 MiB
Process private bytes	5958.28 MiB
Windows GPU Process Memory dedicated usage	2551.55 MiB
nvidia-smi total device memory in use	2552 MiB

These are held-resident samples rather than instrumented peaks across model loading.

Backend coverage

Backend	Build	Runtime result	Status
CUDA, RTX 3090	Passed	Offline, streaming, determinism, and WAV validation passed	Validated
Vulkan, RTX 3090	Passed	Deterministic, but under-generated (3.44 s vs 5.12 s) and diverged on other prompts	Not parity-clean
Vulkan, AMD integrated GPU	Passed	Deterministic, but generated only 0.10 s for the 5.12 s CUDA case	Not parity-clean
CPU	Not run	No runtime measurement recorded	Untested

Validation passed for all saved CUDA offline and streaming cases.

[mirek190]

I am working on performance optimization now and seems a shared Qwen runtime bug: the cached graph’s compact-logit index tensor is overwritten by the allocator after its first use, causing an illegal CUDA access on request two uder
audio.cpp\src\framework\modules\transformers\qwen_causal_decode_runtime.cpp

[mirek190]

MiraTTS Q8 performance update

Commit 5a47c2a substantially improves the native MiraTTS Q8 path on an RTX 3090.

Test	Previous audio.cpp Q8	Optimized audio.cpp Q8	Improvement
Cold request, 5.12 s audio	1707 ms	874 ms	48.8% faster
Repeated request, 5.12 s audio	1561 ms	798 ms	48.9% faster
Second short prompt, 5.12 s audio	1566 ms	798 ms	49.0% faster
Long-form, 15.36 s audio	4771 ms	2530 ms	47.0% faster
Repeat after long-form, 5.12 s audio	1584 ms	835 ms	47.3% faster

Compared with the previously measured warm original MiraTTS implementation, optimized audio.cpp is now approximately 9–15% faster on the repeated and long-form cases. The original cold measurement includes its one-time ONNX fallback/setup cost, so the warm comparisons are the useful ones.

What changed

• Compact LM logits readback: MiraTTS can generate only its 8,192 speech-code tokens plus EOS. The native runtime now keeps the full tied output projection for model semantics, but copies and samples only that 8,193-token alphabet instead of transferring and CPU-sorting all ~166k vocabulary logits at every autoregressive step.
• CUDA DAC fast path enabled: the MiraTTS decoder graph now declares the active backend when it is built. It had silently been treated as a CPU-style graph description and therefore missed audio.cpp's optimized CUDA ConvTranspose1d path. DAC execution for 5.12 seconds of output fell from roughly 375 ms to 32 ms.
• Reference voice reuse: encoded speaker context is cached in a bounded session cache. The default is one reference; --session-option reference_cache_slots=<n> adjusts it and 0 disables it.
• Stable compact prefill handling: compact-logit index tensors are declared as graph inputs, and compact prefill graphs are rebuilt safely to avoid the reused-CUDA-graph illegal-memory-access observed on the second request. Rebuilding costs about 1 ms while preserving the much larger logits-readback saving.
• Stage profiling: prompt, generator, processor, DAC build/upload/compute/readback, FlashSR, reference hashing, and reference encoding timings are now recorded.
• Memory-conscious decoder behavior: a retained two-shape DAC graph cache was tested but rejected. It saved only about 8 ms while increasing private process memory by roughly 2.6 GB after short- and long-form requests, so the final patch does not keep those graph buffers resident.

Validation

Check	Result
Exact output frame counts	Pass, all 5 offline cases
Minimum waveform cosine vs previous native output	0.9999929
Minimum log-mel cosine vs previous native output	0.9999813
Offline cold/repeat determinism	Pass
Streaming repeat determinism	Pass
Streaming output	21.46 s in 3.46 s, RTF 0.161, 4 events
CUDA build/runtime	Pass
Vulkan build/resident-session runtime	Pass

The CUDA fast ConvTranspose path changes floating-point operation ordering, so WAV hashes differ from the slower path; the strict frame-count and high waveform/log-mel similarity checks confirm that the resulting audio remains effectively equivalent.

[0xShug0]

@mirek190 Could you isolate the Qwen3 runtime changes into a separate PR? It would make it easier to track commits that introduce behavioral changes. The only affected model by the change is Music3 and I'm not sure about the perf impact on low end GPUs. Another solution is probably not to disable reuse. It should mark the token-id tensor as input, then re-upload prefill_logits_readback_token_ids_ in run_prefill(), the same way the runtime already re-uploads positions and attention mask before cached graph compute.

BTW, do you know how to stop people from flooding the repo with draft PRs? Pull request limits do not work.

[mirek190]

@mirek190 Could you isolate the Qwen3 runtime changes into a separate PR? It would make it easier to track commits that introduce behavioral changes. The only affected model by the change is Music3 and I'm not sure about the perf impact on low end GPUs. Another solution is probably not to disable reuse. It should mark the token-id tensor as input, then re-upload prefill_logits_readback_token_ids_ in run_prefill(), the same way the runtime already re-uploads positions and attention mask before cached graph compute.

Sure

#426

BTW, do you know how to stop people from flooding the repo with draft PRs? Pull request limits do not work.

If they do not listen ... ban them ;)
I saw it. It looked like a spam.

It is not worth to giving them attention.