sanotts: sanoTTS voice family community model (seven voices, 294k-2.27M params, en/vi/id, GGUF FP32) - #449
Open
voidash wants to merge 7 commits into
Open
sanotts: sanoTTS voice family community model (seven voices, 294k-2.27M params, en/vi/id, GGUF FP32)#449voidash wants to merge 7 commits into
voidash wants to merge 7 commits into
Conversation
Two GGUF packages from ampixa/sanoTTS on Hugging Face: heart-nano (294,279 parameters, int8, 357 KB) as the default, and heart (2,272,145, f32, 9.1 MB). Both are converted losslessly from the blobs the project already ships. Rebuilding those blobs from the GGUF reproduces them byte for byte, and the golden gate on the rebuilt weights gives the same correlation against the float PyTorch references as the originals -- 0.989703 and 1.000000 against a 0.98 threshold -- including when the GGUF is fetched from Hugging Face rather than built locally. Session options mirror inflect_v2's, since sanoTTS needs the same external eSpeak-ng phonemizer and must not embed it.
Groundwork for a ggml-native sanoTTS community model.
model_specs/sanotts.json heart-nano (294,279 params, 24 kHz) from
ampixa/sanoTTS on Hugging Face. Session options
mirror inflect_v2's, since sanoTTS needs the same
external eSpeak-ng phonemizer.
assets.{h,cpp} Reads config.json and the GGUF tensors. config.json
carries (tensor, offset) regions emitted by the
packaging tool, so nothing here translates region
names and the two cannot drift. Every shape constant
is compared against the build's own, because a
lineage mismatch would read weights at the wrong
offsets and synthesize noise rather than fail.
frontend.{h,cpp} Text -> the 62-symbol phoneme ids the model was
trained on: eSpeak-ng IPA, then misaki's E2M rewrite,
then the character-level vocabulary. Ported from the
project's own JavaScript and Python front ends so all
three agree symbol for symbol. eSpeak-ng is opened at
runtime and never linked -- it is GPL-3.0 and must
not be embedded here, the same treatment inflect_v2
gives it.
The packaging is verified upstream: rebuilding both weight blobs from the GGUF
reproduces the originals byte for byte, and the golden gate on the rebuilt
weights matches the float PyTorch reference at 0.989703 against a 0.98
threshold -- including when the GGUF is fetched from Hugging Face.
The inference graph is next, built on the framework's module library rather
than a vendored runtime, so sanoTTS gets the shared backends like every other
model here.
Three cached graphs (duration, token stage, frame stage + ConvNeXt decoder) with the reference implementations' exact semantics: ATen-compatible MT19937 noise, torch.linspace/expand_features float behavior, LayerNorm eps 1e-6, erf GELU, torch.istft trim, and the 0.9973-pole DC blocker. The frontend gains phonemizer-compatible punctuation preservation and the correct eSpeak-ng tie mode so token streams are byte-identical to the Python front end. Verified against the project's numpy reference (same text, seed, and eSpeak-ng build): correlation 0.999999985, identical sample count; the reference is itself gated 0.987-1.000 against float PyTorch. 38.7 s of audio renders in 0.22 s wall on CPU (peak RSS 76 MB). Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we
The codepoint chunker cannot see phoneme counts, so a dense 280-codepoint chunk can exceed the duration model's 207-token training limit. encode() now throws a typed SanoTtsTooLongError and the session splits the chunk at the whitespace nearest its middle and recurses, so the shared long-form case (6 kB of text, 6.2 minutes of audio) renders instead of failing. Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we
Same graph, wider and deeper; the runtime now derives the expected tensor count from the config instead of hardcoding heart-nano's 103, and the weight arena covers the 9.1 MB FP32 payload. Verified like heart-nano: correlation 0.999999985 against the numpy reference at identical sample count, installed end to end from the published Hugging Face package. Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we
Second graph in the family: duration and acoustic students into a 192-channel latent, then a 3-stage ConvTranspose1d decoder with dilated residual banks (kristin adds a learned post filter). Deterministic, 22.05 kHz. The shared front-end structure moves into graph_common.h; the session dispatches on the config's graph field. The piperlite front end reproduces Piper's convention exactly: untied eSpeak-ng phonemes through the phonemizer punctuation pipeline, NFD decomposition to codepoints, the per-voice phoneme_id_map with [BOS, PAD, (id, PAD)..., EOS] framing, the schwa fallback for ids outside a component's trained vocab, and regional-variant-first voice selection (phonemizer rejects bare language codes on espeak-ng >= 1.49, so 'en' must resolve to en-us in both stacks). All five voices verified against the project's numpy reference with the same eSpeak-ng build: correlation >= 0.99999996 at identical sample counts, installed end to end from the published Hugging Face packages. Vietnamese and Indonesian exercise their own espeak voices and language validation. Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we
voidash
changed the title
The duration and token stages run under identical tensor names in both lineages, so their four graph builders collapse into one shared build_front_graph over a small stage spec; the frame-level acoustic stage both decoders open with becomes acoustic_frame_stage. BackendState now carries the backend/weights/CUDA-duration-mirror boilerplate once, a cached_graph template replaces the six cache accessors, and the host-side feature builders and duration rounding are shared (rounding unified to double, which is what the numpy references both runtimes are gated against actually compute). All seven voices re-verified after the refactor: correlations unchanged (>= 0.99999996) at identical sample counts; unit test and the shared long-form case pass. Net -363 lines. Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the sanoTTS voice family as a community model: seven very small offline TTS voices (294k–2.27M parameters; English, Vietnamese, Indonesian), the smallest of which also runs on microcontrollers. Upstream project: https://github.com/Ampixa/sanoTTS · Packages: ampixa/sanoTTS
gguf/— FP32 tensors in PyTorch shapes withaudiocpp.tensor_ranks/shapesexact-shape metadata and embedded model specs, fully standalone.Two graphs share the
sanottsfamily, dispatched by the package config:HostLogMagnitudePhaseISTFT. A seed picks one of many valid renderings.Implementation
inflect_v2pattern throughout:SpecBackedVoiceModelConfig,BackendWeightStore,CacheSlots-cached graphs keyed by token/frame count, framework modules (LayerNormModuleeps 1e-6,GeluModuleExactErf,LinearModule,EmbeddingModule,ggml_conv_1d_dw,ConvTranspose1dModulewith the CPU padding construction). Shared front-end structure lives in an internalgraph_common.h.dlopenat runtime, never linked — same treatment asinflect_v2. The nano front end reproduces the phonemizer punctuation-preservation pipeline plus the misaki E2M rewrite (exact tie-mode bits); the piperlite front end reproduces Piper's convention (untied phonemes, NFD decomposition to codepoints, per-voicephoneme_id_map,[BOS, PAD, (id, PAD)…, EOS]framing, schwa fallback for out-of-vocab ids, and regional-variant-first voice selection matching phonemizer's behavior on espeak-ng ≥ 1.49).torch.linspace/expand_featuresfloat behaviour, ties-to-even duration rounding,torch.istftcentre trim, the reference'sH(z) = (1 − z⁻¹)/(1 − 0.9973 z⁻¹)DC blocker (nano).inflect_v2TF32 rationale; CUDA itself is untested here — this PR claims CPU only.Verification (Ubuntu x86-64, gcc, 12 cores, default 4 threads)
End to end from clean installs (
tools/model_manager_v2.py install sanotts_<voice>_orig), each voice compared against the project's numpy reference (same text, same seed where applicable, same systemlibespeak-ng.so.1; the references are themselves gated ≥ 0.987 against the float PyTorch models):Max sample delta ~1.7e-05 in every case — the WAV's own int16 quantisation step.
Token stream parity: byte-identical phoneme id streams vs the Python front end on a 10-text punctuation corpus (quotes, parentheses, hyphens, ellipses, apostrophes, clock times), plus identical duration decisions on every parity run above.
Long-form (the shared case from
tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json, 6 kB of text): heart-nano renders 373 s of audio in 1.3 s wall (~283× real time, 220 MB peak RSS); amy renders 394 s in 18.5 s (~21× real time, 497 MB peak — the piperlite decoder convolves at audio rate). A chunk that phonemizes past a voice's token limit is bisected at whitespace and recursed.Determinism: nano's default seed derives from each chunk's text (
sha256(text)[:8], the reference convention); piperlite voices are deterministic and document the seed as ignored.Checks:
tools/check_loader_catalog_sync.pypasses;sanotts_frontend_testpasses under-DENGINE_BUILD_TESTS=ON -DENGINE_BUILD_MODEL_TESTS=ON; per-stage timings and graph-cache hits are traced via--log.Limitations
apt install espeak-ng/brew install espeak-ng) or pointed to viasanotts.espeak_library_path/sanotts.espeak_data_pathsession options.https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we