Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -630,6 +630,19 @@ audiocpp_add_model(personaplex
engine::models::personaplex::make_personaplex_loader
)

audiocpp_add_model(sanotts
SOURCES
src/community_models/sanotts/assets.cpp
src/community_models/sanotts/frontend.cpp
src/community_models/sanotts/piper_runtime.cpp
src/community_models/sanotts/runtime.cpp
src/community_models/sanotts/session.cpp
INCLUDES
engine/community_models/sanotts/session.h
LOADERS
engine::models::sanotts::make_sanotts_loader
)

audiocpp_add_model(inflect_v2
SOURCES
src/community_models/inflect_v2/assets.cpp
Expand Down Expand Up @@ -2545,6 +2558,13 @@ if (ENGINE_BUILD_TESTS)
COMMAND echo_tts_host_units
)

add_engine_unittest(sanotts_frontend_test tests/unittests/test_sanotts_frontend.cpp)
target_include_directories(sanotts_frontend_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/tests/unittests)
add_test(
NAME sanotts_frontend_test
COMMAND sanotts_frontend_test
)

add_engine_unittest(inflect_v2_frontend_test tests/unittests/test_inflect_v2_frontend.cpp)
target_include_directories(inflect_v2_frontend_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/tests/unittests)
add_test(
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ Community model ports live under `community_models` to make the ownership bounda
| **moss_voicegen** | Voice Design | en, zh | GGUF | Joost [@jrohde](https://github.com/jrohde) | [MOSS-VoiceGenerator](docs/community_models/moss_voicegen.md) speech in a voice designed from a written instruction |
| **outetts** | TTS, Clone | en, ar, zh, nl, fr, de, it, ja, ko, lt, ru, es, pt, be, bn, ka, hu, lv, fa, pl, sw, ta, uk | GGUF | Mirek [@mirek190](https://github.com/mirek190) | Llama-OuteTTS-1.0-1B TTS and voice cloning support |
| **parakeet_tdt** | ASR | auto, bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, ru, sk, sl, sv, uk | GGUF F32/16/Q8, Stream | [@dleiferives](https://github.com/dleiferives) | [Parakeet-TDT 0.6B v3](docs/community_models/parakeet_tdt.md) offline, long-form, and buffered-streaming ASR support |
| **sanotts** | TTS | en, vi, id | GGUF FP32 | Ashish [@voidash](https://github.com/voidash) | [sanoTTS voice family](docs/community_models/sanotts.md) seven voices from 294k to 2.27M parameters, native offline synthesis |
| **sense_asr** | ASR | auto, zh, en, yue, ja, ko, pt, ru, es, it, fr, de, nl, pl, tr, ar, hi, vi, th, id, ms, fa, nospeech | GGUF Q8, Stream | Jason Chen [@jasonchen31](https://github.com/jasonchen31), [@LauraGPT](https://github.com/LauraGPT) / FunASR | [SenseVoice-Small](docs/community_models/sense_asr.md) offline/streaming SAN-M + CTC transcription with event/emotion/language tags and ITN |
| **soprano_tts** | TTS | en | GGUF Q8, Stream | [@WalkingCat](https://github.com/WalkingCat) | [Soprano-1.1-80M](https://huggingface.co/WalkingCat/Soprano-1.1-80M-GGUF) ultra-lightweight TTS with Qwen3 LM + Vocos decoder |
| **vietneu_tts** | TTS, Clone | vi, en | GGUF | Phuoc [@phuocnguyen90](https://github.com/phuocnguyen90) | [VieNeu-TTS-v3-Turbo](docs/community_models/vietneu_tts.md) TTS and voice cloning support |
Expand Down
1 change: 1 addition & 0 deletions docs/community_models/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Practical expectations:
| **outetts** | TTS, voice cloning | en, ar, zh, nl, fr, de, it, ja, ko, lt, ru, es, pt, be, bn, ka, hu, lv, fa, pl, sw, ta, uk | Mirek [@mirek190](https://github.com/mirek190) | [Llama-OuteTTS-1.0-1B](outetts.md) TTS and voice cloning support |
| **voxcpm1** | TTS, voice cloning | zh, en, ja, ko | Community | [VoxCPM1](voxcpm1.md) tokenizer-free 0.5B TTS with 16 kHz output, streaming, and continuation-mode voice cloning |
| **parakeet_tdt** | ASR | auto, bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, ru, sk, sl, sv, uk | [@dleiferives](https://github.com/dleiferives) | [Parakeet-TDT 0.6B v3](parakeet_tdt.md) offline, long-form, and buffered-streaming ASR support |
| **sanotts** | TTS | en, vi, id | Community | [sanoTTS voice family](sanotts.md) seven voices from 294k to 2.27M parameters, FP32 offline synthesis |
| **sense_asr** | ASR | auto, zh, en, yue, ja, ko, pt, ru, es, it, fr, de, nl, pl, tr, ar, hi, vi, th, id, ms, fa, nospeech | Jason Chen [@jasonchen31](https://github.com/jasonchen31), [@LauraGPT](https://github.com/LauraGPT) / FunASR | [SenseVoice-Small](sense_asr.md) offline/streaming SAN-M + CTC transcription with event/emotion/language tags and ITN |
| **vietneu_tts** | TTS, voice cloning | vi, en | Phuoc [@phuocnguyen90](https://github.com/phuocnguyen90) | [VieNeu-TTS-v3-Turbo](vietneu_tts.md) TTS and voice cloning support |
| **moss_voicegen** | Voice design | en, zh | Joost [@jrohde](https://github.com/jrohde) | [MOSS-VoiceGenerator](moss_voicegen.md) voice design from a written instruction, on the MOSS delay architecture |
129 changes: 129 additions & 0 deletions docs/community_models/sanotts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# sanoTTS voice family

`sanotts` provides native GGML inference for the
[sanoTTS](https://github.com/Ampixa/sanoTTS) voice family — very small
text-to-speech models, the smallest of which also runs on microcontrollers.
All packages download from Hugging Face
([ampixa/sanoTTS](https://huggingface.co/ampixa/sanoTTS) `gguf/`) as
standalone FP32 GGUFs with embedded model specs. Offline FP32 inference only.

Two graphs share one family:

- **nano** — duration student → contextual acoustic student → mel-100 →
noise-fed ConvNeXt-1D decoder → [log-magnitude | phase] head → inverse
STFT. 24 kHz. A seed picks one of many valid renderings.
- **piperlite** — duration student → contextual acoustic student → 192-channel
latent → 3-stage ConvTranspose1d decoder with dilated residual banks →
tanh waveform. 22.05 kHz. Fully deterministic (no seed).

| Package | Voice | Graph | Params | Language | Notes |
|---|---|---|---:|---|---|
| `sanotts_heart_orig` | heart | nano | 2,272,145 | en | best quality of the nano pair |
| `sanotts_heart_nano_orig` | heart-nano | nano | 294,279 | en | microcontroller-class |
| `sanotts_amy_orig` | amy | piperlite | 1,454,284 | en | Piper-distilled |
| `sanotts_hfc_orig` | hfc | piperlite | 1,834,380 | en | largest piperlite voice |
| `sanotts_kristin_orig` | kristin | piperlite | 1,396,151 | en | carries a learned post filter |
| `sanotts_vi_orig` | vi | piperlite | 1,565,484 | vi | Vietnamese |
| `sanotts_id_orig` | id | piperlite | 1,562,124 | id | Indonesian |

## Install

Install eSpeak-ng and its voice data first. On Debian or Ubuntu:

```bash
sudo apt install espeak-ng libespeak-ng1
```

On macOS:

```bash
brew install espeak-ng
```

Then install any package, e.g.:

```bash
python3 tools/model_manager_v2.py install sanotts_heart_orig --models-root models
python3 tools/model_manager_v2.py install sanotts_amy_orig --models-root models
```

## Run

```bash
audiocpp_cli --task tts --family sanotts \
--model models/sanoTTS-heart-GGUF --backend cpu \
--text "Hello from sano T T S, a very small neural text to speech model." \
--out sanotts.wav
```

Swap `--model` for any installed package directory
(`models/sanoTTS-amy-GGUF`, `models/sanoTTS-vi-GGUF`, ...). The Vietnamese
and Indonesian voices accept `--language vi` / `--language id`; a session
rejects text tagged with a language the voice was not trained on.

eSpeak-ng is loaded dynamically at runtime, never linked. If it is not on the
default library path:

```bash
audiocpp_cli --task tts --family sanotts \
--model models/sanoTTS-heart-GGUF --backend cpu \
--session-option sanotts.espeak_library_path=/path/to/libespeak-ng.so \
--session-option sanotts.espeak_data_path=/path/to/espeak-ng-data \
--text "A configured eSpeak installation." --out sanotts.wav
```

## Options

- `speaking_rate` (request, 0.5..2.0, default 1.0) — duration multiplier on
the voice's tuned length scale; larger is slower.
- `seed` (request, default 0) — nano voices only: the decoder is noise-fed,
so a given seed picks one of many valid renderings. `0` derives the seed
from each text chunk as `sha256(text)[:8]`, which is what the reference
implementations do; an explicit seed advances by one per long-form chunk.
Piperlite voices are deterministic and ignore the seed.
- `text_chunk_size` (request, default 280) — maximum codepoints per long-form
chunk; chunks split on sentence punctuation first, and a chunk that
phonemizes past the voice's token limit is bisected at whitespace.

## Determinism and parity

The runtimes reproduce the reference implementations' exact semantics:

- Front ends: the phonemizer punctuation-preservation pipeline through the
same eSpeak-ng library. The nano voices add the misaki E2M rewrite with
tie characters; the piperlite voices use Piper's NFD-decompose-to-
codepoints convention, per-voice `phoneme_id_map`, `[BOS, PAD, (id, PAD)…,
EOS]` framing, and the schwa fallback for ids outside a component's
trained vocabulary.
- nano: ATen-compatible MT19937 noise (24-bit uniform, Box–Muller in blocks
of 16), torch.istft window normalisation and centre trim, and the
reference's DC blocker `H(z) = (1 - z^-1)/(1 - 0.9973 z^-1)`.
- Shared: torch.linspace / expand_features float behaviour, LayerNorm eps
1e-6 (nano), ties-to-even duration rounding.

Measured against the project's numpy references (same text, same
eSpeak-ng build), every voice: **correlation ≥ 0.99999996 with identical
sample counts**; max sample delta ~1.7e-05 is the WAV's own int16
quantisation. The numpy references are themselves gated ≥ 0.987 against the
float PyTorch models.

## Performance

CPU-only, 12-thread x86 (default 4 backend threads), FP32, the shared 6 kB
long-form text:

| Voice | Audio | Wall | vs real time | Peak RSS |
|---|---:|---:|---:|---:|
| heart-nano | 373 s | 1.3 s | ~283× | 220 MB |
| amy | 394 s | 18.5 s | ~21× | 497 MB |

The nano decoder runs at frame rate with a host iSTFT; the piperlite decoder
runs convolutions at audio rate, which is why it is heavier. Graphs are
cached per token count (duration and token stages) and per frame count
(decoder); `--log` prints cache hits and per-stage timings.

## Licensing

The sanoTTS runtimes and weights are MIT-licensed. eSpeak-ng is GPL-3.0 and
is therefore opened with `dlopen` at runtime and never linked, matching how
`inflect_v2` treats it.
22 changes: 22 additions & 0 deletions docs/tts.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
| GLM-TTS | `glm_tts` | `tts`, `clon` | [GLM-TTS](#glm-tts) |
| Inflect Micro v2 | `inflect_v2` | `tts` | [Inflect v2](#inflect-v2) |
| OuteTTS | `outetts` | `tts`, `clon` | [OuteTTS](#outetts) |
| sanoTTS voice family | `sanotts` | `tts` | [sanoTTS](#sanotts) |
| Supertonic | `supertonic` | `tts` | [Supertonic](#supertonic) |
| VieNeu-TTS | `vietneu_tts` | `tts`, `clon` | [VieNeu-TTS](community_models/vietneu_tts.md) |
| VibeVoice | `vibevoice` | `tts` | [VibeVoice](#vibevoice) |
Expand Down Expand Up @@ -768,6 +769,27 @@ See the [Inflect v2 community model guide](community_models/inflect_v2.md) for
eSpeak-ng paths, long-form behavior, source/conversion instructions, and
limitations.

## sanoTTS

sanoTTS is a family of very small offline TTS voices (English, Vietnamese,
Indonesian; 294k to 2.27M parameters) with native GGML runtimes; the
smallest voice also runs on microcontrollers. The GGUF packages are
standalone and download from Hugging Face. sanoTTS requires an external
eSpeak-ng installation:

```bash
python3 tools/model_manager_v2.py install sanotts_heart_nano_orig --models-root models

audiocpp_cli --task tts --family sanotts \
--model models/sanoTTS-heart-nano-GGUF --backend cpu \
--text "Hello from sano T T S, a very small neural text to speech model." \
--request-option speaking_rate=1.0 \
--out sanotts.wav
```

See the [sanoTTS community model guide](community_models/sanotts.md) for
eSpeak-ng paths, seed semantics, parity evidence, and performance numbers.

## Supertonic

Supertonic 3 is a preset-voice multilingual TTS model. It does not use external speaker references in the current integration.
Expand Down
91 changes: 91 additions & 0 deletions include/engine/community_models/sanotts/assets.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#pragma once

#include "engine/framework/assets/resource_bundle.h"
#include "engine/framework/assets/tensor_source.h"

#include <array>
#include <cstdint>
#include <filesystem>
#include <memory>
#include <string>
#include <unordered_map>
#include <vector>

namespace engine::models::sanotts {

struct SanoTtsConfig {
int64_t vocab_size = 62;
int64_t sample_rate = 24000;
int64_t hop_length = 256;
int64_t n_fft = 1024;
int64_t mels = 100;
int64_t dim = 0;
int64_t blocks = 0;
int64_t pw_hidden = 0;
int64_t noise_channels = 4;
int64_t dw_kernel = 7;
int64_t embed_kernel = 7;

int64_t duration_hidden = 0;
int64_t duration_depth = 0;
int64_t duration_kernel = 5;
int64_t duration_max_tokens = 207;
int64_t duration_max_frames = 80;

int64_t acoustic_hidden = 0;
int64_t acoustic_token_depth = 0;
int64_t acoustic_depth = 0;
int64_t acoustic_kernel = 5;

std::string voice;
};

enum class SanoTtsGraph {
Nano, // mel-100 -> ConvNeXt-1D -> iSTFT, noise-fed (heart, heart-nano)
Piperlite, // 192-ch latent -> 3-stage ConvTranspose1d, deterministic (amy, ...)
};

struct SanoTtsPiperConfig {
std::string voice;
std::string language; // short code the session validates against: en, vi, id
std::string espeak_voice;
int64_t sample_rate = 22050;
double duration_length_scale = 1.0;

int64_t duration_vocab = 0;
int64_t duration_hidden = 0;
int64_t duration_depth = 0;
int64_t duration_kernel = 5;
int64_t duration_max_tokens = 0;
int64_t duration_max_frames = 0;

int64_t acoustic_vocab = 0;
int64_t acoustic_hidden = 0;
int64_t acoustic_depth = 0;
int64_t acoustic_token_depth = 0;
int64_t acoustic_kernel = 5;
int64_t acoustic_out_channels = 0;

std::array<int64_t, 4> channels = {0, 0, 0, 0};
std::array<std::vector<int64_t>, 3> stage_branches;
int64_t post_filter_channels = 0;
int64_t post_filter_layers = 0;
int64_t post_filter_kernel = 9;
double post_filter_scale = 0.0;

/** Piper phoneme_id_map: one UTF-8 codepoint -> id. */
std::unordered_map<std::string, int32_t> phoneme_id_map;
};

struct SanoTtsAssets {
assets::ResourceBundle resources;
SanoTtsGraph graph = SanoTtsGraph::Nano;
SanoTtsConfig config; // valid when graph == Nano
SanoTtsPiperConfig piper; // valid when graph == Piperlite
std::shared_ptr<const assets::TensorSource> weights;
};

std::shared_ptr<const SanoTtsAssets> load_sanotts_assets(
const std::filesystem::path & model_path);

} // namespace engine::models::sanotts
86 changes: 86 additions & 0 deletions include/engine/community_models/sanotts/frontend.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#pragma once

#include <cstdint>
#include <stdexcept>
#include <filesystem>
#include <memory>
#include <string>
#include <unordered_map>
#include <vector>

namespace engine::models::sanotts {

/** Thrown by encode() when a chunk phonemizes past the duration model's
* token limit; the session responds by bisecting the chunk. */
struct SanoTtsTooLongError : std::runtime_error {
using std::runtime_error::runtime_error;
};

struct SanoTtsEncoded {
std::vector<int32_t> token_ids;
std::string dropped; // symbols outside the vocabulary, for tracing
};

/**
* Text -> the 62-symbol phoneme ids the sanoTTS front end was trained on.
*
* eSpeak-ng produces the IPA; misaki's E2M table then rewrites it into the
* character-level inventory this model uses. Both steps are reproduced from
* the project's own JavaScript and Python front ends so the three agree
* symbol for symbol.
*
* eSpeak-ng is opened at runtime and never linked, matching how inflect_v2
* treats it: it is GPL-3.0 and must not be embedded in this project.
*/
class SanoTtsFrontend {
public:
SanoTtsFrontend(
std::filesystem::path espeak_library_path,
std::filesystem::path espeak_data_path,
int64_t max_tokens);
~SanoTtsFrontend();

[[nodiscard]] SanoTtsEncoded encode(const std::string & text) const;

/** Long-form splitting on sentence punctuation, then a codepoint budget. */
[[nodiscard]] static std::vector<std::string> split_text(
const std::string & text,
int64_t max_codepoints);

/** Pause inserted between chunks, longer after a sentence end. */
[[nodiscard]] static double boundary_pause_seconds(const std::string & chunk);

private:
struct Impl;
std::unique_ptr<Impl> impl_;
int64_t max_tokens_;
};

/**
* Text -> Piper phoneme ids for the piperlite voices.
*
* Reproduces piper's phonemize_espeak / phonemes_to_ids convention through
* the same eSpeak-ng library: phonemizer-style punctuation preservation,
* NFD decomposition to single codepoints, then the voice's phoneme_id_map
* with [BOS, PAD, (id, PAD)..., EOS] framing. Deterministic per voice.
*/
class SanoTtsPiperFrontend {
public:
SanoTtsPiperFrontend(
std::filesystem::path espeak_library_path,
std::filesystem::path espeak_data_path,
std::string espeak_voice,
std::unordered_map<std::string, int32_t> phoneme_id_map,
int64_t max_tokens);
~SanoTtsPiperFrontend();

[[nodiscard]] SanoTtsEncoded encode(const std::string & text) const;

private:
struct Impl;
std::unique_ptr<Impl> impl_;
std::unordered_map<std::string, int32_t> id_map_;
int64_t max_tokens_;
};

} // namespace engine::models::sanotts
Loading
Loading