中文 | English

Vox Engine Framework

Local, fully offline simultaneous interpretation (Chinese ↔ English) that runs on a single 8 GB consumer GPU. A cascade of streaming ASR → sentence-level translation → streaming TTS, with cloned voices and sub-second time-to-first-audio.

Why v5.0

v5.0 is a rewrite of the real-time pipeline. The legacy Flask/Socket.IO app is still in the repository for reference, but everything under realtime/ is new: one process owns the models, the browser only captures audio and plays back what the server streams.

Stage	v5.0	Notes
ASR	faster-whisper large-v3-turbo, CTranslate2, CUDA fp16	rolling re-decode every 0.45 s, endpoint on 0.55 s silence
MT	llama.cpp llama-server + Hy-MT2-1.8B-Q4_K_M	whole committed sentences only, never partial hypotheses
TTS	gsv.cpp GPT-SoVITS V5 v5turbo (ggml/Vulkan)	4 sampling steps, 48 kHz, zero-shot voice cloning

Measured on an RTX 5060 8 GB (Windows 11):

• TTS: RTF ≈ 0.16 (6 s of speech synthesized in ~1 s), first audio chunk in ~0.5 s, ~1.2 GB VRAM
• Translation: first token in 16–78 ms on a warm 1.8 B model
• End to end: last ASR final → first TTS audio to the browser in ≈0.6 s
• Accuracy on a 24-clip labelled corpus (test/asr_eval.py): zh error 7.3 %, en 3.4 %

Architecture

mic ──► browser (16 kHz PCM over WebSocket)
          │
          ▼
   realtime/server.py ──► WhisperStreamingASR (faster-whisper, CUDA)
          │                     │ partial → UI only
          │                     ▼ final
          │              OpenAICompatibleMT (llama.cpp) ──► translation deltas → UI
          │                     │ complete clause
          ▼                     ▼
      TtsBridge ──► gsv.cpp /tts (stream) ──► float32 chunks ──► browser playback

Design decisions that matter:

• Partials are display-only. Translation always receives the committed sentence, so punctuation and wording never come from unstable hypotheses.
• Clause-level TTS. Translation deltas are buffered and flushed at clause boundaries, so a long sentence starts speaking before its translation finishes.
• One model instance per process. ASR, MT and TTS are shared across WebSocket connections; the first synthesis is warmed up at boot so nobody pays the Vulkan shader compile cost.
• Voice cloning from any 3–30 s clip. Uploads are trimmed, levelled and resampled to 32 kHz, then transcribed with the same Whisper model for the prompt text.

Requirements

• Windows 10/11, Python 3.11+
• NVIDIA GPU with CUDA 12.8+ (tested on RTX 5060 8 GB, sm_120)
• ~12 GB disk for models
• Optional: Vulkan 1.2+ driver for the gsv.cpp engine

Setup

1. Python environment

python -m venv .venv
.\.venv\Scripts\activate
pip install -r requirements-realtime.txt

requirements-realtime.txt pulls nvidia-cublas-cu12 / nvidia-cuda-runtime-cu12, which CTranslate2 needs on a machine whose toolkit is CUDA 13.

2. Models

models/
├── stt/faster-whisper-large-v3-turbo/   # CTranslate2 Whisper
├── translate/Hy-MT2-1.8B-Q4_K_M.gguf    # translation GGUF
└── tts/
    ├── references/en_ref.wav + .txt     # default cloned voices
    ├── references/zh_ref.wav + .txt
    └── references/custom/               # user voices (created at runtime)

Whisper and the GGUF can be fetched from ModelScope / Hugging Face. The TTS engine ships its own GGUF set — see below.

3. TTS engine (gsv.cpp)

Download the Windows package from KakaruHayate/GSV.CPP, extract it under tools/gsv-cpp-pkg, then start the text sidecar and the engine:

cd tools\gsv-cpp-pkg
set GSV_G2PW_MODEL=%CD%\runtime\models\frontend\g2pw
set GSV_BERT_DIR=%CD%\runtime\models\frontend\bert_tokenizer
set NLTK_DATA=%CD%\runtime\models\frontend\nltk_data
runtime\python\python.exe runtime\frontend\text_sidecar.py --port 9882

runtime\gsv\gsv_server.exe --models "%CD%\runtime\models" ^
    --device vulkan --sidecar 127.0.0.1:9882 --port 9880 --allow-path *

Pass --models as an absolute path: relative paths break voice manifests in 0.1.x builds.

4. Translation backend

tools\llama\bin\llama-server.exe -m models\translate\Hy-MT2-1.8B-Q4_K_M.gguf ^
    --host 127.0.0.1 --port 8080 -ngl 99 -c 4096 -np 1 --jinja

Any OpenAI-compatible server works; point --mt-url at it.

5. Realtime server

.\.venv\Scripts\python.exe -m realtime.server --port 8000 ^
    --mt-url http://127.0.0.1:8080/v1 ^
    --asr-model models\stt\faster-whisper-large-v3-turbo ^
    --tts --tts-backend gsv-cpp --tts-url http://127.0.0.1:9880

Open http://127.0.0.1:8000. Click 开始, speak, and the source transcript, translation and synthesized speech stream in.

Command line

python -m realtime.server accepts:

Flag	Default	Purpose
--host / --port	127.0.0.1:8000	HTTP + WebSocket bind
--asr-model	models/stt/faster-whisper-large-v3-turbo	CTranslate2 model dir
--device / --compute-type	cuda / float16	must stay float16 on Blackwell
--silence	0.5	seconds of silence that commits an utterance
--mt-url / --mt-model	http://127.0.0.1:8080/v1 / default	OpenAI-compatible endpoint
--glossary	–	JSON {"source": "target"}; also biases ASR vocabulary
--context	–	usage setting injected into the translation prompt
--tts / --tts-backend	off / gsv-cpp	gsv-cpp (V5, fast) or gsv (GSV-TTS-Lite fallback)
--tts-url	http://127.0.0.1:9880	gsv.cpp service
--tts-steps	4	v5turbo sampling steps
--tts-speed	1.0	playback speed of the cloned voice

python -m realtime --mock runs the pipeline without models, and --wav file.wav feeds a file instead of the microphone.

Benchmarks & tests

# unit tests (session ordering, partial handling)
python -m unittest discover -s test -p "test_streaming_core.py"

# ASR accuracy on the labelled corpus (CER / char error, build the corpus once)
python test/asr_corpus.py
python test/asr_eval.py                       # current pipeline
python test/asr_eval.py --legacy              # pre-v5.0 final decode, for comparison

# TTS engine throughput (first chunk, RTF)
python test/bench_tts_engine.py --steps 4

# end-to-end WebSocket probe with a WAV file
python test/ws_probe.py test/asr_eval/zh_00_snrclean.wav zh en

Project structure

realtime/
├── server.py     # aiohttp + WebSocket, voice API, TTS bridge
├── asr.py        # streaming Whisper wrapper (rolling re-decode, endpointing)
├── mt.py         # OpenAI-compatible translation, glossary + scene prompt
├── tts.py        # GsvTts (GSV-TTS-Lite) and GsvCppTTS (V5 over HTTP)
├── voices.py     # custom voice store: trim, level, transcribe, persist
├── session.py    # ASR → MT orchestration, utterance-level stats
├── audio.py      # WAV / microphone sources, resampling
└── web/          # index.html + static/app.js + static/style.css

Legacy v0.4 files (app.py, funasr_asr.py, templates/, static/) are kept for reference and are not used by realtime/.

Troubleshooting

Symptom	Fix
cublas64_12.dll is not found	pip install nvidia-cublas-cu12 nvidia-cuda-runtime-cu12; the ASR module registers the DLL directory automatically
CUDA CUBLAS_STATUS_NOT_SUPPORTED	force --compute-type float16; int8_float16 is not supported on Blackwell
No audio with a custom voice	re-select the voice after re-recording; gsv.cpp caches conditioning per reference path
Cloned voice speaks very slowly	the reference clip sets the tempo — record 4–8 s of continuous speech and keep --tts-speed 1.0
gsv.cpp service is not reachable	start gsv_server before the realtime server, or drop --tts
Traditional Chinese in the transcript	v5.0 pins zh decoding with a Simplified prompt; check initial_prompt

License

MIT — see LICENSE. Model weights keep their upstream licenses (faster-whisper: MIT, Hy-MT2: Tencent license, GPT-SoVITS: MIT).

Acknowledgments

• faster-whisper / CTranslate2
• llama.cpp
• gsv.cpp — GPT-SoVITS V5 inference in C++/ggml
• GPT-SoVITS
• GSV-TTS-Lite — PyTorch fallback backend