Local Voice Clone TTS

Local-first CLI for generating Japanese speech from text using a short reference recording of your own voice. The MVP targets Qwen3-TTS local inference through the qwen-tts Python package and does not use cloud TTS APIs.

Use this only with voices you own or voices for which you have explicit consent. Reference audio is sensitive biometric-adjacent data: keep it local, do not upload it, and do not commit it.

Hardware Target

• Ubuntu or WSL2 Ubuntu
• NVIDIA RTX 3090, 24 GB VRAM
• Python 3.12
• CUDA-capable PyTorch for GPU inference

Models:

Qwen/Qwen3-TTS-12Hz-0.6B-Base   # fast smoke tests
Qwen/Qwen3-TTS-12Hz-1.7B-Base   # higher quality

Phase 1: CPU Tests And Dry Run

Create and activate the local environment:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[test]"

Optional conda setup:

conda create -n voiceclone python=3.12
conda activate voiceclone
python -m pip install --upgrade pip
python -m pip install -e ".[test]"

Optional uv setup:

uv venv --python 3.12
source .venv/bin/activate
uv pip install -e ".[test]"

Run CPU-only checks:

.venv/bin/python -m pytest
.venv/bin/python -m compileall src
.venv/bin/voiceclone inspect-env
.venv/bin/voiceclone inspect-env --json

Run a dry-run without a model download or private reference audio:

.venv/bin/voiceclone synth \
  --provider qwen3 \
  --model Qwen/Qwen3-TTS-12Hz-0.6B-Base \
  --ref-audio data/refs/ref_medical_01.wav \
  --ref-text data/refs/ref_medical_01.txt \
  --input data/scripts/antibiotic_case_01.speech.txt \
  --out out/final/antibiotic_case_01_qwen06b.wav \
  --language Japanese \
  --dry-run

Phase 2: GPU Dependency Installation

The helper script installs only into .venv and never uses sudo:

scripts/setup_gpu_ubuntu.sh

It performs reviewable steps for:

• CUDA-enabled PyTorch install
• qwen-tts install
• optional flash-attn manual install command
• sound/audio dependency notes

The default PyTorch command in the script targets CUDA 12.1 wheels:

.venv/bin/python -m pip install --upgrade torch --index-url https://download.pytorch.org/whl/cu121

If this does not match your driver, use the PyTorch selector and install the wheel it recommends. flash-attn is optional; if it fails to build, skip it and pass --disable-flash-attn during synthesis.

After installation, inspect the environment:

.venv/bin/voiceclone inspect-env
.venv/bin/voiceclone inspect-env --json

Important fields:

• nvidia_smi_available and nvidia_driver_visible
• torch_import_ok, torch_version, cuda_available, cuda_version
• gpu_name, gpu_capability, bf16_supported
• recommended_dtype
• flash_attn_import_ok
• qwen_tts_import_ok, qwen_tts_version
• disk_free_bytes

Phase 3: Place And Prepare Private Reference Audio

Place your own reference audio and exact transcript under data/refs/:

data/refs/ref_medical_01.m4a
data/refs/ref_medical_01.txt

Convert the .m4a to the normalized local WAV used by synthesis:

.venv/bin/voiceclone prepare-ref \
  --input-audio data/refs/ref_medical_01.m4a \
  --input-text data/refs/ref_medical_01.txt \
  --name ref_medical_01 \
  --out-dir data/refs \
  --sample-rate 24000 \
  --mono

This creates:

data/refs/ref_medical_01.wav

It leaves the original .m4a and existing transcript in place. It refuses to replace an existing WAV unless --overwrite is supplied. Inspect the prepared reference without printing transcript content:

.venv/bin/voiceclone inspect-ref \
  --audio data/refs/ref_medical_01.wav \
  --text data/refs/ref_medical_01.txt

Guidelines:

• Use your own voice, or a voice with explicit consent.
• Keep the transcript as close as possible to the spoken audio.
• Use clean audio with minimal background noise.
• Do not upload reference audio anywhere.
• Do not commit data/refs/*.

Phase 3B: Repair And Smoke-Test Qwen Runtime

If inspect-env reports libcudart.so.13 while the NVIDIA driver is 575.x, keep the host driver unchanged and repair the Python package stack back to CUDA 12.1-compatible PyTorch packages:

scripts/repair_qwen_runtime_cu121.sh

The repair script uses the existing .venv, does not use elevated privileges, snapshots package state under reports/, pins torch==2.5.1, torchaudio==2.5.1, and torchvision==0.20.1 from the cu121 PyTorch wheel index, then reinstalls qwen-tts==0.1.1 without letting it replace torch packages. It does not download model weights or run synthesis.

Run import-only smoke tests before synthesis:

.venv/bin/voiceclone inspect-env
.venv/bin/voiceclone qwen-smoke
.venv/bin/voiceclone qwen-smoke --json

Only proceed to real synthesis after qwen_tts_import_ok: True.

Phase 4: Run 0.6B Generation

Print the manual smoke-test sequence:

.venv/bin/voiceclone print-gpu-smoke-test

Then run the real 0.6B smoke test manually:

.venv/bin/voiceclone inspect-env
.venv/bin/voiceclone synth \
  --provider qwen3 \
  --model Qwen/Qwen3-TTS-12Hz-0.6B-Base \
  --ref-audio data/refs/ref_medical_01.wav \
  --ref-text data/refs/ref_medical_01.txt \
  --input data/scripts/antibiotic_case_01.speech.txt \
  --out out/final/antibiotic_case_01_qwen06b.wav \
  --language Japanese

If flash-attn is not installed or fails:

.venv/bin/voiceclone synth \
  --provider qwen3 \
  --model Qwen/Qwen3-TTS-12Hz-0.6B-Base \
  --ref-audio data/refs/ref_medical_01.wav \
  --ref-text data/refs/ref_medical_01.txt \
  --input data/scripts/antibiotic_case_01.speech.txt \
  --out out/final/antibiotic_case_01_qwen06b.wav \
  --language Japanese \
  --disable-flash-attn

Phase 5: Run 1.7B Generation

.venv/bin/voiceclone synth \
  --provider qwen3 \
  --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --ref-audio data/refs/ref_medical_01.wav \
  --ref-text data/refs/ref_medical_01.txt \
  --input data/scripts/antibiotic_case_01.speech.txt \
  --out out/final/antibiotic_case_01_qwen17b.wav \
  --language Japanese

For lower memory pressure, try:

.venv/bin/voiceclone synth \
  --provider qwen3 \
  --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --ref-audio data/refs/ref_medical_01.wav \
  --ref-text data/refs/ref_medical_01.txt \
  --input data/scripts/antibiotic_case_01.speech.txt \
  --out out/final/antibiotic_case_01_qwen17b.wav \
  --language Japanese \
  --chunk-max-chars 120 \
  --disable-flash-attn

Phase 6: Compare Outputs And Adjust Text

Listen to the generated WAV files in out/final/. If pronunciation is poor:

1. Edit data/scripts/terms.yaml for stable replacements.
2. Regenerate data/scripts/antibiotic_case_01.speech.txt:

.venv/bin/voiceclone normalize-text \
  --input data/scripts/antibiotic_case_01.raw.txt \
  --terms data/scripts/terms.yaml \
  --out data/scripts/antibiotic_case_01.speech.txt

3. Keep medical normalization conservative; do not aggressively rewrite terms.

Batch Synthesis

.venv/bin/voiceclone batch \
  --provider qwen3 \
  --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --ref-audio data/refs/ref_medical_01.wav \
  --ref-text data/refs/ref_medical_01.txt \
  --input-dir data/scripts \
  --out-dir out/final \
  --language Japanese

Use --continue-on-error to keep processing later files after a failure.

Troubleshooting

Problem	What To Check	Action
no nvidia-smi	inspect-env shows nvidia_smi_available: False	Install or fix the NVIDIA driver on Ubuntu/WSL2. The setup script cannot do this because it does not use sudo.
torch installed but CUDA unavailable	torch_import_ok: True, cuda_available: False	Reinstall CUDA-enabled PyTorch in .venv; confirm the wheel index matches your driver.
flash-attn build failure	pip build errors for CUDA headers, compiler, or PyTorch ABI	Skip it and run with --disable-flash-attn. It is optional for MVP inference.
qwen-tts import failure	qwen_tts_available or qwen_tts_import_ok is false	Run .venv/bin/python -m pip install --upgrade qwen-tts; inspect the import error in JSON output.
Hugging Face/model download issues	model load fails before synthesis	Check network, disk space, cache location, and Hugging Face authentication if the model requires it. Tests never download models.
out-of-memory	CUDA OOM during 1.7B or long text	Try 0.6B first, lower --chunk-max-chars, close other GPU processes, or use --disable-flash-attn if flash attention is unstable.
bad pronunciation	output misreads terms or numbers	Add conservative replacements to data/scripts/terms.yaml, rerun normalize-text, and keep raw text unchanged.
reference audio mismatch	cloned voice is unstable or content leaks from prompt	Make sure ref_medical_01.txt is the exact transcript of ref_medical_01.wav; use cleaner, shorter reference audio.

Phase 4 Quality Improvement Workflow

Use this loop after baseline 0.6B and 1.7B synthesis works. First improve the reference bank, speech text, chunking, and generation parameters. Only consider fine-tuning after reproducible evaluation shows persistent defects.

Reference Bank

Keep reference metadata in a local index such as data/refs/ref_index.yaml:

refs:
  - id: ref_medical_01
    audio: data/refs/ref_medical_01.wav
    text: data/refs/ref_medical_01.txt
    style: medical_explanation
    notes: initial 8-second reference

Inspect all references without printing transcript contents:

.venv/bin/voiceclone inspect-refs --index data/refs/ref_index.yaml

Prepare .m4a entries into local WAV files while preserving originals:

.venv/bin/voiceclone prepare-refs \
  --index data/refs/ref_index.yaml \
  --sample-rate 24000 \
  --mono

Add --overwrite only when you intentionally want to replace existing WAV files.

Speech Text V2

Create a shorter, pause-friendly speech script while preserving the original raw text:

.venv/bin/voiceclone make-speech-v2 \
  --input data/scripts/antibiotic_case_01.speech.txt \
  --terms data/scripts/terms.yaml \
  --out data/scripts/antibiotic_case_01.speech.v2.txt

This also creates data/scripts/antibiotic_case_01.notes.md for listening observations.

Audit the pronunciation terms before changing files:

.venv/bin/voiceclone terms-audit \
  --input data/scripts/antibiotic_case_01.speech.v2.txt \
  --terms data/scripts/terms.yaml

Use --write only after reviewing the proposed conservative replacements.

Chunk QA

Create a CSV template from the synthesis manifest and mark problem chunks manually:

.venv/bin/voiceclone qa-template \
  --manifest out/manifest.jsonl \
  --out reports/quality/qa_antibiotic_case_01.csv

The CSV is meant for chunk-level labels such as misread, unnatural_intonation, bad_pause, too_fast, too_slow, voice_similarity, noise, truncation, or other.

Regenerate One Chunk

Preview a selected chunk regeneration without loading the model:

.venv/bin/voiceclone regenerate-chunk \
  --provider qwen3 \
  --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --ref-audio data/refs/ref_medical_01.wav \
  --ref-text data/refs/ref_medical_01.txt \
  --input data/scripts/antibiotic_case_01.speech.v2.txt \
  --chunk-id 3 \
  --out out/final/antibiotic_case_01_qwen17b_regen.wav \
  --language Japanese \
  --disable-flash-attn \
  --dry-run

Run the actual single-chunk repair manually after checking the preview:

.venv/bin/voiceclone regenerate-chunk \
  --provider qwen3 \
  --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --ref-audio data/refs/ref_medical_01.wav \
  --ref-text data/refs/ref_medical_01.txt \
  --input data/scripts/antibiotic_case_01.speech.v2.txt \
  --chunk-id 3 \
  --out out/final/antibiotic_case_01_qwen17b_regen.wav \
  --language Japanese \
  --disable-flash-attn

If reusable chunks from the previous successful run are missing, add --regenerate-missing.

Experiment Matrix

Dry-run a controlled matrix first. This prints planned runs and does not import or load Qwen:

.venv/bin/voiceclone experiment \
  --models Qwen/Qwen3-TTS-12Hz-0.6B-Base,Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --refs data/refs/ref_index.yaml \
  --input data/scripts/antibiotic_case_01.speech.v2.txt \
  --out-dir out/experiments \
  --chunk-max-chars 80,120,180 \
  --seeds 1,2 \
  --language Japanese \
  --disable-flash-attn \
  --dry-run

Run a small real experiment only when you are ready for model loading and generation:

.venv/bin/voiceclone experiment \
  --models Qwen/Qwen3-TTS-12Hz-0.6B-Base \
  --refs data/refs/ref_index.yaml \
  --input data/scripts/antibiotic_case_01.speech.v2.txt \
  --out-dir out/experiments \
  --chunk-max-chars 120 \
  --seeds 1 \
  --language Japanese \
  --disable-flash-attn \
  --gen temperature=0.7 \
  --run

Generation kwargs use --gen key=value and are passed to generate_voice_clone.

Compare Outputs

After an experiment run creates out/experiments/<timestamp>/, build a Markdown listening sheet:

.venv/bin/voiceclone compare-report \
  --experiment-dir out/experiments/<timestamp> \
  --out reports/quality/compare_<timestamp>.md

Use the report to rate pronunciation, intonation, naturalness, voice similarity, and overall quality before deciding on more reference recordings or text edits.

Final Target Scripts

The final reading targets are tracked under data/scripts/ with separate raw and speech-oriented files:

• final_01_heart_failure.raw.txt and final_01_heart_failure.speech.txt
• final_02_polypharmacy.raw.txt and final_02_polypharmacy.speech.txt
• final_03_antibiotic.raw.txt and final_03_antibiotic.speech.txt

data/scripts/final_targets.yaml lists the target ids, titles, source paths, speech paths, and expected 1.7B output WAV paths. Keep raw text unchanged; make pronunciation or pacing fixes in the .speech.txt files and conservative mappings in data/scripts/terms.yaml.

Dry-run all final targets without loading the model or generating audio:

scripts/dry_run_final_targets.sh

Generate all three final 1.7B WAV files manually:

scripts/generate_final_targets_17b.sh

Expected outputs:

out/final/final_01_heart_failure_qwen17b.wav
out/final/final_02_polypharmacy_qwen17b.wav
out/final/final_03_antibiotic_qwen17b.wav

The generation script refuses to overwrite existing outputs. To intentionally regenerate them:

OVERWRITE=1 scripts/generate_final_targets_17b.sh

After listening, fill out:

reports/quality/final_targets_listening_checklist.md

If a final target has read errors, update only the relevant .speech.txt phrasing or add conservative entries to data/scripts/terms.yaml. Do not add clinical facts, remove uncertainty, or erase family conflict, resource limitation, or treatment-choice context. Reference audio under data/refs/ and generated outputs under out/ are ignored and should not be committed.

Phase 6: Fluency-First Final Audio

Historical note: CustomVoice and VoiceDesign were evaluated as fluency-first options, but they are now experimental fallback paths only. The primary final-generation path is Qwen3-TTS Base personal voice clone with stable clone call modes.

Base clone uses Qwen/Qwen3-TTS-12Hz-1.7B-Base with --ref-audio and --ref-text. CustomVoice uses preset speakers, not personal voice cloning. VoiceDesign creates a designed narrator from an instruction prompt. For this project, keep both as experimental comparisons rather than the recommended final workflow.

Inspect expected CustomVoice speaker names without loading model weights:

.venv/bin/voiceclone qwen-speakers --json

Dry-run the CustomVoice final targets:

scripts/dry_run_final_targets_customvoice.sh

Generate the CustomVoice final targets manually:

scripts/generate_final_targets_customvoice_17b.sh

Dry-run the VoiceDesign final targets:

scripts/dry_run_final_targets_voicedesign.sh

Generate the VoiceDesign final targets manually:

scripts/generate_final_targets_voicedesign_17b.sh

The CustomVoice script uses Ono_Anna, data/instructs/medical_narration_ja.txt, paragraph mode, temperature=0.45, top_p=0.85, and 500 ms inter-chunk silence. The VoiceDesign script uses data/instructs/voice_design_medical_ja.txt with the same generation settings. Both real generation scripts refuse to overwrite existing WAVs unless OVERWRITE=1 is set.

Run a fluency experiment matrix without loading Qwen:

.venv/bin/voiceclone experiment \
  --models Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
  --task-modes custom_voice \
  --speakers Ono_Anna \
  --instruct-files data/instructs/medical_narration_ja.txt,data/instructs/medical_narration_warm_ja.txt,data/instructs/medical_narration_slow_ja.txt \
  --input data/scripts/final_03_antibiotic.speech.txt \
  --out-dir out/experiments \
  --generation-modes paragraph,whole \
  --seeds 1,2 \
  --language Japanese \
  --disable-flash-attn \
  --gen temperature=0.45 \
  --gen top_p=0.85 \
  --dry-run

Compare CustomVoice, VoiceDesign, and Base clone outputs in reports/quality/final_targets_listening_checklist.md. Score pronunciation, fluency, prosody, pause naturalness, speed, voice consistency, clinical appropriateness, and overall quality. Return to personal voice cloning only if the final narration needs the user's voice strongly enough to justify more reference-bank work or later fine-tuning. Consider fine-tuning only after repeatable listening notes show persistent defects that prompt, chunking, speaker choice, and generation parameters cannot solve.

Phase 7: Stable Base Clone Final Generation

The primary final-generation path is again Qwen3-TTS Base personal voice clone:

Qwen/Qwen3-TTS-12Hz-1.7B-Base
task_mode=clone
ref_audio=data/refs/ref_medical_01.wav
ref_text=data/refs/ref_medical_01.txt
x_vector_only_mode=False

CustomVoice and VoiceDesign are experimental fallback paths only. They remain in the repository for comparison, but they are not the primary path for this project. They are not recommended for the final workflow because observed output quality can change between sentences and prosody can be inconsistent.

Stable clone modes are available with --clone-call-mode:

• batch_chunks: build one voice prompt, pass a text list to one generate_voice_clone call, then join returned wavs. This is the recommended first path.
• whole: build one voice prompt and call generate_voice_clone once for the entire file. Use as a comparison; if it fails, fall back to batch_chunks.
• paragraph: build one prompt and generate one paragraph at a time.
• per_chunk: old fallback behavior.

Dry-run the recommended stable Base clone outputs:

bash scripts/dry_run_final_targets_clone_stable.sh

Generate the recommended stable Base clone outputs manually:

scripts/generate_final_targets_clone_stable_17b.sh

Dry-run whole-file clone comparison:

bash scripts/dry_run_final_targets_clone_whole.sh

Generate whole-file clone comparison manually:

scripts/generate_final_targets_clone_whole_17b.sh

Compare optional reference recordings without failing if they are absent:

bash scripts/dry_run_final_targets_clone_refs.sh

Audit final outputs after manual generation:

.venv/bin/voiceclone final-output-audit --final-dir out/final

Use reports/quality/final_targets_listening_checklist.md to compare stable Base clone, whole clone, prior clone, and the experimental CustomVoice/VoiceDesign outputs. Use reports/quality/reference_audio_plan.md before recording additional references.

Phase 8: Silence Diagnosis And Reading Patches

Fix post-processing first, then fix reading patches, then compare new references, then regenerate. Base clone remains the primary path; CustomVoice and VoiceDesign remain experimental only.

silence_ms is handled as milliseconds. The expected conversion is:

silence_samples = int(sample_rate * silence_ms / 1000)

At 24 kHz, silence_ms=600 inserts 14,400 samples, or 0.6 seconds. It must not become 600 seconds.

Audit final WAVs for long leading, trailing, or internal silence:

bash scripts/audit_final_outputs.sh

Or audit one file:

.venv/bin/voiceclone audio-audit \
  --audio out/final/final_01_heart_failure_clone_stable_17b.wav

Trim leading and trailing silence without modifying the original:

.venv/bin/voiceclone trim-silence \
  --input out/final/final_01_heart_failure_clone_stable_17b.wav \
  --out out/final/final_01_heart_failure_clone_stable_17b_trimmed.wav \
  --max-leading-silence-ms 300 \
  --max-trailing-silence-ms 800

Batch-trim known final clone outputs to _trimmed.wav files:

bash scripts/trim_final_outputs.sh

Refresh the private reference index after adding recordings, then prepare and inspect references without printing transcript contents:

bash scripts/prepare_and_inspect_refs.sh

Record remaining kanji misreads in data/scripts/reading_patches.yaml. Keep patches disabled until reviewed. Audit patch coverage without printing full script text:

.venv/bin/voiceclone reading-patch-audit \
  --targets data/scripts/final_targets.yaml \
  --patches data/scripts/reading_patches.yaml

Apply enabled patches to create .speech.v2.txt files while preserving raw and original speech files:

.venv/bin/voiceclone apply-reading-patches \
  --targets data/scripts/final_targets.yaml \
  --patches data/scripts/reading_patches.yaml \
  --suffix speech.v2

Dry-run corrected v2 generation. The dry-run script falls back to original speech files if v2 files are absent and reports that fallback:

bash scripts/dry_run_final_targets_clone_stable_v2.sh

Generate corrected v2 final audio manually only after patches are reviewed. The real script fails if v2 files are missing:

scripts/generate_final_targets_clone_stable_v2_17b.sh

Compare references after refreshing data/refs/ref_index.yaml:

bash scripts/dry_run_final_targets_clone_refs.sh

Generated audio, reference audio, and data/refs/ref_index.yaml are ignored and should not be committed.

Privacy

This project has no telemetry. It does not upload reference audio or text. Never commit .env, data/refs/*, generated audio, or API keys. Manifest records store hashes and paths, not full transcripts by default.

Phase 9: Final V3 Scripts

The v3 scripts are the current preferred scripts for final Base clone generation. speech.v3.user.txt preserves the user's exact updated wording. speech.v3.txt is the TTS-stabilized version with only conservative numeral and known reading-risk normalization.

Base clone remains the primary path. CustomVoice and VoiceDesign remain experimental fallback paths only.

Dry-run the recommended paragraph-mode v3 workflow first:

bash scripts/dry_run_final_targets_clone_v3_17b.sh

Generate paragraph-mode v3 WAV files manually:

bash scripts/generate_final_targets_clone_v3_17b.sh

After generation, audit silence and duration:

bash scripts/audit_final_outputs.sh
.venv/bin/voiceclone audio-audit --glob "out/final/*clone_v3_17b.wav"

Trim leading/trailing silence without destroying originals if needed:

.venv/bin/voiceclone trim-silence   --input out/final/final_01_heart_failure_clone_v3_17b.wav   --out out/final/final_01_heart_failure_clone_v3_17b_trimmed.wav   --max-leading-silence-ms 300   --max-trailing-silence-ms 800

If paragraph mode still drifts, dry-run and try whole mode:

bash scripts/dry_run_final_targets_clone_v3_whole_17b.sh
bash scripts/generate_final_targets_clone_v3_whole_17b.sh

If whole mode fails, runs out of memory, adds long silence, or degrades prosody, use paragraph mode as the final path.

Use reports/quality/final_v3_listening_checklist.md to score pronunciation, fluency, prosody, pause naturalness, voice consistency, trailing silence, and overall quality.

Phase 10: Best Mode And V4 Reading Patches

V3 listening selected the best current generation mode per file:

• final_01_heart_failure: whole mode is preferred.
• final_02_polypharmacy: paragraph mode is preferred.
• final_03_antibiotic: paragraph mode is preferred.

CustomVoice and VoiceDesign remain experimental only. The best current path is to keep v3 as the baseline, select the preferred v3 outputs when no further reading correction is needed, and create v4 only for confirmed kanji or phrase misreads.

terms.yaml alone is not sufficient unless those replacements are applied to the speech text passed to synthesis. The actual v4 synthesis input is speech.v4.txt, produced by applying data/scripts/reading_patches_v4.yaml to the v3 stabilized scripts.

Audit v4 patch coverage without printing full script text:

.venv/bin/voiceclone reading-patch-audit   --targets data/scripts/final_targets_v3.yaml   --patches data/scripts/reading_patches_v4.yaml

Apply enabled patches to create speech.v4.txt files. With all placeholders disabled, this still creates v4 files identical to v3:

.venv/bin/voiceclone apply-reading-patches   --targets data/scripts/final_targets_v3.yaml   --patches data/scripts/reading_patches_v4.yaml   --suffix speech.v4

Dry-run mixed-mode v4 generation:

bash scripts/dry_run_final_targets_clone_v4_mixed_17b.sh

Generate mixed-mode v4 manually:

bash scripts/generate_final_targets_clone_v4_mixed_17b.sh

Audit selected best and v4 outputs:

bash scripts/audit_best_and_v4_outputs.sh

If v4 is not needed, select the preferred v3 outputs manually:

bash scripts/select_best_final_v3.sh

The real generation and selection scripts refuse to overwrite outputs unless OVERWRITE=1 is set.

Phase 11: Reference Audio Quality Repair

If generated outputs contain motorcycle- or engine-like noise across many files, first suspect the reference audio. Base clone remains the primary path; CustomVoice and VoiceDesign remain experimental only.

Re-recording is preferred over denoising. Use this recording protocol for new references:

• 10 to 20 seconds.
• Quiet room.
• Same microphone.
• No fan, traffic, engine noise, music, or room echo where possible.
• No re-takes inside one file.
• Exact transcript.
• Calm final narration style.

Audit current references without printing transcript contents:

bash scripts/prepare_and_inspect_refs.sh
.venv/bin/voiceclone ref-noise-audit --index data/refs/ref_index.yaml

Create conservative cleaned reference variants locally. Mild cleaning converts to 24 kHz mono, applies a high-pass filter, and uses light loudness normalization. Aggressive denoise can reduce voice quality or create artifacts, so compare it carefully before use.

bash scripts/clean_refs_mild.sh
bash scripts/build_clean_ref_index.sh
.venv/bin/voiceclone ref-noise-audit --index data/refs/ref_index_clean.yaml

Dry-run short reference probes. This uses the same short text for every original and cleaned ref:

bash scripts/dry_run_ref_probe_17b.sh

Generate probe WAVs manually:

bash scripts/generate_ref_probe_17b.sh

Use reports/quality/ref_probe_listening_checklist.md to reject refs with engine-like noise first, then choose the most stable and natural voice. Prefer newly recorded clean refs over denoised refs if quality is similar.

Generate final outputs with the selected reference by setting REF_AUDIO, REF_TEXT, and REF_ID:

REF_AUDIO=data/refs/clean/ref_medical_01_clean_mild.wav REF_TEXT=data/refs/clean/ref_medical_01_clean_mild.txt REF_ID=ref_medical_01_clean_mild bash scripts/dry_run_final_targets_clone_mixed_v4_17b_with_ref.sh

REF_AUDIO=data/refs/clean/ref_medical_01_clean_mild.wav REF_TEXT=data/refs/clean/ref_medical_01_clean_mild.txt REF_ID=ref_medical_01_clean_mild bash scripts/generate_final_targets_clone_mixed_v4_17b_with_ref.sh

The final script keeps mixed modes: final_01 uses whole mode, while final_02 and final_03 use paragraph mode. Real probe and final generation scripts refuse overwrite unless OVERWRITE=1 is set.

Phase 12: GPT-SoVITS Evaluation Scaffold

Gemma 4 is not used as the local TTS generation path for this project. Gemini TTS is separate and can only be treated as a cloud baseline if you choose to compare it manually later.

The active local model paths are:

1. Qwen3-TTS 1.7B-Base improved path, which remains implemented and primary.
2. GPT-SoVITS zero-shot as a local comparison path.
3. GPT-SoVITS few-shot fine-tuning only if the zero-shot probe is promising.

GPT-SoVITS setup is conservative and manual. Print the setup plan first:

bash scripts/setup_gpt_sovits.sh

After reviewing the upstream GPT-SoVITS README and the generated notes, clone the repository only when ready:

ACTUALLY_RUN=1 bash scripts/setup_gpt_sovits.sh

The setup helper does not use sudo, does not download model weights, and does not start WebUI automatically. It writes notes to:

reports/model_comparison/gpt_sovits_setup_notes.md

Prepare the shared Qwen vs GPT-SoVITS probe plan without running inference:

bash scripts/prepare_model_comparison_probe.sh

After GPT-SoVITS is installed, inspect local entrypoints and follow the manual zero-shot WebUI workflow:

bash scripts/run_gpt_sovits_zero_shot_probe.sh

Before starting the WebUI, verify the installed GPT-SoVITS imports with the repository root and package directory on PYTHONPATH:

bash scripts/check_gpt_sovits_imports.sh

Start the Japanese inference WebUI with the same PYTHONPATH setup:

bash scripts/start_gpt_sovits_inference_webui.sh

For VS Code port-forwarded use on port 9872:

bash scripts/start_gpt_sovits_vscode_9872.sh

GPT-SoVITS Japanese Inference Error

If Japanese inference fails with Mecab_load() Cannot open ... open_jtalk_dic_utf_8-1.11 or RuntimeError: Failed to initalize Mecab, pyopenjtalk cannot find the OpenJTalk dictionary. Install the dictionary into the gpt-sovits conda environment:

bash scripts/setup_openjtalk_dict_gpt_sovits.sh

The startup scripts export OPEN_JTALK_DICT_DIR to:

$CONDA_PREFIX/share/open_jtalk/open_jtalk_dic_utf_8-1.11

Check Japanese G2P without starting the WebUI:

bash scripts/check_gpt_sovits_japanese_g2p.sh

GPT-SoVITS Zero-Shot Sounds Non-Native

If GPT-SoVITS zero-shot works but sounds like a non-native speaker of Japanese, first improve the prompt and comparison method before judging the model. Qwen3-TTS Base remains the current baseline.

Record native-style Japanese medical prompt references from:

reports/model_comparison/gpt_sovits_japanese_ref_plan.md

Use the GPT-SoVITS-specific probe text:

data/scripts/ref_probe_gpt_sovits.no_comma_split.txt

List available reference inputs without printing transcript contents:

bash scripts/list_gpt_sovits_probe_inputs.sh

In the WebUI, use Japanese / ja for both prompt and target language, avoid Auto or cross-lingual settings, and avoid comma-level segmentation. Prefer sentence or paragraph segmentation. Track the comparison in:

reports/model_comparison/gpt_sovits_zero_shot_settings_checklist.md
reports/model_comparison/gpt_sovits_zero_shot_probe_matrix.md

Move to few-shot fine-tuning only if native-style zero-shot references still sound foreign-accented while voice similarity is otherwise acceptable. The few-shot preparation notes are in:

reports/model_comparison/gpt_sovits_fewshot_next_steps.md

Use:

data/scripts/ref_probe.speech.txt

as the shared probe text. Export the GPT-SoVITS probe to:

out/model_compare/gpt_sovits/ref_medical_01_probe.wav

The detailed local WebUI workflow is in:

reports/model_comparison/gpt_sovits_zero_shot_workflow.md

Do not start few-shot training until zero-shot output is good enough to justify it. The preparation plan is in:

reports/model_comparison/gpt_sovits_fewshot_data_plan.md

Use the comparison checklist after generating probes:

reports/model_comparison/model_comparison_checklist.md

Compare Qwen3-TTS 1.7B-Base v4 mixed, GPT-SoVITS zero-shot, and only later GPT-SoVITS few-shot on reading accuracy, intonation, voice consistency, background noise, engine-like artifacts, natural pauses, long-form stability, voice similarity, clinical narration suitability, and overall quality.