Skip to content

fix(tts): honor per-scene speed on ElevenLabs and Gemini - #39

Merged
shreyaskarnik merged 1 commit into
mainfrom
fix/tts-speed-elevenlabs-gemini
Aug 18, 2026
Merged

fix(tts): honor per-scene speed on ElevenLabs and Gemini#39
shreyaskarnik merged 1 commit into
mainfrom
fix/tts-speed-elevenlabs-gemini

Conversation

@shreyaskarnik

Copy link
Copy Markdown
Owner

Fixes #38 (the second half — the transcribe.js lazy import is handled by #36).

The bug

TTSEngineOptions.speed is honored by kokoro, openai, sarvam, mlx-audio and transformers. ElevenLabs and Gemini both render server-side, have no rate parameter, and post-convert through convertToWav() — which ignored the option entirely. A manifest asking for speed: 1.2 got 1x audio, with no warning that the request went nowhere.

Reported by @salir-admin with a working local patch; this is that approach plus the range handling it was missing.

The fix

convertToWav(buffer, speed) now applies atempo. atempo rather than asetrate because it resamples in the time domain — speeding up does not raise the pitch.

A single atempo instance is clamped to 0.5–2.0 and ffmpeg exits non-zero outside it. Since convertToWav uses execFileSync, that surfaces as a hard throw mid-TTS. So buildAtempoChain() splits anything larger across several instances whose product is the requested speed — speed: 3 becomes atempo=2,atempo=1.5.

Input guard

guardSpeed() throws on 0, negatives, NaN and Infinity. This is not defensive padding: the chain divides by each stage factor until the remainder lands in range, so those values never converge. Left unguarded they hang TTS in a synchronous loop — no output, no error, no timeout (vitest cannot interrupt a sync loop either; the RED for these cases was an actual hang).

Merely extreme values clamp to 0.25–4.0 with a warning rather than throwing — past two stages either way the voice stops being intelligible, so the intent is still legible even when the number is silly.

Notes

  • The clip cache key already includes speed, so cached clips do not go stale once this lands.
  • No README change needed — this restores documented behavior rather than adding any.
  • Engine-level tests asserting ElevenLabs/Gemini forward the option are not included: on main neither SDK is installed, so neither is mockable. build(deps)!: make TTS engine SDKs optional peer dependencies #36 makes them devDependencies; worth adding on top of that.

Test plan

16 new tests in tests/tts/speed.test.ts, written test-first:

  • chain arithmetic — no filter at 1x, single stage in range, chained stages for 3x and 0.25x, every stage within ffmpeg's 0.5–2.0 limit
  • input guard — throws on 0/negative/NaN/Infinity, names the offending value, caps at 4x and 0.25x with a warning, stays quiet in range
  • convertToWav — no -filter:a by default, correct chain passed to ffmpeg when a speed is requested

Full suite: 667 pass. The 3 failures are pre-existing and environmental (no Playwright browser binary on this machine); they are unrelated to this change and pass in CI.

`TTSEngineOptions.speed` was honored by kokoro, openai, sarvam, mlx-audio and
transformers, but silently dropped by ElevenLabs and Gemini — both render
server-side with no rate parameter and post-convert via `convertToWav()`,
which ignored the option entirely. A manifest asking for `speed: 1.2` got 1x
audio with no warning.

`convertToWav()` now takes a speed and applies `atempo`, chosen over
`asetrate` because it resamples in the time domain and so does not shift
pitch. A single `atempo` instance is clamped to 0.5-2.0, so
`buildAtempoChain()` splits larger changes across several instances whose
product is the requested speed.

`guardSpeed()` rejects 0, negatives, NaN and Infinity: the chain divides by
each stage factor until the remainder lands in range, so those values never
converge and would hang TTS synchronously with no output and no error.
Merely extreme values clamp to 0.25-4.0 with a warning instead.

The clip cache key already includes speed, so cached clips do not go stale.

Reported-by: @salir-admin
@shreyaskarnik
shreyaskarnik merged commit b0457ad into main Aug 18, 2026
4 checks passed
@shreyaskarnik
shreyaskarnik deleted the fix/tts-speed-elevenlabs-gemini branch August 18, 2026 04:40
Joilence added a commit to Joilence/argo that referenced this pull request Aug 20, 2026
The Gemini TTS models answer with `audio/L16;codec=pcm;rate=24000`, RFC 2586
linear PCM: samples and nothing else, no RIFF header and no magic bytes.
`convertToWav` opened it with `ffmpeg -i pipe:0` and no format hint, so ffmpeg
had nothing to recognise and every clip died with "Invalid data found when
processing input". The engine could not produce audio at all.

`parseRawAudioMime` reads the format off the media type and `convertToWav`
takes it as an optional third argument, so a headerless stream is declared and
everything self-describing keeps going through ffmpeg's own probing. Reading
the rate rather than assuming 24kHz matters because guessing wrong does not
fail, it pitches and stretches the voice.

Verified against the live API as well as the suite, since the suite asserts the
argv and not the bytes: a clip comes back as 24kHz mono float32 WAV, and the
`speed` added in shreyaskarnik#39 now reaches the audio, 5.24s at 1.0 against 6.75s at 0.75.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

have to patch the npm package to make it work

1 participant