fix(tts): honor per-scene speed on ElevenLabs and Gemini - #39
Merged
Conversation
`TTSEngineOptions.speed` was honored by kokoro, openai, sarvam, mlx-audio and transformers, but silently dropped by ElevenLabs and Gemini — both render server-side with no rate parameter and post-convert via `convertToWav()`, which ignored the option entirely. A manifest asking for `speed: 1.2` got 1x audio with no warning. `convertToWav()` now takes a speed and applies `atempo`, chosen over `asetrate` because it resamples in the time domain and so does not shift pitch. A single `atempo` instance is clamped to 0.5-2.0, so `buildAtempoChain()` splits larger changes across several instances whose product is the requested speed. `guardSpeed()` rejects 0, negatives, NaN and Infinity: the chain divides by each stage factor until the remainder lands in range, so those values never converge and would hang TTS synchronously with no output and no error. Merely extreme values clamp to 0.25-4.0 with a warning instead. The clip cache key already includes speed, so cached clips do not go stale. Reported-by: @salir-admin
Joilence
added a commit
to Joilence/argo
that referenced
this pull request
Aug 20, 2026
The Gemini TTS models answer with `audio/L16;codec=pcm;rate=24000`, RFC 2586 linear PCM: samples and nothing else, no RIFF header and no magic bytes. `convertToWav` opened it with `ffmpeg -i pipe:0` and no format hint, so ffmpeg had nothing to recognise and every clip died with "Invalid data found when processing input". The engine could not produce audio at all. `parseRawAudioMime` reads the format off the media type and `convertToWav` takes it as an optional third argument, so a headerless stream is declared and everything self-describing keeps going through ffmpeg's own probing. Reading the rate rather than assuming 24kHz matters because guessing wrong does not fail, it pitches and stretches the voice. Verified against the live API as well as the suite, since the suite asserts the argv and not the bytes: a clip comes back as 24kHz mono float32 WAV, and the `speed` added in shreyaskarnik#39 now reaches the audio, 5.24s at 1.0 against 6.75s at 0.75.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #38 (the second half — the
transcribe.jslazy import is handled by #36).The bug
TTSEngineOptions.speedis honored by kokoro, openai, sarvam, mlx-audio and transformers. ElevenLabs and Gemini both render server-side, have no rate parameter, and post-convert throughconvertToWav()— which ignored the option entirely. A manifest asking forspeed: 1.2got 1x audio, with no warning that the request went nowhere.Reported by @salir-admin with a working local patch; this is that approach plus the range handling it was missing.
The fix
convertToWav(buffer, speed)now appliesatempo.atemporather thanasetratebecause it resamples in the time domain — speeding up does not raise the pitch.A single
atempoinstance is clamped to 0.5–2.0 and ffmpeg exits non-zero outside it. SinceconvertToWavusesexecFileSync, that surfaces as a hard throw mid-TTS. SobuildAtempoChain()splits anything larger across several instances whose product is the requested speed —speed: 3becomesatempo=2,atempo=1.5.Input guard
guardSpeed()throws on0, negatives,NaNandInfinity. This is not defensive padding: the chain divides by each stage factor until the remainder lands in range, so those values never converge. Left unguarded they hang TTS in a synchronous loop — no output, no error, no timeout (vitest cannot interrupt a sync loop either; the RED for these cases was an actual hang).Merely extreme values clamp to 0.25–4.0 with a warning rather than throwing — past two stages either way the voice stops being intelligible, so the intent is still legible even when the number is silly.
Notes
speed, so cached clips do not go stale once this lands.mainneither SDK is installed, so neither is mockable. build(deps)!: make TTS engine SDKs optional peer dependencies #36 makes them devDependencies; worth adding on top of that.Test plan
16 new tests in
tests/tts/speed.test.ts, written test-first:convertToWav— no-filter:aby default, correct chain passed to ffmpeg when a speed is requestedFull suite: 667 pass. The 3 failures are pre-existing and environmental (no Playwright browser binary on this machine); they are unrelated to this change and pass in CI.