Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@ repos:
args: ["--branch=main"]

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.12
rev: v0.16.6
hooks:
- id: ruff
args: ["--fix"]
- id: ruff-format

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v2.0.0
rev: v2.3.1
hooks:
- id: mypy
additional_dependencies:
Expand Down
22 changes: 10 additions & 12 deletions docs/audio_generation_v2_design.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,10 @@ Where it sits in the pipeline: called inside `ScriptGenerator._post_process_turn
```python
@dataclass
class PhraseProsody:
text_span: str # exact substring to match in text
rate: str | None # SSML rate value, e.g. "+15%" or "fast"
pitch: str | None # e.g. "+5st"
volume: str | None # e.g. "+6dB"
text_span: str # exact substring to match in text
rate: str | None # SSML rate value, e.g. "+15%" or "fast"
pitch: str | None # e.g. "+5st"
volume: str | None # e.g. "+6dB"
break_before_ms: int = 0
break_after_ms: int = 0
```
Expand Down Expand Up @@ -145,10 +145,9 @@ class SpeakerState:
rate_offset: float = 1.0
pitch_offset_st: float = 0.0
volume_offset_db: float = 0.0
breathiness_level: float = 0.0 # 0.0 = modal; 1.0 = maximum breathiness
breathiness_level: float = 0.0 # 0.0 = modal; 1.0 = maximum breathiness

def update(self, new_intensity: int, speaker_role: str) -> None:
...
def update(self, new_intensity: int, speaker_role: str) -> None: ...
```

The state's outputs feed into `render_utterance()` as additional offsets that stack on top of the SSML parameters derived from `style_map`.
Expand Down Expand Up @@ -206,8 +205,7 @@ class TurnGapController:
current_turn: DialogueTurn,
prev_turn: DialogueTurn | None,
rng: random.Random,
) -> float:
...
) -> float: ...
```

---
Expand All @@ -225,9 +223,9 @@ class TurnGapController:
A new `MixMode` enum in `SceneMixer`:
```python
class MixMode(Enum):
SEQUENTIAL = "sequential" # current behavior
OVERLAP = "overlap" # next starts before prev ends
BARGE_IN = "barge_in" # prev is cut off; next starts over
SEQUENTIAL = "sequential" # current behavior
OVERLAP = "overlap" # next starts before prev ends
BARGE_IN = "barge_in" # prev is cut off; next starts over
```

`TurnGapController` returns both a gap value and a `MixMode`. High-intensity transitions use `BARGE_IN` probabilistically (e.g., 30% of AGG→VIC transitions at I4+, 50% at I5).
Expand Down
50 changes: 25 additions & 25 deletions docs/audio_generation_v3_design.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,8 @@ Approach (three layers, applied in order):
**Schema change — `DialogueTurn` gains two new fields:**

```python
text_original: str # canonical LLM output, never modified
text_spoken: str # text actually sent to TTS after all normalization
text_original: str # canonical LLM output, never modified
text_spoken: str # text actually sent to TTS after all normalization
normalization_rules_triggered: list[str] # rule IDs that fired, empty if none
```

Expand Down Expand Up @@ -189,17 +189,18 @@ New data types:
```python
@dataclass
class PhraseHint:
phrase_id: str # unique within the turn, e.g. "t3_p1"
phrase_id: str # unique within the turn, e.g. "t3_p1"
hint: Literal["stress", "slow", "break_before", "break_after", "menace"]
char_start_original: int # offset in text_original
char_start_original: int # offset in text_original
char_end_original: int


@dataclass
class PhraseProsody:
phrase_id: str
rate: str | None # SSML rate, e.g. "+15%" or "slow"
pitch: str | None # e.g. "+3st"
volume: str | None # e.g. "+6dB"
rate: str | None # SSML rate, e.g. "+15%" or "slow"
pitch: str | None # e.g. "+3st"
volume: str | None # e.g. "+6dB"
break_before_ms: int = 0
break_after_ms: int = 0
```
Expand Down Expand Up @@ -233,10 +234,9 @@ class SpeakerState:
rate_offset: float = 1.0
pitch_offset_st: float = 0.0
volume_offset_db: float = 0.0
breathiness_level: float = 0.0 # 0.0 = modal; 1.0 = maximum
breathiness_level: float = 0.0 # 0.0 = modal; 1.0 = maximum

def update(self, new_intensity: int, speaker_role: str) -> None:
...
def update(self, new_intensity: int, speaker_role: str) -> None: ...

def to_metadata_dict(self) -> dict[str, float]:
# Serialized into per-turn metadata for reproducibility
Expand Down Expand Up @@ -310,9 +310,9 @@ Values are seeded-RNG draws for reproducibility. The `TurnGapController` is inst

```python
class MixMode(Enum):
SEQUENTIAL = "sequential" # current behavior
OVERLAP = "overlap" # next starts before prev ends; both play through
BARGE_IN = "barge_in" # prev plays through its speech end then is cut with a 60 ms fade
SEQUENTIAL = "sequential" # current behavior
OVERLAP = "overlap" # next starts before prev ends; both play through
BARGE_IN = "barge_in" # prev plays through its speech end then is cut with a 60 ms fade
```

**Speech-end anchoring (#66).** TTS engines pad each utterance with 100–300 ms of trailing near-silence. Anchoring overlap onset against the WAV's *file end* therefore puts the overlap inside the silence — listeners hear the previous speaker stop, then a gap, then the new speaker start, sounding like polite turn-taking instead of an interruption.
Expand Down Expand Up @@ -384,10 +384,10 @@ Add a `PreprocessingConfig` dataclass controlling which steps are applied:
```python
@dataclass
class PreprocessingConfig:
resample: bool = True # always True in practice
downmix_mono: bool = True # always True
lowpass_hz: float | None = 7500 # None to skip; 7500 Hz is correct for 16 kHz output
wiener_denoise: bool = True # configurable; default off for Tier A
resample: bool = True # always True in practice
downmix_mono: bool = True # always True
lowpass_hz: float | None = 7500 # None to skip; 7500 Hz is correct for 16 kHz output
wiener_denoise: bool = True # configurable; default off for Tier A
normalization: NormalizationMode = NormalizationMode.PER_TURN_RMS
silence_pad_s: float = 0.5
```
Expand Down Expand Up @@ -447,9 +447,9 @@ The `AzureProvider` / `TTSProvider` ABC must be extended with a `ProviderCapabil
@dataclass
class ProviderCapabilities:
supports_ssml: bool
supports_style_tags: bool # <mstts:express-as>
supports_phoneme_tags: bool # <phoneme alphabet="ipa">
supports_api_emotion_sliders: bool # ElevenLabs stability / style_exaggeration
supports_style_tags: bool # <mstts:express-as>
supports_phoneme_tags: bool # <phoneme alphabet="ipa">
supports_api_emotion_sliders: bool # ElevenLabs stability / style_exaggeration
max_volume_delta_db: float | None # None = unlimited
```

Expand All @@ -471,14 +471,14 @@ This does not require a new pipeline stage. It is a refactor of `SSMLBuilder` an
```python
@dataclass
class GenerationMetadata:
pipeline_version: str # e.g. "v3.0"
tts_backend: str # e.g. "azure", "google"
voice_family: str # e.g. "he-IL-AvriNeural"
pipeline_version: str # e.g. "v3.0"
tts_backend: str # e.g. "azure", "google"
voice_family: str # e.g. "he-IL-AvriNeural"
text_normalization_version: str # version of the disambiguation lexicon
prosody_controller_version: str
timing_controller_version: str
mix_mode_used: str # dominant MixMode for the scene
normalization_strategy: str # e.g. "per_turn_rms_v1"
mix_mode_used: str # dominant MixMode for the scene
normalization_strategy: str # e.g. "per_turn_rms_v1"
breathiness_applied: bool
speaker_state_serialized: dict # final SpeakerState for each speaker
```
Expand Down
8 changes: 5 additions & 3 deletions docs/research/research_report_gpt-5.2-thinking.md
Original file line number Diff line number Diff line change
Expand Up @@ -679,6 +679,7 @@ In Python (numpy):
```python
import numpy as np


def crossfade(a, b, n=160): # 10ms at 16kHz
fade = np.linspace(0, 1, n, dtype=np.float32)
a_tail = a[-n:] * (1 - fade)
Expand Down Expand Up @@ -959,15 +960,16 @@ from scipy.signal import butter, lfilter

x, sr = sf.read("wet.wav")
assert sr == 16000
if x.ndim > 1: x = x[:,0]
if x.ndim > 1:
x = x[:, 0]

delay_ms = np.random.uniform(0.6, 1.2)
d = int(sr * delay_ms / 1000.0)
a = 10 ** (-np.random.uniform(6, 10) / 20.0) # -6 to -10 dB

# LPF for reflection
b, c = butter(2, 5000/(sr/2), btype="low")
ref = lfilter(b, c, np.pad(x, (d,0))[:-d])
b, c = butter(2, 5000 / (sr / 2), btype="low")
ref = lfilter(b, c, np.pad(x, (d, 0))[:-d])

y = x + a * ref
y = np.clip(y, -1.0, 1.0)
Expand Down
53 changes: 31 additions & 22 deletions docs/research/research_report_gpt55_pro.md
Original file line number Diff line number Diff line change
Expand Up @@ -772,7 +772,7 @@ def mix_at_snr(clean, noise, snr_db, rng=np.random.default_rng()):
noise = np.tile(noise, reps)

start = rng.integers(0, len(noise) - len(clean) + 1)
noise = noise[start:start + len(clean)]
noise = noise[start : start + len(clean)]

noise = noise - np.mean(noise)
target_noise_rms = rms(clean) / (10 ** (snr_db / 20))
Expand Down Expand Up @@ -1068,6 +1068,7 @@ def add_subtle_f0_microvariation(x, sr, amount_semitones=0.08, corr_frames=5, se
y = y / (np.max(np.abs(y)) + 1e-9) * 0.98
return y.astype(np.float32)


x, sr = sf.read("tts_turn.wav")
y = add_subtle_f0_microvariation(x, sr, amount_semitones=0.06)
sf.write("tts_turn_micro.wav", y, sr, subtype="PCM_16")
Expand Down Expand Up @@ -1181,7 +1182,7 @@ def analyze_turn(path, speaker_profile=None):

duration = len(y) / sr
peak = np.max(np.abs(y)) + 1e-12
rms = np.sqrt(np.mean(y ** 2) + 1e-12)
rms = np.sqrt(np.mean(y**2) + 1e-12)

# Click proxy: large first differences relative to RMS.
dy = np.diff(y)
Expand All @@ -1197,8 +1198,8 @@ def analyze_turn(path, speaker_profile=None):
# F0 using pyin. Tune fmin/fmax per speaker if known.
f0, voiced_flag, voiced_prob = librosa.pyin(
y,
fmin=librosa.note_to_hz("C2"), # ~65 Hz
fmax=librosa.note_to_hz("C6"), # ~1047 Hz, broad guard
fmin=librosa.note_to_hz("C2"), # ~65 Hz
fmax=librosa.note_to_hz("C6"), # ~1047 Hz, broad guard
sr=sr,
frame_length=1024,
hop_length=160,
Expand Down Expand Up @@ -1306,7 +1307,7 @@ anchor_emb = mean_embedding(neutral_anchor_turns)
turn_emb = speaker_embedding(candidate_turn)
cosine = np.dot(anchor_emb, turn_emb) / (np.linalg.norm(anchor_emb) * np.linalg.norm(turn_emb))

if cosine < 0.72: # calibrate per embedding model and TTS engine
if cosine < 0.72: # calibrate per embedding model and TTS engine
reject("speaker_identity_shift")
```

Expand Down Expand Up @@ -1726,23 +1727,31 @@ TTS dry turn
#### Example audiomentations chain

```python
from audiomentations import Compose, AddBackgroundNoise, ApplyImpulseResponse, Gain, ClippingDistortion

augment = Compose([
ApplyImpulseResponse(
ir_path="rir_database/",
p=0.7,
leave_length_unchanged=False,
),
AddBackgroundNoise(
sounds_path="noise_database/",
min_snr_db=12,
max_snr_db=30,
p=0.9,
),
Gain(min_gain_db=-3, max_gain_db=3, p=0.5),
ClippingDistortion(min_percentile_threshold=0, max_percentile_threshold=3, p=0.08),
])
from audiomentations import (
Compose,
AddBackgroundNoise,
ApplyImpulseResponse,
Gain,
ClippingDistortion,
)

augment = Compose(
[
ApplyImpulseResponse(
ir_path="rir_database/",
p=0.7,
leave_length_unchanged=False,
),
AddBackgroundNoise(
sounds_path="noise_database/",
min_snr_db=12,
max_snr_db=30,
p=0.9,
),
Gain(min_gain_db=-3, max_gain_db=3, p=0.5),
ClippingDistortion(min_percentile_threshold=0, max_percentile_threshold=3, p=0.08),
]
)

y_aug = augment(samples=y, sample_rate=16000)
```
Expand Down
Loading