Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ soundfile. Model-based backends are optional extras, and the test suite never do

> **Status: alpha (0.1.0.dev0).** Every tool runs and is tested on synthetic signals. Interfaces may still change before 1.0.

Experimental room-tone synthesis is available as `opendubstage roomtone`, with an opt-in
`specrepair --synthesize-tone` fallback. See [method and limitations](docs/roomtone.md).

## Quickstart

```bash
Expand Down
8 changes: 7 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,13 @@ A sentence-level leveller in the spirit of a dialogue-leveller plug-in, as oppos
- Tests: synthetic phrases at known levels converge to the target within ±1 LU, gain never moves inside a word, and
the output is bit-exact where no ride is needed.

### Room-tone synthesis
### Room-tone synthesis (experimental mono implementation)

Implemented as `opendubstage roomtone`, with opt-in `specrepair --synthesize-tone` fallback;
see [method, defaults and limitations](docs/roomtone.md). Synthetic tests cover spectral/level
match, long-lag nonperiodicity and protected samples. Real-world listening validation remains pending.

Original goal:
Generate arbitrary lengths of a recording's own room tone, for fills longer than any clean stretch available
(an "ambience match"):
- Learn the noise floor's spectral envelope and its slow modulation from gaps between words; synthesise by
Expand Down
78 changes: 78 additions & 0 deletions docs/roomtone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Room-tone synthesis (experimental)

`opendubstage roomtone` generates a mono statistical approximation of the recording's
own ambience. It does not loop a captured waveform, download a model, detect words,
or promise that matching spectra sounds indistinguishable. Listening validation is pending.

```sh
opendubstage roomtone input.wav tone.wav --duration 30 --protected words.json \
--excluded events.json --seed 0 --report tone.json
```

Timing files use seconds: `[[start, end], ...]` or `{"words": [{"start": 0.5, "end": 1.2}]}`.
Protected words/kept sounds and excluded events are never learning material. Their complement
must genuinely be clean room tone. Without timings the entire source is explicitly assumed
clean. Mark breaths, music and noises as protected/excluded too; this is not a voice detector.
Invalid/out-of-bounds intervals are rejected, overlaps are merged. Output is floating-point WAV
with the source sample rate. Output and report paths must not overwrite either audio or timing inputs.
A supplied but unrecognized/malformed timing schema fails closed, never silently
removes protection.

## Method and defaults

- Pad protected/excluded intervals by **20 ms** as a conservative word-edge guard.
- Analyze only complete **128 ms** Hann frames within clean gaps, with **75% overlap**.
No analysis frame spans a word, event or separate gap. This resolves useful low bands
without requiring multi-second clean stretches. Gaps shorter than one frame are discarded.
- Require **0.5 seconds total** usable clean material and at least **four frames**. More,
representative clean material is better. Insufficient evidence raises an error, never
substitutes unrelated noise. A genuinely silent reference produces silence.
- Normalize each analysis spectrum by its frame power, average it, and independently retain
smoothed frame-level statistics. Synthesis randomizes every frame's phase, Hann-overlap-adds
frames, and compensates the sum of squared windows. Padding/cropping removes boundary
startup fades. DC and Nyquist bins have random real signs.
- Approximate slow amplitude variation by drawing from smoothed log-RMS statistics at
**0.5 second** knots with cosine interpolation. Deviations are clamped to **±6 dB**.
This retains approximate modulation depth, not the original trajectory or modulation spectrum.
- Match overall RMS to the mean source analysis-frame power. This is an RMS measure,
not LUFS. Output duration is rounded to the nearest sample. Peak above full scale is an
error rather than silent clipping/attenuation. There is no limiter.
- A seed (default **0**) makes output reproducible. Different seeds produce different waveforms;
duration is not constrained by the clean material's length. Processing is in-memory and
long requested output uses proportionate memory/time.

These are engineering defaults, not a perceptual standard. Mono only: stereo width,
interchannel correlation, narrow tonal hum, and strongly nonstationary room events are not
claimed to be reproduced faithfully. Random phase is particularly inappropriate when a
specific tonal phase trajectory must be preserved.

The JSON report records method, seed, sample count/rate, used clean intervals, analysis-frame
count, source/output RMS, sample peak, learned modulation deviation, and limitations. It does
not claim a quality score or hide insufficient-evidence failures.

## Opt-in spectral-repair fallback

```sh
opendubstage specrepair input.wav repaired.wav --events events.json --protected words.json \
--synthesize-tone --tone-seed 0 --report repairs.json
```

Existing contiguous-tone selection remains first choice and all defaults remain unchanged.
Only if no candidate meets `tone_min_len` does the opt-in fallback gather shorter quiet islands
inside `context`. It retains the repairer's protected/excluded/event-gap exclusions and quiet
energy limits, then requires the minimum evidence above. It reports `room_tone_synthesis` and
embeds the synthesis report (clean intervals use absolute source time, with the analyzed
window offset also recorded). If unavailable it reports a skipped repair and reason. The same
patch splicing and integer-PCM writer preserve every sample outside reported patches exactly.
The synthesizer learns the short islands' level directly, so the legacy post-synthesis EQ match
is not applied again. Users must still review the repaired transition by ear.

## Verification

Synthetic tests verify, for stationary colored noise, third-octave spectral power within
**±1.5 dB** over resolved bands **100–6300 Hz at 16 kHz**, RMS within **0.2 dB**, and maximum
absolute normalized autocorrelation below **0.05** at **0.1–5 s** lags. Short-lag correlation is
expected from the spectral colour; the bound is not universal for arbitrary tonal recordings.
Tests also cover modulation, timing exclusion, repeatable/fresh seeds, exact length, silence,
malformed input, opt-in fallback, insufficient/context-limited evidence, CLI, and bit-exact
integer-PCM samples outside patches. Numerical checks do not substitute for blind listening.
1 change: 1 addition & 0 deletions opendubstage/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

# tool -> (module, one-line help). Modules are imported lazily so `--help` stays fast and light.
TOOLS = {
"roomtone": ("opendubstage.restore.roomtone", "synthesize non-looping mono ambience from clean gaps"),
"leveller": ("opendubstage.mix.leveller", "phrase-safe dialogue gain rides from word timings"),
"specrepair": ("opendubstage.restore.specrepair", "spectral repair: room-tone replace, attenuate, LF thumps"),
"memofix": ("opendubstage.restore.memofix", "remove car/room rumble learned from word gaps, LTASS matching EQ"),
Expand Down
211 changes: 211 additions & 0 deletions opendubstage/restore/roomtone.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
"""Room tone from explicitly clean gaps, using random-phase overlap-add synthesis.

Input timing JSON accepts [[start, end], ...] or {"words": [{"start": ..., "end": ...}]}.
Words/exclusions are protected, NOT automatically detected. Complete Hann analysis
frames are collected inside their complement; no frame crosses a protected boundary.
A normalized average power spectrum learns colour independently of level. Slow level
variation is approximated by randomly resampled, smoothed frame RMS statistics, not
by replaying a recording. Independent random phases avoid a copied loop period.

Engineering defaults: 128 ms FFT for low-frequency resolution; 75% overlap to smooth
random frames; 20 ms word guard; 0.5 s minimum total clean material (at least four
frames); 0.5 s envelope knots, +/-6 dB envelope clamp for conservative modulation.
These are not a perceptual guarantee. Mono only; no stereo-coherence claims.
"""
from __future__ import annotations

import argparse
import json
from dataclasses import dataclass
from pathlib import Path

import numpy as np
from scipy import signal
from scipy.ndimage import uniform_filter1d


@dataclass(frozen=True)
class Params:
frame_s: float = 0.128
guard_s: float = 0.020
min_clean_s: float = 0.5
modulation_s: float = 0.5
modulation_limit_db: float = 6.0


def _audio(x, sr):
x = np.asarray(x, dtype=np.float64)
if x.ndim != 1 or not len(x) or not np.isfinite(x).all():
raise ValueError("input must be nonempty finite mono audio")
if not isinstance(sr, (int, np.integer)) or isinstance(sr, bool) or sr < 1000:
raise ValueError("sample rate must be an integer >= 1000")
return x


def _params(p):
vals = (p.frame_s, p.guard_s, p.min_clean_s, p.modulation_s, p.modulation_limit_db)
if not all(np.isfinite(v) for v in vals) or p.frame_s <= 0 or p.guard_s < 0 or p.min_clean_s <= 0 or p.modulation_s <= 0 or p.modulation_limit_db < 0:
raise ValueError("invalid room-tone parameters")


def clean_intervals(duration, protected=(), excluded=(), guard_s=0.02):
"""Complement of validated, padded protected/excluded intervals in seconds."""
busy = []
for a, b in list(protected) + list(excluded):
if not np.isfinite([a, b]).all() or a < 0 or b <= a or b > duration + 1e-9:
raise ValueError("intervals must satisfy 0 <= start < end <= duration")
busy.append((max(0., a - guard_s), min(duration, b + guard_s)))
cursor, out = 0., []
for a, b in sorted(busy):
if a > cursor:
out.append((cursor, a))
cursor = max(cursor, b)
if cursor < duration:
out.append((cursor, duration))
return out


def learn(x, sr, protected=(), excluded=(), p=Params()):
"""Learn model from gap samples only. Raises rather than inventing missing tone."""
x = _audio(x, sr); _params(p)
nfft = max(32, int(round(p.frame_s * sr / 2)) * 2)
hop = nfft // 4
win = signal.windows.hann(nfft, sym=False)
spectra, levels, slow = [], [], []
used, intervals = 0, []
for a, b in clean_intervals(len(x) / sr, protected, excluded, p.guard_s):
i, j = int(np.ceil(a * sr)), int(np.floor(b * sr))
if j - i < nfft:
continue
seg = x[i:j]
frames = np.lib.stride_tricks.sliding_window_view(seg, nfft)[::hop]
powers = np.mean(frames ** 2, axis=1)
# Silent frames cannot teach spectral colour; silence-only input is handled separately.
keep = powers > 1e-30
if keep.any():
z = np.fft.rfft(frames[keep] * win, axis=1)
spectra.append(np.abs(z) ** 2 / powers[keep, None] / np.sum(win ** 2) * nfft)
lv = 10 * np.log10(powers[keep])
slow.extend(uniform_filter1d(lv, size=max(1, int(p.modulation_s * sr / hop)), mode="nearest"))
levels.extend(powers)
used += len(seg); intervals.append([i / sr, j / sr])
if used < p.min_clean_s * sr or len(levels) < 4:
raise ValueError("insufficient clean gaps: need minimum clean duration and four complete analysis frames")
power = float(np.mean(levels))
spec = np.mean(np.concatenate(spectra), axis=0) if spectra else np.zeros(nfft // 2 + 1)
log_level = np.asarray(slow) if slow else np.array([0.])
variation = np.clip(log_level - np.mean(log_level), -p.modulation_limit_db, p.modulation_limit_db)
return dict(sr=int(sr), nfft=nfft, hop=hop, spectrum=spec, rms=np.sqrt(power), modulation_db=variation,
modulation_s=p.modulation_s, clean_seconds=used / sr, frames=len(levels), intervals=intervals)


def synthesize(model, duration, seed=0):
"""Return (mono audio, JSON-safe report). Seed makes synthesis repeatable."""
if not np.isfinite(duration) or duration <= 0:
raise ValueError("duration must be positive and finite")
sr, nfft, hop = model["sr"], model["nfft"], model["hop"]
count = int(round(duration * sr))
if count < 1:
raise ValueError("duration is shorter than one sample")
if not isinstance(seed, (int, np.integer)) or isinstance(seed, bool) or seed < 0:
raise ValueError("seed must be a nonnegative integer")
seed = int(seed)
rng = np.random.default_rng(seed)
# Pad both ends, then crop into the stationary overlap region.
total = count + 2 * nfft
y, weight = np.zeros(total + nfft), np.zeros(total + nfft)
win = signal.windows.hann(nfft, sym=False)
mag = np.sqrt(model["spectrum"])
for start in range(0, total, hop):
phase = rng.uniform(-np.pi, np.pi, len(mag))
z = mag * np.exp(1j * phase)
z[0] = mag[0] * rng.choice([-1., 1.])
z[-1] = mag[-1] * rng.choice([-1., 1.])
y[start:start+nfft] += np.fft.irfft(z, n=nfft) * win
weight[start:start+nfft] += win ** 2
y = y[nfft:nfft+count] / np.sqrt(weight[nfft:nfft+count])
step = max(1, int(round(model["modulation_s"] * sr)))
knots = np.arange(0, count + step, step)
env = rng.choice(model["modulation_db"], len(knots))
# Cosine interpolation gives zero slope at every knot, with no repeated audio blocks.
pos = np.arange(count) / step
index = pos.astype(int); blend = (1 - np.cos(np.pi * (pos-index))) / 2
y *= 10 ** ((env[index] * (1-blend) + env[index+1] * blend) / 20)
measured = np.sqrt(np.mean(y ** 2))
if measured > 0:
y *= model["rms"] / measured
peak = float(np.max(np.abs(y)))
# Never silently clip or lower learned level.
if peak > 1:
raise ValueError("synthesized tone exceeds full scale; supply quieter clean material")
report = dict(method="random_phase_overlap_add", seed=seed, sample_rate=sr, samples=count,
duration=count/sr, clean_seconds=model["clean_seconds"], analysis_frames=model["frames"],
clean_intervals=model["intervals"], frame_samples=nfft,
source_rms=float(model["rms"]), output_rms=float(np.sqrt(np.mean(y*y))), peak=peak,
modulation_std_db=float(np.std(model["modulation_db"])),
limitations=["mono", "explicit clean-gap selection", "statistical approximation; listening validation required"])
return y, report


def generate(x, sr, duration, protected=(), excluded=(), seed=0, p=Params()):
return synthesize(learn(x, sr, protected, excluded, p), duration, seed)



def read_intervals(path):
"""Strict timing schema: a supplied but unrecognized file must never mean no protection."""
if not path:
return []
data = json.loads(Path(path).read_text())
if isinstance(data, dict):
keys = [key for key in ("words", "events", "intervals") if key in data]
if len(keys) != 1:
raise ValueError("timing object must contain exactly one of words/events/intervals")
data = data[keys[0]]
if not isinstance(data, list):
raise ValueError("timings must be a list")
result = []
for item in data:
if isinstance(item, list) and len(item) == 2:
a, b = item
elif isinstance(item, dict) and "start" in item and "end" in item:
a, b = item["start"], item["end"]
elif isinstance(item, dict) and "t" in item and "e" in item:
a, b = item["t"], item["e"]
else:
raise ValueError("each timing needs [start,end] or start/end (t/e) fields")
if any(isinstance(value, bool) or not isinstance(value, (int, float)) for value in (a,b)):
raise ValueError("timing endpoints must be numbers")
if not np.isfinite([a,b]).all() or a < 0 or b <= a:
raise ValueError("timings must have finite 0 <= start < end")
result.append((float(a),float(b)))
return result


def main(argv=None):
import soundfile as sf
ap = argparse.ArgumentParser(prog="opendubstage roomtone", description=__doc__)
ap.add_argument("src"); ap.add_argument("dst")
ap.add_argument("--duration", type=float, required=True)
ap.add_argument("--protected", help="word/kept-sound timing JSON; all other samples must be clean tone")
ap.add_argument("--excluded", help="other unwanted event timing JSON")
ap.add_argument("--seed", type=int, default=0)
ap.add_argument("--report")
a = ap.parse_args(argv)
inputs = {Path(v).resolve() for v in (a.src, a.protected, a.excluded) if v}
outputs = [Path(v).resolve() for v in (a.dst, a.report) if v]
if inputs.intersection(outputs) or len(set(outputs)) != len(outputs):
ap.error("output/report must not overwrite audio or timing inputs, or each other")
try:
x, sr = sf.read(a.src, dtype="float64")
y, report = generate(x, sr, a.duration, read_intervals(a.protected), read_intervals(a.excluded), a.seed)
except ValueError as exc:
ap.error(str(exc))
sf.write(a.dst, y, sr, subtype="FLOAT")
if a.report:
Path(a.report).write_text(json.dumps(report, indent=2, allow_nan=False) + "\n")
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading