Skip to content

Repository files navigation

YTP Machine

Type a sentence, get a video of it being said, cut word by word from recorded speech. The shipped corpus is Michael Rosen reading his own poetry. Nothing in the code is specific to him — point the ingest scripts at any speaker.

Words the corpus has are spliced in directly. Words it doesn't are built from phonemes taken out of other words, so rice can come from the r of one word and the ice of another.

See the repository audit for the recent ingestion, security, performance and editor fixes, their verification, and remaining work.

TLDR: get it running

From a clone:

docker compose up

Open http://localhost:8765 and type a sentence. That is the whole thing — on first start the container downloads the published Michael Rosen corpus into its volume, so a fresh clone already works.

That pulls the published image. Add --build only when you have changed the code, which saves compiling 3 GB that already exists on GHCR.

The image is on GHCR if you would rather not use compose:

docker pull ghcr.io/kapsikkum/ytp-machine:latest

It carries no corpus — one has to be mounted, because a voice is 100 MB to 2 GB of video and does not belong inside an image. So point it at one:

docker run -p 127.0.0.1:8765:8765 -v ytp-data:/app/data -v ./corpora/some-voice:/corpus:ro ghcr.io/kapsikkum/ytp-machine:latest

Any of three sources works: a corpus directory or a .tar.zst bundle mounted at /corpus, bundles dropped in /packs, or CORPUS_URL pointing at one over HTTP. With none of them it says exactly that and stops, rather than serving an app that answers everything with "no clips found".

With no corpus of your own, CORPUS_URL is the way in. The shipped voice is a bundle on the corpus release, which is the only place a 100 MB+ bundle can sit at a stable public URL:

docker run -p 127.0.0.1:8765:8765 -v ytp-data:/app/data -e CORPUS_URL=https://github.com/kapsikkum/ytp-machine/releases/download/corpus/michael-rosen.tar.zst ghcr.io/kapsikkum/ytp-machine:latest

The bundle is packed and uploaded by hand (see The shipped corpus).

Pin the tag for anything you care about — :latest moves under you on the next push, and every commit is also tagged by its full sha:

docker pull ghcr.io/kapsikkum/ytp-machine:e4f6b8250ad18f0157acae513c779e83e7d7ca6e

No GPU needed to run it. You do want one to build a new corpus.

Compose publishes port 8765 on localhost by default. The operator API can edit and replace corpora and has no built-in login. For remote use, put an authenticated reverse proxy in front of it and set MRS_BIND_ADDRESS to the interface the proxy needs. Browser origin checks and Matrix allow-lists do not provide authentication for the HTTP API.

For an HTTPS reverse proxy, preserve Host and set X-Forwarded-Proto to the external scheme. Set the app container's FORWARDED_ALLOW_IPS to the proxy's IP as seen inside the container (for example, 127.0.0.1,10.88.0.1 for a host nginx reaching Podman's default bridge). Compose accepts this in .env; NixOS OCI containers can set it in environment.FORWARDED_ALLOW_IPS. Uvicorn otherwise trusts only loopback and sees HTTP, causing HTTPS browser writes to fail with not from another origin. Trust specific proxy addresses, not *, and have the proxy overwrite forwarded headers. Requests from other origins must still receive 403.

Mounted corpus directories are seeded with a consistent SQLite snapshot, including committed WAL writes and pronunciation overrides. Imports validate in staging before activation. Forced replacements retain the previous corpus under corpora/.backups/; loose unpack --force keeps replaced members under .corpus-backups/. Allow disk space for staging and these retained backups. Stop the app before running the legacy migration command.


TLDR: train it on a channel

1. One environment with a GPU, somewhere your file sync will not touch (OneDrive locks DLLs mid-install and torch dies with a WinError 32 naming an innocent file):

python -m venv C:/Users/you/.venvs/ytp
C:/Users/you/.venvs/ytp/Scripts/python.exe -m pip install --index-url https://download.pytorch.org/whl/cu128 torch torchaudio
C:/Users/you/.venvs/ytp/Scripts/python.exe -m pip install -r requirements.txt

2. Build it. One command does download, transcribe, align and pack:

python scripts/build_corpus.py @SomeChannel --limit 10 --data-dir C:/Users/you/ytp-corpora/some-channel

Ten long videos is a good first target — roughly an hour of GPU time and about 5,000 distinct words. --limit 0 takes the whole channel. Re-running skips what is already ingested, so a run you interrupt just carries on.

3. Install it wherever the server runs:

python scripts/corpus.py install some-channel.tar.zst --name some-channel

Then pick the voice from the dropdown, or POST /api/corpus.

4. Optional, but it is what makes it sound good. The build leaves a pronunciations.csv in the data directory listing every word it cannot pronounce, commented out. Uncomment a line and say what the word rhymes with:

wii,wee
nug,nugget
usb,=letters

Then POST /api/reload. To rebuild that list later, point it at the corpus:

MRS_DATA_DIR=C:/Users/you/ytp-corpora/some-channel MRS_CORPUS=some-channel python scripts/verify_corpus.py --unspliceable --write-template --per-corpus

Run it

docker compose up --build

http://localhost:8765. On a fresh volume it needs a corpus — see below.

No GPU needed. Torch comes from the CPU wheel index; the default index pulls gigabytes of CUDA payload a headless server never touches.

Without Docker:

pip install -r requirements.txt
python -m uvicorn main:app --port 8765

ffmpeg must be on PATH either way.

How it works

  • Ingest downloads a video, transcribes with Whisper, refines timings with stable-ts.
  • Alignment (app/forced_align.py) runs torchaudio's CTC aligner (Wav2Vec2 BASE 960h) for character-level timings. That's what makes sub-word cutting work.
  • Generation (app/generate.py) resolves a sentence into clips, preferring the longest real spoken runs it can find, and falls back to phoneme splicing per word. ffmpeg cuts and concatenates.

Votes on a clip are stored in splice_ratings and weight it up or down for that word next time.


The corpus

One voice: a directory holding one database and the videos it indexes.

corpora/michael-rosen/corpus.db      timings: this word, this file, 4.12s-4.61s
corpora/michael-rosen/downloads/     the videos those timings point into
corpora/michael-rosen/transcripts/   true text, where there is any

The database stores only timings, so the two halves are useless apart and a corpus always moves as a whole. Paths inside are relative to the corpus directory, so copying it anywhere works. Michael Rosen's is ~110 MB: 48 videos, ~9,500 clips, ~1,400 distinct words.

Corpora are not in git — binary that never delta-compresses, and derived data besides. They travel as a bundle: one tarball of database + videos + transcripts.

The shipped corpus

Not in the repo. The Michael Rosen voice is published as a bundle on the corpus release, and that bundle is the only copy that ships: compose and the image seed from it, and CI installs it to test against. A clone has no corpora/ at all; corpora/ is ignored, so nothing built locally ends up in history.

To get it locally:

gh release download corpus --pattern michael-rosen.tar.zst
python scripts/corpus.py install michael-rosen.tar.zst --name michael-rosen

The release is made by hand, from a corpus that has been checked: refine its boundaries, listen, then pack and upload over the old asset.

python scripts/corpus.py pack --corpus michael-rosen --out michael-rosen.tar.zst
python scripts/corpus.py info michael-rosen.tar.zst
gh release upload corpus michael-rosen.tar.zst --clobber

To move one somewhere else:

python scripts/corpus.py pack                       # -> corpus-YYYY-MM-DD.tar.zst
python scripts/corpus.py unpack corpus-*.tar.zst
CORPUS_URL=https://example.invalid/corpus.tar.gz docker compose up

The entrypoint takes any of three: loose files at /corpus, a bundle at /corpus, or CORPUS_URL. With none and no database it says what's missing and stops.

output/ stays out of git — about a gigabyte, all reproducible in seconds.

Building a new corpus

Any speaker with a decent amount of clear, single-voice footage works. More distinct words means fewer splices and better output.

One command does the lot — download, transcribe, align, pack:

python scripts/build_corpus.py @GarbageTime420 --limit 40
python scripts/build_corpus.py https://youtube.com/@Name --name my-corpus --model medium

It names the corpus from the channel handle unless you pass --name, and defaults to a target of 40 total videos in the corpus (--limit 0 for the whole channel). Videos already ingested count towards that target, so interrupting at 25/40 and rerunning adds up to 15 more. The standalone ingest_channel.py counts new videos instead. Use --skip-ingest / --skip-refine / --skip-pack to redo part of it, and --data-dir to build somewhere other than the project root.

The build transcribes first, refines word boundaries second, and places phonemes last. Each model runs in its own process, which keeps Whisper and the phoneme model from sharing GPU memory. --skip-align also skips alignment during ingest. Phoneme results are checkpointed every 50 clips and on a handled interruption, so resuming skips completed clips without a database transaction for each one. If a clip is relabeled or retimed while alignment runs, the checkpoint leaves that edit intact. Standalone channel ingestion also resumes missing phoneme alignment for previously ingested videos. Empty legacy sources remain eligible for ingestion.

It checks the dependencies, ffmpeg and the GPU before starting, because the alternative is finding out four hours in.

Use a GPU. All the cost is transcription and alignment. Every script takes --device auto|cuda|cpu (or $MRS_DEVICE), default auto. On an RTX 3070 Ti transcription runs ~10x realtime against ~1x on CPU.

requirements.txt installs CPU torch on purpose, so a GPU ingest wants its own environment:

python -m venv .venv-cuda
.venv-cuda/bin/pip install --index-url https://download.pytorch.org/whl/cu128 torch torchaudio
.venv-cuda/bin/pip install -r requirements.txt

On Windows keep that venv, and the corpus, out of OneDrive. The sync engine locks DLLs mid-install and torch then fails with WinError 32 naming an innocent-looking file.

The stages by hand

build_corpus.py runs 1, 2 and 5. Steps 3 and 4 need human input and are optional.

1. Ingest. One video, a local file, or a channel:

python scripts/ingest.py https://www.youtube.com/watch?v=XXXXXXXXXXX
python scripts/ingest.py /path/to/video.mp4 --model medium
python scripts/ingest_channel.py https://www.youtube.com/@SomeChannel/videos --limit 40 --skip-errors

--model trades speed for accuracy. Every later step inherits these labels, so small or medium is worth it. --normalise re-encodes to the corpus format (480x270), which makes bundles far smaller and generation cheaper. Original imports also render correctly: generation checks their actual size, frame rate and display rotation, then scales with letterboxing when needed. Local imports refuse to overwrite an existing recording with the same filename.

For English speech, --language en avoids detecting the language from the opening audio. The English-only models (small.en, medium.en) and turbo are also accepted by build, ingest and realign. Compare models on a short, manually transcribed sample from the speaker before committing to a whole channel: the fastest model and the most accurate one depend on the material. The word and phoneme aligners currently use English models and pronunciations.

Supply a short vocabulary hint when names or recurring terms are misheard:

python scripts/build_corpus.py @SomeChannel --model medium.en --language en --initial-prompt "Names and terms: Morshu, Hyrule, rupees." --limit 10

These options also work with scripts/realign.py, which reuses downloaded videos. Forced alignment adjusts timings; misheard words still need a better transcription or a corrected transcript. Empty transcriptions and invalid word durations are rejected, and refinement preserves hand-edited clip boundaries.

2. Check the words. Whisper now and then garbles a stretch without saying so — "only thing that happened, I got a rotten headache" came out as "hes all down hes all" — and every later stage believes it. A second transcription with Qwen3-ASR catches that:

pip install qwen-asr                                    # pins its own transformers: a venv of its own is easiest
python scripts/second_opinion.py --all                  # dry run: every change it would make
python scripts/second_opinion.py --all --apply          # before refine_boundaries

Where the two agree nothing changes. A word swapped one for one is relabelled where it stands; a stretch heard as a different number of words is replaced by Qwen's words, which the refinement then places; words only Qwen heard are added; words only Whisper heard are kept but marked "disputed" (rarely picked, and left out of the refinement's alignment). Spelling variants ("play time" / "playtime", -ise / -ize) and anything next to a hand-edited clip are left alone. build_corpus.py runs it when qwen-asr is installed.

3. Sharpen the timings. Whisper's word boundaries run early and wander by 100–350 ms, so short clips grab the tail of the previous word:

python scripts/realign.py --model small                 # redo timestamps, e.g. after a model change
python scripts/refine_boundaries.py --source-id 3       # CTC alignment, dry run on one
python scripts/refine_boundaries.py --all --apply       # frame-accurate, everything

refine_boundaries matters most for splice quality. The aligner decides which stretch of sound is which word; the cut itself is then placed from the audio — at the edges of a pause, or the quietest point between two words that run together — because CTC fires late on a word's first letter and early on its last. Clips whose letters the aligner cannot find where Whisper put them -- mostly noises transcribed as words ("chk chk chk", a breath as "a") -- are marked with an "unheard" boundary report rather than removed, since some are real words said too quickly to score: a marked take is rarely picked, still there when a word has no other, and listed for review in the editor. Hand- edited clips are never moved or marked. Nor does a cut run on into speech no word accounts for — an unwritten "erm" straight after "Brian" — where the aligner's model hears letters there: it stops at the dip before it. A dry run prints what it would mark. It needs --all or a --source-id to pick targets and --apply to write.

python scripts/measure_tone.py                         # after refining: tone of every take

measure_tone describes how each take sounds — pitch, where its pitch starts and ends, how much it moves, loudness against its own recording, pace and brightness — and groups the corpus into a few tone categories. The generator then keeps a sentence in one tone where the words allow it, joins takes that start near where the last one ended, and stays in the same recording: the same tone setting, 0 to turn it off. It runs at build time, takes a minute or two, needs no model, and must be re-run after boundaries move (refining or editing clears the rows it measured). Corpus schema v2 carries it; v1 corpora upgrade on open and pick exactly as before until measured.

Re-transcription refuses sources with hand-edited clips unless you pass --replace-edited. It also aborts if clips or the recording change during the run. Replacement removes stale dependent alignments and votes. Pinned splice recipes are remapped when a new clip unambiguously contains the same cut; otherwise their clip pin is released while retaining the sound preference.

4. Fix misheard labels (optional, needs a real transcript). Relabels only high-confidence phonetically-close swaps, never timestamps. Put the true text in the active corpus's transcripts/<video_id>.txt (--transcripts-dir overrides the directory). Relabeling clears the affected character and phoneme alignment so it can be measured against the corrected word:

python scripts/correct.py --all            # dry run
python scripts/correct.py --all --apply

5. Catch non-verbal noises — clicks, pops, spews — which Whisper skips:

python scripts/find_noises.py              # dry run
python scripts/find_noises.py --apply

Not generic: it carries a hardcoded list of two Michael Rosen videos, because scanning everything turned up mostly breaths. For another speaker edit CURATED at the top of main(), or skip the step.

6. Pack.

python scripts/corpus.py pack --corpus <name>
python scripts/corpus.py info corpus-*.tar.zst

pack takes a consistent SQLite backup, including committed WAL writes, without checkpointing the live database. Bundles carry the effective global and voice pronunciation rules together, with voice overrides last. They are .tar.zst, or .tar.gz if zstandard isn't installed. unpack reads either.

Markup in the input

Two things are not plain words:

typed does
~word~ plays that word backwards
*noise* a non-verbal clip (*spew*, *click*) rather than the spoken word

Reversal happens at cut time, not in the corpus. Reversed speech is not speech: Whisper transcribes it into confident nonsense, so a corpus built from a backwards recording is a set of noises labelled with words nobody said, and it poisons every later splice. Reversing on the way out costs nothing when unused and works for any word in any corpus.

A run is one clip, so it is reversed or not as a whole — marking one word ends the run there rather than quietly reversing its neighbours.

YTPMV: sing a MIDI file

http://localhost:8765/ytpmv.html

Drop in a MIDI file and the corpus plays it: one tile per instrument, the clip re-triggered on every note, and every note pitch perfect. The clip's wobbling speech pitch is measured frame by frame and laid back down exactly on the note's frequency (TD-PSOLA, app/ytpmv/pitch.py), which keeps the voice's character while putting it in tune — solo notes measure within a few cents.

You don't need to know any music theory. The file is read for you:

  • key, tempo, time signature and length, described in plain words
  • what each part is: tune, backing, chords, bass, or a piece of the drum kit (channel 10 is split into kick, snare, hats, toms and cymbals)
  • a word for each part, picked for the kind of sound it makes: a plosive for a kick, a hiss for a hi-hat, a long steady vowel for a tune. Every take of the candidate words is measured, so the take picked is the one that holds a pitch best.
  • an octave for each part, so a bass line at E1 is sung where the voice can sing it. The notes keep their names, so it is still the same song.

Change any of it: type a word or phrase, cycle through takes, click an alternative, press ▶ part to hear a part on its own, move an octave, mute a part, choose tape for the classic chipmunk sound instead of pitch perfect. Render goes through the same queue as sentences.

Voice parts also have a corpus dropdown containing the installed voices. Choose a different performer for each part, or use the Song default group. The closed menu shows the actual performer in either case. Default voice and Global chip stay above the arrangement; Auto tracks follow that chip, while explicitly chosen track instruments stay fixed. Switching a part's corpus refreshes its suggested sounds, takes and vocabulary; previews and the finished video use that same corpus. The voice at the top is this page's default and leaves the server's active corpus unchanged. Explicit part choices survive changes to that default and are restored on page reload.

The DAW transport plays from the playhead to the end when no range is selected. Shift + drag selects a playback range; Clear range returns to whole-song playback. Playback uses the same mixer as the video render, including speed, key, pan, levels, chip, variety and balance, within YTPMV_MAX_SECONDS (15 minutes by default). Space plays/pauses throughout the editor except while editing text or a control. The transport clock and timeline cursor use the audio clock; return-to-start and ±10-second buttons seek it directly. There is one playback UI; the browser's native audio controls are hidden. Solo affects playback only. Stop also cancels pending audio preparation.

Click the sound name in a track header for its instrument window: voice, sample/take, up to twelve suggestions, chip instrument and GM patch controls, a note-audition keyboard and a waveform decoded from the actual audition audio. Opening the instrument prepares all thirteen visible keys in one audio bank. The browser decodes it once and plays cached notes locally through Web Audio, including overlapping key presses, without per-key generation requests. Changing the voice, sample/take, instrument/patch or sound settings rebuilds the bank; returning to a recently used sound reuses its cache. Decoded banks are bounded to six entries and 48 MiB in the tab. Initial preparation shows a loading state. Keys audition one note; Play track pattern auditions only that track from its first note. The dock's part button auditions in place without opening the instrument window. Instrument auditions are separate from song playback.

MIDI library replaces the editor while browsing. Search, sort, rename inline or delete an uploaded MIDI with confirmation. Rename updates an open song; deleting that song clears its local arrangement and returns to the library. Existing rendered files remain. Files used by queued/running renders cannot be deleted until those jobs stop. MIDI files are shared server-side; track settings remain local to the device.

The header switches between Simple (the original stacked parts and render layout) and Advanced (the DAW workbench). Both share the current engine, fixes, corpus selection and saved arrangements. The interface preference is remembered on this device; ?mode=simple or ?mode=advanced selects it explicitly.

The track workbench shows real MIDI activity above a dock for the selected part. Pick a track to edit its voice, sample or chip instrument; its corpus selector, quick sound button and M/S controls work directly from the header. Track, Song & render, and Video preview keep local settings, global options, and the composition in separate dock panels. Find & manage tracks holds the search and role filters. OPL and SNES expose their GM patch banks beside the instrument selector; automatic chip sounds follow a changed MIDI patch. Chip instruments play independently of corpus clips and draw waveform tiles.

The timeline scrolls vertically through tracks and horizontally when zoomed. Its ruler and track controls stay pinned, and large arrangements draw only nearby track rows. Use Fit song, the zoom slider, Focus range, or Part start to navigate. Shift + wheel scrolls through time; Ctrl + wheel zooms around the pointer. Click or drag to seek, or Shift + drag to choose a playback range. The playhead follows the actual audio; Follow playback keeps it in view. Focus the timeline for arrows to seek, Shift + arrows to jump ten seconds, and Home/End to move to the ends. Zoomed views request bounded note detail for their visible time range without re-running voice recommendations. The MIDI note view is read-only.

Each MIDI keeps its own arrangement, voice and render options in this browser's local storage. Undo reverses edits in the current session. Settings reveals pitch, pan, note length and tile controls. Use takes & clip to watch the sample and choose a numbered take inline. Bulk voice controls live with the song settings. If browser storage fills or is blocked, the save indicator reports the failure.

Video exports use a 1920×1080 (16:9) canvas. Video preview offers Classic grid, Balanced grid, Two rows, Two columns, Vertical strips and Featured track. Classic grid restores equal 16:9 tiles using the original grid arrangement. Center tiles centers the grid and incomplete rows by default: three streams have two on top and one centered below. Turn it off for top-left alignment. Unused space stays black. The other layouts share the full available space.

Framing → Fit whole clip preserves the entire picture, centered inside each tile with black bars where needed. Fill tile (crop) covers the tile with a center crop. Neither stretches the source. The preview and actual video use the same framing. Drag tiles to reorder them with animated movement, or use Alt + left/right on a focused tile. Layout, alignment, framing and order persist per MIDI and work in both Simple and Advanced interfaces.

Rendering keeps the last successful export visible. Cancel render stops a queued or running job; a refresh reconnects to an acknowledged job, and reconnect recovers from interrupted polling. Each export records its song and settings, and the page marks it when the arrangement has changed. Recent exports are remembered on this device; server cleanup can eventually remove their media, and a server restart can discard unfinished in-memory jobs.

From a terminal:

python scripts/ytpmv.py song.mid --info
python scripts/ytpmv.py song.mid --part lead=yeah --part kick=boom --max 30
python scripts/ytpmv.py song.mid --only bass --check     # measure the tuning

YTPMV_MAX_SECONDS caps the length of a render (default 900, fifteen minutes).

On a console

chip= to the bot, --chip from the terminal, or the dropdown plays the song on an emulated console -- sound and picture both:

python scripts/ytpmv.py song.mid --chip md          # four-operator FM
python scripts/ytpmv.py song.mid --chip snes-voice  # him, through a SNES
setting sound picture
md the YM2612's FM operators; the voice is gone 320x224, 9-bit colour
md-voice him, out through the Mega Drive's 8-bit DAC 320x224, 9-bit colour
nes the 2A03's pulses, triangle and noise; the voice is gone 256x240, the 64-colour palette
nes-voice him, out through the NES's delta-modulation channel 256x240, the 64-colour palette
sms the SN76489's three squares and noise; the voice is gone 256x192, 6-bit colour
sms-voice him, hammered out of the Master System's volume register 256x192, 6-bit colour
snes a General MIDI bank cut down to SNES-sized BRR samples, through the S-DSP 256x224, 15-bit colour
snes-voice him, through the S-DSP's BRR, interpolation and echo 256x224, 15-bit colour
gb the Game Boy's pulses, wave table and noise; the voice is gone 160x144, four greens
gb-voice him, pushed through the wave table four bits a step 160x144, four greens
gbc the same chip as gb 160x144, the Color's washed-out LCD
gbc-voice the same as gb-voice 160x144, the Color's washed-out LCD
opl2 an AdLib's OPL2 playing a Doom-format GENMIDI bank 320x200, EGA's sixteen colours
opl2-voice him, off an original Sound Blaster's 8-bit DAC 320x200, EGA's sixteen colours
opl3 a Sound Blaster's OPL3, same bank, all eight waveforms 320x200, VGA
opl3-voice him, off a 16-bit Sound Blaster 320x200, VGA
sid the C64's 6581 SID: pulse, saw, triangle, noise, and its filter 320x200, the VIC-II's sixteen
sid-voice him, hammered out of the 6581's volume register 320x200, the VIC-II's sixteen
sid8580 the later 8580 SID 320x200, the VIC-II's sixteen
sid8580-voice the same trick on an 8580, where it barely works 320x200, the VIC-II's sixteen

The picture is not a separate choice. A Mega Drive soundtrack over a picture the Mega Drive could never have drawn is two machines, and nobody asking for one meant that.

The synthesised settings take every part, drums included, and keep only how long each note lasts and how loud it was. The -voice settings keep the whole trick -- the clip still pitch-tracked onto the exact note -- and play it out through the channel that console used for speech. The SNES has no other mode: it synthesises nothing, so it is always him.

Any part can be given its own instrument: the page's instrument picker on each part, or the bot's tone:<part>=. Left on auto, a part plays what the chip picks for it; anything chosen stays whatever the chip is, so one song can mix machines -- an NES triangle bass under a SID lead. voice keeps a part as him, untouched, while everything else goes through the chip (clean cannot mean that, because the page used to send it back for every part). Switching chip on the page puts a part back to auto only if its choice belonged to the machine being left behind.

A part's General MIDI instrument can be swapped too -- the page's patch, shown when an OPL plays the part, or the bot's program:<part>= (0 to 127, so program:lead=29 is overdriven guitar). On an OPL that is the patch. The other chips mostly choose by the part's job, and only fall back to the instrument for a part with no job they know.

app/ytpmv/ym2612.py -- the Mega Drive. A model rather than a cycle-exact emulator like Nuked-OPN2: the log-domain sine and 14-bit operators, the real envelope generator, all eight algorithms, feedback, detune, 53267 samples a second, and the output stage, which matters because FM lands a sideband on DC and without the console's coupling capacitor a song rumbles. Its bass is Sonic 2's own voice $00, read out of the disassembly. SMPS numbers operators backwards from the hardware, so its op1 is operator 4; get that wrong and operators 2 and 3 swap, which in a chain is a bright FM bass or a near-pure sine.

app/ytpmv/nes.py -- the NES, checked against Blargg's Nes_Snd_Emu. It generates at four times the output rate and filters on the way down, because the noise register clocks at up to 447 kHz. The triangle runs out of timer resolution an octave before the pulses and goes flat at the top of its range; that is the hardware.

app/ytpmv/sms.py -- the Master System, checked against SMSPower's notes: register $0FE is 440.4 Hz. It cannot go below 109 Hz, so anything written lower is raised by octaves. Its documented noise taps are not a maximal-length polynomial and repeat after 57337 steps, not 65535.

app/ytpmv/snes.py -- the SNES, checked against fullsnes: the four BRR predictors and the 512-entry interpolation table, whose taps sum to 2048 at every phase. BRR works in fifteen-bit integers; fed floats between -1 and 1 it quantises everything to nothing. Most of the famous muffle is not the chip but the 64 KB the whole soundtrack had to fit in, which kept samples far below the DSP's rate -- stored_hz models that practice, and says so.

Its instruments are DitherEmotion's ExpressiveSNES, a General MIDI SoundFont of samples ripped from SNES games, cut down by scripts/build_snes_bank.py to what a game carried: one sample per General MIDI program and one per drum key, loops on whole 16-sample BRR blocks, kept at the DSP's 32 kHz since they came out of the chip's own format in the first place (and so not compressed twice). 3.8 MB, app/ytpmv/data/snes_bank.npz, credited in SNES-BANK-SOURCE.txt as the font's author asks of edited versions. Each keeps the font's volume envelope, tuning and level, and the bank records which font it came from. Playback goes through the chip's Gaussian table and the echo, and a note may go past the two octaves the pitch register allows, since a game chose its samples to fit and a General MIDI part cannot.

Any General MIDI font can replace it -- python scripts/build_snes_bank.py some.sf2, with --store-hz 32000 --ripped for one ripped from games. Fonts thrown together from rips are often labelled an octave out; --fix-octaves corrects a sample whose loop plays a whole octave from its label, but check it by ear, since an organ's quiet sub-octave can fool it.

app/ytpmv/gb.py -- the Game Boy, which the Color shares unchanged. Formulas from the gbdev Pan Docs; the output capacitor from SameBoy. Each channel's DAC idles at +1, so a held square sits off centre and the capacitor drags it back the whole time -- squares sag, and that is much of the sound. The gbdev wiki gives the Color a separate capacitor figure that works out to a 700 Hz high-pass, which would take the bass off entirely; SameBoy uses one figure for every model, a 28 Hz corner, and so does this. The two machines differ in their screens: ares's four greens for the Game Boy, and ares's model of the Color's LCD, which bled each colour into the others and never reached full brightness.

app/ytpmv/opl.py -- the OPL2 and OPL3. Tables from Nuked-OPL3, whose log-sine and exponent ROMs turn out to be exact formulas; envelope timing measured by running Nuked's counter logic rather than derived (attack rate 1 comes out at 2.84 s against a published 2.83). Patches are Freedoom's GENMIDI (BSD-3-Clause, app/ytpmv/data/), in the format Doom read, so a part gets the patch for its General MIDI instrument and a drum the patch for its key. DMX, Doom's music driver, numbers notes an octave under MIDI and the bank's offsets assume it. The OPL2 plays through a YM3014 DAC -- coarse when loud, fine when quiet -- and only reads two waveform bits, so patches asking for OPL3 waveforms fall back to an absolute or half sine, as they did on real cards.

app/ytpmv/sid.py -- the SID, from reSID: the envelope rates and sustain levels, the oscillator and 23-bit noise register, the twelve-bit R-2R DAC (the 6581's is missing a termination resistor and is not linear), and both measured cutoff curves. The 6581's floors at 220 Hz and steps back down as its register passes 0x7f; the 8580's is near linear to 12.5 kHz. Filter settings are register values, so the same patch sounds different on each chip, as tunes did. Not modelled: combined waveforms, the 6581's filter distortion and its chip-to-chip variation, and the ADSR delay bug.

The octave a part is moved by to suit a speaking voice is not applied when a chip synthesises it, since a chip has no trouble with 49 Hz.

No voice

The last entry in the page's voice dropdown, none: synthesised, leaves the corpus out entirely. It belongs to that page alone -- choosing it posts nothing, so the sentence generator and the bot keep whatever voice they had. Every part is played by the chip (off becomes the Mega Drive, and a -voice setting becomes its machine playing), no part is muted for want of a word, and the word controls are hidden. Instruments can still be chosen per part, from any machine's synthesised ones.

Each tile is then an oscilloscope of its part (app/ytpmv/scope.py): a neon red trace on black, triggered on a rising zero crossing so a held note stands still, a few cycles of the part's middle note wide. It is drawn in the console's own pixels and put through its palette like any other picture, with colours picked to land on a real red in each -- except the Game Boy, which has none, and draws in its lightest shade. Over the API it is options.synth.

Balance

Every part is measured and moved to where its job says it should sit, rather than trusting a table that cannot tell a whisper from a brass patch. On by default; --no-balance, balance=off, or the checkbox.

The measurement is the broadcast one (ITU-R BS.1770), and the part that earns its keep is the gating: a cymbal might be four hits in three minutes, and averaged across the song it looks nearly silent. Anything balancing on plain RMS would crank it until those four hits took the roof off. Gated, the same cymbal played forty times and four times measures within about a decibel.

It nudges rather than levels. Parts inside one role are not meant to be equally loud -- several decibels between two things both called "rhythm" is usually somebody's arrangement -- so anything within DEADBAND of its target is left alone and anything past that is brought back by STRENGTH of the excess. Flattening them outright was tried, measured tidier, and sounded worse. The targets in app/ytpmv/master.py are themselves measured, not guessed: render songs the old way, see what each role came to, take the median.

The palettes

app/ytpmv/screen.py. Each machine's resolution, and its colours and no others.

The Mega Drive's eight levels per channel are not evenly spaced: measured off hardware they run 0, 52, 87, 116, 144, 172, 206, 255. The NES has sixty-four colours burnt in that it could neither mix nor change; there is no single correct palette for it, since the chip emits composite video and the television decodes it, so this uses FCEUX's. The Master System is two bits a channel, taken as even quarters rather than measured. The SNES is five bits a channel on a straight ramp. Console renders are encoded harder than normal, because at the usual setting the codec smears the palette the filter just restricted.

Matrix bot

A bot account that makes videos when asked in a Matrix room:

!ytp say nice chocolate cake
!ytp mv                          the last MIDI file posted in the room
!ytp mv info                     what the song is and which words it would use
!ytp mv lead=yeah kick=boom max=60 octave:bass=+1 mode:lead=tape
!ytp voices
!ytp help

Post a .mid file, then !ytp mv (or reply to the file with it). In a direct message the prefix is optional and plain text is said.

It runs as a second compose service against the same image, talking to the web app over HTTP — so it uses the same queue, and a bot that falls over cannot take the site with it. Put its settings in a .env beside docker-compose.yml:

MATRIX_HOMESERVER=https://matrix.example.org
MATRIX_USER=@ytp:example.org
MATRIX_PASSWORD=...                 # used once; the token is saved to the volume
MATRIX_ALLOWED_USERS=*:example.org  # or MATRIX_ALLOWED_ROOMS=!abc:example.org
MATRIX_ADMINS=@you:example.org      # who may switch the voice for everyone
YTP_PUBLIC_URL=https://ytp.example.org   # optional: link to videos too big to upload
docker compose --profile matrix up -d
docker compose run --rm bot --check-config

Invite the bot to a room and it joins. Without either allow-list it answers anyone who invites it (and logs a warning saying so). Each person gets one request at a time and a cooldown (MATRIX_COOLDOWN, 20 s); song videos are capped at MATRIX_MV_MAX_SECONDS (120). Encrypted rooms are not supported — the bot leaves them rather than sit somewhere it cannot read.

Fixing a corpus by hand

http://localhost:8765/editor.html

The checker below finds words that are labelled wrong. This finds the other half: words whose timings are wrong, which is most of what makes a generation sound bad. An aligner puts an edge where it stops being confident, not where the sound starts and stops — that is why "mrs rosen" lost its N and why "time" once took its T from the word after it.

Pick a source, click a word. You get the video seeked to it, a waveform of the three seconds around it with its edges drawn on, and the neighbouring words' edges in grey. Click the waveform to move the start, shift-click to move the end, or nudge by 10ms and 50ms. Space plays just that clip, so you can hear the edit before saving it.

The optional spectrogram shows measured audio frequencies, using a bounded STFT calculation when opened. Corpus and source changes discard stale editor responses. Generation history records its voice, so clip actions apply to the corpus that produced the result. API clients can send X-Corpus: <slug> to pin their request; a write for a corpus that is no longer selected returns 409.

Words are flagged when their shape has caused trouble before — very short, very long, or overlapping the next word. On one ASMR source, 335 of 1,011 clips overlap their neighbour.

You can also relabel a word Whisper misheard, delete a clip that is unusable, or add one it skipped entirely. Every edit drops the generator's cache, so the next sentence uses the new timings. Nothing touches the videos — only the corpus database, which is what travels in the bundle.

Checking a corpus

Whisper's mistakes are invisible from inside. A word it mishears is stored confidently under the wrong spelling, so the corpus looks complete and simply never produces that word — "rupees" was transcribed as "rubies", the most quotable word in that corpus, unreachable by typing it.

python scripts/verify_corpus.py --all
python scripts/verify_corpus.py --source-id 3 --show 40

This fetches YouTube's own captions and matches them to the stored labels by timestamp, reporting missed words (caption text with no clip near it, so speech the transcription skipped) and disagreements (both systems heard something at the same moment and spelled it differently).

Captions are not ground truth. Uploader-written subtitles would be, but most channels do not publish any; what is available is Google's speech recognition, with its own failures. The value is that it is a different system from Whisper — the two agreeing means something, and the two disagreeing is worth a person deciding. Nothing is edited: to act on a finding, put a transcript you trust in transcripts/<video_id>.txt and run correct.py.

Repeated disagreements matter far more than one-offs. A word both systems disagree about every time it is said is a systematic mishearing; a single one is usually the two of them splitting a phrase differently.

Teaching it words

Any channel invents words a dictionary has never held — names, in-jokes, brand names, coinages — and a word with no pronunciation cannot be spliced at all. There are two places to teach it one:

$MRS_DATA_DIR/pronunciations.csv     every corpus on this machine
corpora/<name>/pronunciations.csv    just that one, and packs with its bundle

Use the global file. Most of what needs teaching is not specific to a speaker — "usb", "wii" and "kilometres" are the same words whoever says them — and a per-corpus copy would have to be maintained again for every channel ingested. The per-corpus file is for a speaker's own coinages; it overrides the global one. Packing includes both files' rules in the bundle's dictionary, so a corpus arrives with its pronunciations on a machine that has never heard of them.

Two columns, and the second can be written whichever way suits:

nug,N AH G          # ARPAbet, if you know it
wii,wee             # or just a word that already sounds right
mcnug,mick nug      # several words are fine
usb,=letters        # read it out letter by letter
hevexum,=skip       # leave it unsayable on purpose

The second form is the one to reach for. Nobody should have to learn ARPAbet to say that "wii" rhymes with "wee".

To find out what is worth adding:

python scripts/verify_corpus.py --unspliceable --write-template

That lists every stored word with no pronunciation, worst first, and writes them into the global CSV commented out, ready to fill in (--per-corpus to keep them with one corpus instead). It needs no network, and edits take effect on the next POST /api/reload.

Entries here beat everything built in, and a line that cannot be parsed is reported rather than ignored — silence would leave the word unsayable with nothing to explain why.

How hard it tries

A corpus can only say what it has heard. When a word isn't there, the splicer builds it from phonemes taken out of other words — and when even those aren't there, what happens next is a per-corpus setting, stored in the corpus database so it travels inside the bundle:

mode behaviour
strict real clips and clean splices only; anything else is reported missing (default)
loose substitutes a near-enough phoneme, and guesses a pronunciation for words the dictionary doesn't have
desperate as loose, and drops sounds nothing can cover — always produces something

Set it from the effort dropdown next to the voice selector, or:

curl -X POST localhost:8765/api/splice-mode -H 'Content-Type: application/json' -d '{"mode":"loose"}'

The modes are additive: substitutions and dropped phonemes are priced far above any achievable saving, so wherever strict finds a splice, all three modes give the identical result. Turning it up only changes words that would otherwise come back missing.

A word that needed a substitution, a dropped sound, or a guessed pronunciation is marked approx in the API and shown with a dashed border, so an approximation is never passed off as a faithful splice.

Which you want depends on the corpus. With 7,000 distinct words, strict rarely gives up on anything and the other modes barely fire. With 30 words, strict can say almost nothing, and desperate is the difference between a corpus that works and one that only quotes itself.

Installing a corpus on a running server

Drop a bundle in packs/ and restart — the entrypoint installs anything it finds there. Or do it directly:

python scripts/corpus.py install my-corpus.tar.zst --name my-corpus
curl -X POST localhost:8765/api/reload

MRS_CORPUS pins which one is served. Without it the first alphabetically wins, so adding a corpus can silently switch voices.

To add to a corpus that's already running, the container carries the ingest dependencies too:

docker compose run --rm app python scripts/ingest.py <youtube-url>
docker compose run --rm app python scripts/refine_boundaries.py --all --apply
curl -X POST localhost:8765/api/reload

API

Generation is queued, not done in the request: a long sentence takes minutes and any proxy in front will 504 first. Submitting returns a job id; poll it.

id=$(curl -s -X POST localhost:8765/api/generate -H 'Content-Type: application/json' -d '{"text":"nice chocolate cake"}' | jq -r .id)
curl -s localhost:8765/api/jobs/$id | jq

?wait=1 blocks instead, which is easier from a script.

One job runs at a time. Generation is ffmpeg-bound on a shared box, and two at once exhausted the container's thread ceiling before the queue existed.

Endpoint Purpose
POST /api/generate {"text": "..."} → 202 with a job id. ?wait=1 to block
GET /api/jobs/{id} status, stage, progress, result
GET /api/queue queued and running counts
GET /api/words vocabulary with clip counts; the frontend fetches it once
GET /api/suggest autocomplete from real spoken runs
POST /api/rate vote a clip up or down for that word
GET/POST /api/splice-mode read or set how hard the splicer tries
POST /api/reload drop the clip cache after an ingest or correction
GET /api/stats corpus totals
POST /api/ytpmv/midi upload a MIDI file (multipart midi) → key, tempo, parts, suggested sounds
GET /api/ytpmv/midi/{id}/part/{part_id}?corpus={slug} suggested sounds for one part from an installed corpus
POST /api/ytpmv/sample {"text","take","corpus"} → the clip as an instrument, and the note it is on; corpus is optional
POST /api/ytpmv/preview {"midi_id","part"} → a few seconds of one part, audio only
GET /api/ytpmv/library uploaded MIDIs with searchable song metadata
PATCH /api/ytpmv/midi/{id} {"title"} → rename a MIDI and invalidate its cached metadata
DELETE /api/ytpmv/midi/{id} remove the MIDI and sidecars; reject deletion while a queued/running job uses it
POST /api/ytpmv/preview-mix {"midi_id","parts","options","start","seconds","part_id","note","notes"} → queued audio; null seconds plays to song end, part_id isolates a short pattern, note auditions one pitch, notes prepares a bank of 1–24 distinct pitches with offsets in the result
POST /api/ytpmv/render {"midi_id","parts","options"} → 202 with a job id, polled like the rest

Tests

python tests/test_tokenize.py        # tokeniser: expansions, markers, boundaries
python tests/test_splice_modes.py    # pronunciations, substitutions, cost order
python tests/test_corpus_select.py   # which corpus a run writes into
python tests/test_bundle.py          # a bundle survives a round trip
python tests/test_corpus_export.py   # live WAL snapshots and portable pronunciation rules
python tests/test_corpus_metadata.py # corrections and edits preserve unrelated clip data
python tests/test_numeric_alignment.py # numbers and ordinals keep their spoken meaning
python tests/test_source_formats.py # original and rotated videos concatenate correctly
python tests/test_clip_cache.py     # voice/source/window identity and actual cached audio
python tests/test_pitch.py           # YTPMV: a wobbling vowel lands on the note
python tests/test_music.py           # YTPMV: tempo, key and parts from a MIDI file
python tests/test_matrix_commands.py # what the Matrix bot hears in a message

These run in CI and need nothing but num2words, nltk, zstandard, numpy and mido.

python tests/test_end_to_end.py      # ~30s: one real video, start to finish

That one is not in CI — it downloads a 63-second video, transcribes it, refines the boundaries and generates sentences from it, so it needs the network, ffmpeg, torch and a Whisper model. Run it before shipping anything that touches ingest, clip selection or the ffmpeg builder. It is the only test that exercises those together, and the bugs that have escaped so far escaped between the parts rather than inside any one of them.

Layout

main.py                       FastAPI app; API, frontend, output/
app/generate.py               sentence -> clips -> ffmpeg
app/phonemes.py               CMU pronunciations, phoneme-level splice planning
app/forced_align.py           torchaudio CTC alignment for sub-word timings
app/database.py               SQLite schema, corpus selection, portable paths
app/device.py                 CPU or CUDA, chosen once for every model
scripts/build_corpus.py       channel link -> packed corpus, one command
scripts/ingest.py             download, transcribe, store word clips
scripts/ingest_channel.py     the same, for a whole channel
scripts/realign.py            better word timestamps via stable-ts
scripts/second_opinion.py     checks Whisper's words against Qwen3-ASR; fixes or marks disagreements
scripts/refine_boundaries.py  word boundaries from CTC alignment and the audio; marks noise-words
scripts/measure_tone.py       each take's tone (pitch, loudness, pace...), grouped into categories
scripts/correct.py            relabel misheard words from a real transcript
scripts/find_noises.py        pull non-verbal noises out of the gaps
app/editor.py                 the corpus editor's API (the only writer
                              outside an ingest)
frontend/editor.html          fix words and timings by hand, with a waveform
app/ytpmv/                    MIDI in, pitch-perfect grid video out
scripts/ytpmv.py              the same from a terminal, with a tuning check
app/matrix_bot.py             the Matrix bot (python -m app.matrix_bot)
frontend/ytpmv.html           the YTPMV page
scripts/verify_corpus.py      check stored labels against YouTube captions
scripts/corpus.py             pack / unpack / migrate / inspect
corpora/<name>/               one voice: corpus.db + downloads/ + transcripts/
packs/                        drop bundles here to install on next start

Upgrading a pre-corpora/ install

Older installs kept the database loose in the data directory with downloads/ beside it. Still read, but it can't sit next to another corpus — only one downloads/ to go round.

The container migrates itself on start, so a redeploy is enough. By hand:

python scripts/corpus.py migrate            # report only
python scripts/corpus.py migrate --apply    # -> corpora/michael-rosen/

It moves rather than copies, and is a no-op once done. If you pinned MRS_CORPUS=default, change it to the migrated name.

Source material

The shipped corpus comes from publicly posted recordings of Michael Rosen reading his own children's poetry. Nothing in it is adult or offensive. By default nothing is synthesised either: a word with no clips and no viable splice is reported missing rather than approximated, and the modes above have to be turned up deliberately, per corpus.

It's a toy for making silly videos out of poetry readings. Keep it there, and don't use it to put words in anyone's mouth in a way that misrepresents them.