Type a sentence, get a video of it being said, cut word by word from recorded speech. The shipped corpus is Michael Rosen reading his own poetry. Nothing in the code is specific to him — point the ingest scripts at any speaker.
Words the corpus has are spliced in directly. Words it doesn't are built from
phonemes taken out of other words, so rice can come from the r of one word
and the ice of another.
See the repository audit for the recent ingestion, security, performance and editor fixes, their verification, and remaining work.
From a clone:
docker compose upOpen http://localhost:8765 and type a sentence. That is the whole thing — on first start the container downloads the published Michael Rosen corpus into its volume, so a fresh clone already works.
That pulls the published image. Add --build only when you have changed the
code, which saves compiling 3 GB that already exists on GHCR.
The image is on GHCR if you would rather not use compose:
docker pull ghcr.io/kapsikkum/ytp-machine:latestIt carries no corpus — one has to be mounted, because a voice is 100 MB to 2 GB of video and does not belong inside an image. So point it at one:
docker run -p 127.0.0.1:8765:8765 -v ytp-data:/app/data -v ./corpora/some-voice:/corpus:ro ghcr.io/kapsikkum/ytp-machine:latestAny of three sources works: a corpus directory or a .tar.zst bundle mounted at
/corpus, bundles dropped in /packs, or CORPUS_URL pointing at one over
HTTP. With none of them it says exactly that and stops, rather than serving an
app that answers everything with "no clips found".
With no corpus of your own, CORPUS_URL is the way in. The shipped voice is a
bundle on the corpus release, which is the only place a 100 MB+ bundle can sit
at a stable public URL:
docker run -p 127.0.0.1:8765:8765 -v ytp-data:/app/data -e CORPUS_URL=https://github.com/kapsikkum/ytp-machine/releases/download/corpus/michael-rosen.tar.zst ghcr.io/kapsikkum/ytp-machine:latestThe bundle is packed and uploaded by hand (see The shipped corpus).
Pin the tag for anything you care about — :latest moves under you on the next
push, and every commit is also tagged by its full sha:
docker pull ghcr.io/kapsikkum/ytp-machine:e4f6b8250ad18f0157acae513c779e83e7d7ca6eNo GPU needed to run it. You do want one to build a new corpus.
Compose publishes port 8765 on localhost by default. The operator API can edit
and replace corpora and has no built-in login. For remote use, put an
authenticated reverse proxy in front of it and set MRS_BIND_ADDRESS to the
interface the proxy needs. Browser origin checks and Matrix allow-lists do not
provide authentication for the HTTP API.
For an HTTPS reverse proxy, preserve Host and set X-Forwarded-Proto to
the external scheme. Set the app container's FORWARDED_ALLOW_IPS to the
proxy's IP as seen inside the container (for example, 127.0.0.1,10.88.0.1
for a host nginx reaching Podman's default bridge). Compose accepts this in
.env; NixOS OCI containers can set it in environment.FORWARDED_ALLOW_IPS.
Uvicorn otherwise trusts only loopback and sees HTTP, causing HTTPS browser
writes to fail with not from another origin. Trust specific proxy addresses,
not *, and have the proxy overwrite forwarded headers. Requests from other
origins must still receive 403.
Mounted corpus directories are seeded with a consistent SQLite snapshot,
including committed WAL writes and pronunciation overrides. Imports validate
in staging before activation. Forced replacements retain the previous corpus
under corpora/.backups/; loose unpack --force keeps replaced members under
.corpus-backups/. Allow disk space for staging and these retained backups.
Stop the app before running the legacy migration command.
1. One environment with a GPU, somewhere your file sync will not touch
(OneDrive locks DLLs mid-install and torch dies with a WinError 32 naming an
innocent file):
python -m venv C:/Users/you/.venvs/ytpC:/Users/you/.venvs/ytp/Scripts/python.exe -m pip install --index-url https://download.pytorch.org/whl/cu128 torch torchaudioC:/Users/you/.venvs/ytp/Scripts/python.exe -m pip install -r requirements.txt2. Build it. One command does download, transcribe, align and pack:
python scripts/build_corpus.py @SomeChannel --limit 10 --data-dir C:/Users/you/ytp-corpora/some-channelTen long videos is a good first target — roughly an hour of GPU time and about
5,000 distinct words. --limit 0 takes the whole channel. Re-running skips what
is already ingested, so a run you interrupt just carries on.
3. Install it wherever the server runs:
python scripts/corpus.py install some-channel.tar.zst --name some-channelThen pick the voice from the dropdown, or POST /api/corpus.
4. Optional, but it is what makes it sound good. The build leaves a
pronunciations.csv in the data directory listing every word it cannot
pronounce, commented out. Uncomment a line and say what the word rhymes with:
wii,wee
nug,nugget
usb,=lettersThen POST /api/reload. To rebuild that list later, point it at the corpus:
MRS_DATA_DIR=C:/Users/you/ytp-corpora/some-channel MRS_CORPUS=some-channel python scripts/verify_corpus.py --unspliceable --write-template --per-corpusdocker compose up --buildhttp://localhost:8765. On a fresh volume it needs a corpus — see below.
No GPU needed. Torch comes from the CPU wheel index; the default index pulls gigabytes of CUDA payload a headless server never touches.
Without Docker:
pip install -r requirements.txt
python -m uvicorn main:app --port 8765ffmpeg must be on PATH either way.
- Ingest downloads a video, transcribes with Whisper, refines timings with stable-ts.
- Alignment (
app/forced_align.py) runs torchaudio's CTC aligner (Wav2Vec2 BASE 960h) for character-level timings. That's what makes sub-word cutting work. - Generation (
app/generate.py) resolves a sentence into clips, preferring the longest real spoken runs it can find, and falls back to phoneme splicing per word. ffmpeg cuts and concatenates.
Votes on a clip are stored in splice_ratings and weight it up or down for
that word next time.
One voice: a directory holding one database and the videos it indexes.
corpora/michael-rosen/corpus.db timings: this word, this file, 4.12s-4.61s
corpora/michael-rosen/downloads/ the videos those timings point into
corpora/michael-rosen/transcripts/ true text, where there is any
The database stores only timings, so the two halves are useless apart and a corpus always moves as a whole. Paths inside are relative to the corpus directory, so copying it anywhere works. Michael Rosen's is ~110 MB: 48 videos, ~9,500 clips, ~1,400 distinct words.
Corpora are not in git — binary that never delta-compresses, and derived data besides. They travel as a bundle: one tarball of database + videos + transcripts.
Not in the repo. The Michael Rosen voice is published as a bundle on the
corpus release, and that bundle is the only copy that ships: compose and the
image seed from it, and CI installs it to test against. A clone has no
corpora/ at all; corpora/ is ignored, so nothing built locally ends up in
history.
To get it locally:
gh release download corpus --pattern michael-rosen.tar.zst
python scripts/corpus.py install michael-rosen.tar.zst --name michael-rosenThe release is made by hand, from a corpus that has been checked: refine its boundaries, listen, then pack and upload over the old asset.
python scripts/corpus.py pack --corpus michael-rosen --out michael-rosen.tar.zst
python scripts/corpus.py info michael-rosen.tar.zst
gh release upload corpus michael-rosen.tar.zst --clobberTo move one somewhere else:
python scripts/corpus.py pack # -> corpus-YYYY-MM-DD.tar.zst
python scripts/corpus.py unpack corpus-*.tar.zst
CORPUS_URL=https://example.invalid/corpus.tar.gz docker compose upThe entrypoint takes any of three: loose files at /corpus, a bundle at
/corpus, or CORPUS_URL. With none and no database it says what's missing
and stops.
output/ stays out of git — about a gigabyte, all reproducible in seconds.
Any speaker with a decent amount of clear, single-voice footage works. More distinct words means fewer splices and better output.
One command does the lot — download, transcribe, align, pack:
python scripts/build_corpus.py @GarbageTime420 --limit 40
python scripts/build_corpus.py https://youtube.com/@Name --name my-corpus --model mediumIt names the corpus from the channel handle unless you pass --name, and
defaults to a target of 40 total videos in the corpus (--limit 0 for the whole
channel). Videos already ingested count towards that target, so interrupting at
25/40 and rerunning adds up to 15 more. The standalone ingest_channel.py
counts new videos instead. Use
--skip-ingest / --skip-refine / --skip-pack to redo part of it, and
--data-dir to build somewhere other than the project root.
The build transcribes first, refines word boundaries second, and places phonemes
last. Each model runs in its own process, which keeps Whisper and the phoneme
model from sharing GPU memory. --skip-align also skips alignment during ingest.
Phoneme results are checkpointed every 50 clips and on a handled interruption,
so resuming skips completed clips without a database transaction for each one.
If a clip is relabeled or retimed while alignment runs, the checkpoint leaves
that edit intact. Standalone channel ingestion also resumes missing phoneme
alignment for previously ingested videos. Empty legacy sources remain eligible
for ingestion.
It checks the dependencies, ffmpeg and the GPU before starting, because the alternative is finding out four hours in.
Use a GPU. All the cost is transcription and alignment. Every script takes
--device auto|cuda|cpu (or $MRS_DEVICE), default auto. On an RTX 3070 Ti
transcription runs ~10x realtime against ~1x on CPU.
requirements.txt installs CPU torch on purpose, so a GPU ingest wants its own
environment:
python -m venv .venv-cuda
.venv-cuda/bin/pip install --index-url https://download.pytorch.org/whl/cu128 torch torchaudio
.venv-cuda/bin/pip install -r requirements.txtOn Windows keep that venv, and the corpus, out of OneDrive. The sync engine
locks DLLs mid-install and torch then fails with WinError 32 naming an
innocent-looking file.
build_corpus.py runs 1, 2 and 5. Steps 3 and 4 need human input and are
optional.
1. Ingest. One video, a local file, or a channel:
python scripts/ingest.py https://www.youtube.com/watch?v=XXXXXXXXXXX
python scripts/ingest.py /path/to/video.mp4 --model medium
python scripts/ingest_channel.py https://www.youtube.com/@SomeChannel/videos --limit 40 --skip-errors--model trades speed for accuracy. Every later step inherits these labels, so
small or medium is worth it. --normalise re-encodes to the corpus format
(480x270), which makes bundles far smaller and generation cheaper.
Original imports also render correctly: generation checks their actual size,
frame rate and display rotation, then scales with letterboxing when needed.
Local imports refuse to overwrite an existing recording with the same filename.
For English speech, --language en avoids detecting the language from the
opening audio. The English-only models (small.en, medium.en) and turbo
are also accepted by build, ingest and realign. Compare models on a short,
manually transcribed sample from the speaker before committing to a whole
channel: the fastest model and the most accurate one depend on the material.
The word and phoneme aligners currently use English models and pronunciations.
Supply a short vocabulary hint when names or recurring terms are misheard:
python scripts/build_corpus.py @SomeChannel --model medium.en --language en --initial-prompt "Names and terms: Morshu, Hyrule, rupees." --limit 10These options also work with scripts/realign.py, which reuses downloaded
videos. Forced alignment adjusts timings; misheard words still need a better
transcription or a corrected transcript. Empty transcriptions and invalid word
durations are rejected, and refinement preserves hand-edited clip boundaries.
2. Check the words. Whisper now and then garbles a stretch without saying so — "only thing that happened, I got a rotten headache" came out as "hes all down hes all" — and every later stage believes it. A second transcription with Qwen3-ASR catches that:
pip install qwen-asr # pins its own transformers: a venv of its own is easiest
python scripts/second_opinion.py --all # dry run: every change it would make
python scripts/second_opinion.py --all --apply # before refine_boundariesWhere the two agree nothing changes. A word swapped one for one is relabelled
where it stands; a stretch heard as a different number of words is replaced by
Qwen's words, which the refinement then places; words only Qwen heard are
added; words only Whisper heard are kept but marked "disputed" (rarely picked,
and left out of the refinement's alignment). Spelling variants ("play time" /
"playtime", -ise / -ize) and anything next to a hand-edited clip are left
alone. build_corpus.py runs it when qwen-asr is installed.
3. Sharpen the timings. Whisper's word boundaries run early and wander by 100–350 ms, so short clips grab the tail of the previous word:
python scripts/realign.py --model small # redo timestamps, e.g. after a model change
python scripts/refine_boundaries.py --source-id 3 # CTC alignment, dry run on one
python scripts/refine_boundaries.py --all --apply # frame-accurate, everythingrefine_boundaries matters most for splice quality. The aligner decides which
stretch of sound is which word; the cut itself is then placed from the audio —
at the edges of a pause, or the quietest point between two words that run
together — because CTC fires late on a word's first letter and early on its
last. Clips whose letters the aligner cannot find where Whisper put them --
mostly noises transcribed as words ("chk chk chk", a breath as "a") -- are
marked with an "unheard" boundary report rather than removed, since some are
real words said too quickly to score: a marked take is rarely picked, still
there when a word has no other, and listed for review in the editor. Hand-
edited clips are never moved or marked. Nor does a cut run on into speech no
word accounts for — an unwritten "erm" straight after "Brian" — where the
aligner's model hears letters there: it stops at the dip before it. A dry run
prints what it would mark.
It needs --all or a --source-id to pick targets and --apply to write.
python scripts/measure_tone.py # after refining: tone of every takemeasure_tone describes how each take sounds — pitch, where its pitch starts
and ends, how much it moves, loudness against its own recording, pace and
brightness — and groups the corpus into a few tone categories. The generator
then keeps a sentence in one tone where the words allow it, joins takes that
start near where the last one ended, and stays in the same recording: the
same tone setting, 0 to turn it off. It runs at build time, takes a minute
or two, needs no model, and must be re-run after boundaries move (refining or
editing clears the rows it measured). Corpus schema v2 carries it; v1 corpora
upgrade on open and pick exactly as before until measured.
Re-transcription refuses sources with hand-edited clips unless you pass
--replace-edited. It also aborts if clips or the recording change during the
run. Replacement removes stale dependent alignments and votes. Pinned splice
recipes are remapped when a new clip unambiguously contains the same cut;
otherwise their clip pin is released while retaining the sound preference.
4. Fix misheard labels (optional, needs a real transcript). Relabels only
high-confidence phonetically-close swaps, never timestamps. Put the true text
in the active corpus's transcripts/<video_id>.txt (--transcripts-dir overrides
the directory). Relabeling clears the affected character and phoneme alignment
so it can be measured against the corrected word:
python scripts/correct.py --all # dry run
python scripts/correct.py --all --apply5. Catch non-verbal noises — clicks, pops, spews — which Whisper skips:
python scripts/find_noises.py # dry run
python scripts/find_noises.py --applyNot generic: it carries a hardcoded list of two Michael Rosen videos, because
scanning everything turned up mostly breaths. For another speaker edit
CURATED at the top of main(), or skip the step.
6. Pack.
python scripts/corpus.py pack --corpus <name>
python scripts/corpus.py info corpus-*.tar.zstpack takes a consistent SQLite backup, including committed WAL writes,
without checkpointing the live database. Bundles carry the effective global
and voice pronunciation rules together, with voice overrides last. They are
.tar.zst, or .tar.gz if zstandard isn't installed. unpack reads either.
Two things are not plain words:
| typed | does |
|---|---|
~word~ |
plays that word backwards |
*noise* |
a non-verbal clip (*spew*, *click*) rather than the spoken word |
Reversal happens at cut time, not in the corpus. Reversed speech is not speech: Whisper transcribes it into confident nonsense, so a corpus built from a backwards recording is a set of noises labelled with words nobody said, and it poisons every later splice. Reversing on the way out costs nothing when unused and works for any word in any corpus.
A run is one clip, so it is reversed or not as a whole — marking one word ends the run there rather than quietly reversing its neighbours.
http://localhost:8765/ytpmv.html
Drop in a MIDI file and the corpus plays it: one tile per instrument, the
clip re-triggered on every note, and every note pitch perfect. The clip's
wobbling speech pitch is measured frame by frame and laid back down exactly on
the note's frequency (TD-PSOLA, app/ytpmv/pitch.py), which keeps the voice's
character while putting it in tune — solo notes measure within a few cents.
You don't need to know any music theory. The file is read for you:
- key, tempo, time signature and length, described in plain words
- what each part is: tune, backing, chords, bass, or a piece of the drum kit (channel 10 is split into kick, snare, hats, toms and cymbals)
- a word for each part, picked for the kind of sound it makes: a plosive for a kick, a hiss for a hi-hat, a long steady vowel for a tune. Every take of the candidate words is measured, so the take picked is the one that holds a pitch best.
- an octave for each part, so a bass line at E1 is sung where the voice can sing it. The notes keep their names, so it is still the same song.
Change any of it: type a word or phrase, cycle through takes, click an
alternative, press ▶ part to hear a part on its own, move an octave, mute a
part, choose tape for the classic chipmunk sound instead of pitch perfect.
Render goes through the same queue as sentences.
Voice parts also have a corpus dropdown containing the installed voices. Choose a different performer for each part, or use the Song default group. The closed menu shows the actual performer in either case. Default voice and Global chip stay above the arrangement; Auto tracks follow that chip, while explicitly chosen track instruments stay fixed. Switching a part's corpus refreshes its suggested sounds, takes and vocabulary; previews and the finished video use that same corpus. The voice at the top is this page's default and leaves the server's active corpus unchanged. Explicit part choices survive changes to that default and are restored on page reload.
The DAW transport plays from the playhead to the end when no range is selected.
Shift + drag selects a playback range; Clear range returns to whole-song
playback. Playback uses the same mixer as the video render, including speed,
key, pan, levels, chip, variety and balance, within YTPMV_MAX_SECONDS (15
minutes by default). Space plays/pauses throughout the editor except while
editing text or a control. The transport clock and timeline cursor use the
audio clock; return-to-start and ±10-second buttons seek it directly. There
is one playback UI; the browser's native audio controls are hidden. Solo
affects playback only. Stop also cancels pending audio preparation.
Click the sound name in a track header for its instrument window: voice, sample/take, up to twelve suggestions, chip instrument and GM patch controls, a note-audition keyboard and a waveform decoded from the actual audition audio. Opening the instrument prepares all thirteen visible keys in one audio bank. The browser decodes it once and plays cached notes locally through Web Audio, including overlapping key presses, without per-key generation requests. Changing the voice, sample/take, instrument/patch or sound settings rebuilds the bank; returning to a recently used sound reuses its cache. Decoded banks are bounded to six entries and 48 MiB in the tab. Initial preparation shows a loading state. Keys audition one note; Play track pattern auditions only that track from its first note. The dock's part button auditions in place without opening the instrument window. Instrument auditions are separate from song playback.
MIDI library replaces the editor while browsing. Search, sort, rename inline or delete an uploaded MIDI with confirmation. Rename updates an open song; deleting that song clears its local arrangement and returns to the library. Existing rendered files remain. Files used by queued/running renders cannot be deleted until those jobs stop. MIDI files are shared server-side; track settings remain local to the device.
The header switches between Simple (the original stacked parts and render
layout) and Advanced (the DAW workbench). Both share the current engine,
fixes, corpus selection and saved arrangements. The interface preference is
remembered on this device; ?mode=simple or ?mode=advanced selects it explicitly.
The track workbench shows real MIDI activity above a dock for the selected part. Pick a track to edit its voice, sample or chip instrument; its corpus selector, quick sound button and M/S controls work directly from the header. Track, Song & render, and Video preview keep local settings, global options, and the composition in separate dock panels. Find & manage tracks holds the search and role filters. OPL and SNES expose their GM patch banks beside the instrument selector; automatic chip sounds follow a changed MIDI patch. Chip instruments play independently of corpus clips and draw waveform tiles.
The timeline scrolls vertically through tracks and horizontally when zoomed. Its ruler and track controls stay pinned, and large arrangements draw only nearby track rows. Use Fit song, the zoom slider, Focus range, or Part start to navigate. Shift + wheel scrolls through time; Ctrl + wheel zooms around the pointer. Click or drag to seek, or Shift + drag to choose a playback range. The playhead follows the actual audio; Follow playback keeps it in view. Focus the timeline for arrows to seek, Shift + arrows to jump ten seconds, and Home/End to move to the ends. Zoomed views request bounded note detail for their visible time range without re-running voice recommendations. The MIDI note view is read-only.
Each MIDI keeps its own arrangement, voice and render options in this browser's local storage. Undo reverses edits in the current session. Settings reveals pitch, pan, note length and tile controls. Use takes & clip to watch the sample and choose a numbered take inline. Bulk voice controls live with the song settings. If browser storage fills or is blocked, the save indicator reports the failure.
Video exports use a 1920×1080 (16:9) canvas. Video preview offers Classic grid, Balanced grid, Two rows, Two columns, Vertical strips and Featured track. Classic grid restores equal 16:9 tiles using the original grid arrangement. Center tiles centers the grid and incomplete rows by default: three streams have two on top and one centered below. Turn it off for top-left alignment. Unused space stays black. The other layouts share the full available space.
Framing → Fit whole clip preserves the entire picture, centered inside each tile with black bars where needed. Fill tile (crop) covers the tile with a center crop. Neither stretches the source. The preview and actual video use the same framing. Drag tiles to reorder them with animated movement, or use Alt + left/right on a focused tile. Layout, alignment, framing and order persist per MIDI and work in both Simple and Advanced interfaces.
Rendering keeps the last successful export visible. Cancel render stops a queued or running job; a refresh reconnects to an acknowledged job, and reconnect recovers from interrupted polling. Each export records its song and settings, and the page marks it when the arrangement has changed. Recent exports are remembered on this device; server cleanup can eventually remove their media, and a server restart can discard unfinished in-memory jobs.
From a terminal:
python scripts/ytpmv.py song.mid --info
python scripts/ytpmv.py song.mid --part lead=yeah --part kick=boom --max 30
python scripts/ytpmv.py song.mid --only bass --check # measure the tuningYTPMV_MAX_SECONDS caps the length of a render (default 900, fifteen minutes).
chip= to the bot, --chip from the terminal, or the dropdown plays the song
on an emulated console -- sound and picture both:
python scripts/ytpmv.py song.mid --chip md # four-operator FM
python scripts/ytpmv.py song.mid --chip snes-voice # him, through a SNES| setting | sound | picture |
|---|---|---|
md |
the YM2612's FM operators; the voice is gone | 320x224, 9-bit colour |
md-voice |
him, out through the Mega Drive's 8-bit DAC | 320x224, 9-bit colour |
nes |
the 2A03's pulses, triangle and noise; the voice is gone | 256x240, the 64-colour palette |
nes-voice |
him, out through the NES's delta-modulation channel | 256x240, the 64-colour palette |
sms |
the SN76489's three squares and noise; the voice is gone | 256x192, 6-bit colour |
sms-voice |
him, hammered out of the Master System's volume register | 256x192, 6-bit colour |
snes |
a General MIDI bank cut down to SNES-sized BRR samples, through the S-DSP | 256x224, 15-bit colour |
snes-voice |
him, through the S-DSP's BRR, interpolation and echo | 256x224, 15-bit colour |
gb |
the Game Boy's pulses, wave table and noise; the voice is gone | 160x144, four greens |
gb-voice |
him, pushed through the wave table four bits a step | 160x144, four greens |
gbc |
the same chip as gb |
160x144, the Color's washed-out LCD |
gbc-voice |
the same as gb-voice |
160x144, the Color's washed-out LCD |
opl2 |
an AdLib's OPL2 playing a Doom-format GENMIDI bank | 320x200, EGA's sixteen colours |
opl2-voice |
him, off an original Sound Blaster's 8-bit DAC | 320x200, EGA's sixteen colours |
opl3 |
a Sound Blaster's OPL3, same bank, all eight waveforms | 320x200, VGA |
opl3-voice |
him, off a 16-bit Sound Blaster | 320x200, VGA |
sid |
the C64's 6581 SID: pulse, saw, triangle, noise, and its filter | 320x200, the VIC-II's sixteen |
sid-voice |
him, hammered out of the 6581's volume register | 320x200, the VIC-II's sixteen |
sid8580 |
the later 8580 SID | 320x200, the VIC-II's sixteen |
sid8580-voice |
the same trick on an 8580, where it barely works | 320x200, the VIC-II's sixteen |
The picture is not a separate choice. A Mega Drive soundtrack over a picture the Mega Drive could never have drawn is two machines, and nobody asking for one meant that.
The synthesised settings take every part, drums included, and keep only how
long each note lasts and how loud it was. The -voice settings keep the whole
trick -- the clip still pitch-tracked onto the exact note -- and play it out
through the channel that console used for speech. The SNES has no other mode:
it synthesises nothing, so it is always him.
Any part can be given its own instrument: the page's instrument picker on
each part, or the bot's tone:<part>=. Left on auto, a part plays what the
chip picks for it; anything chosen stays whatever the chip is, so one song can
mix machines -- an NES triangle bass under a SID lead. voice keeps a part as
him, untouched, while everything else goes through the chip (clean cannot
mean that, because the page used to send it back for every part). Switching
chip on the page puts a part back to auto only if its choice belonged to the
machine being left behind.
A part's General MIDI instrument can be swapped too -- the page's patch,
shown when an OPL plays the part, or the bot's program:<part>= (0 to 127, so
program:lead=29 is overdriven guitar). On an OPL that is the patch. The other
chips mostly choose by the part's job, and only fall back to the instrument for
a part with no job they know.
app/ytpmv/ym2612.py -- the Mega Drive. A model rather than a cycle-exact
emulator like Nuked-OPN2: the log-domain sine and 14-bit operators, the real
envelope generator, all eight algorithms, feedback, detune, 53267 samples a
second, and the output stage, which matters because FM lands a sideband on DC
and without the console's coupling capacitor a song rumbles. Its bass is Sonic
2's own voice $00, read out of the disassembly. SMPS numbers operators
backwards from the hardware, so its op1 is operator 4; get that wrong and
operators 2 and 3 swap, which in a chain is a bright FM bass or a near-pure
sine.
app/ytpmv/nes.py -- the NES, checked against Blargg's Nes_Snd_Emu. It
generates at four times the output rate and filters on the way down, because
the noise register clocks at up to 447 kHz. The triangle runs out of timer
resolution an octave before the pulses and goes flat at the top of its range;
that is the hardware.
app/ytpmv/sms.py -- the Master System, checked against SMSPower's notes:
register $0FE is 440.4 Hz. It cannot go below 109 Hz, so anything written
lower is raised by octaves. Its documented noise taps are not a maximal-length
polynomial and repeat after 57337 steps, not 65535.
app/ytpmv/snes.py -- the SNES, checked against fullsnes: the four BRR
predictors and the 512-entry interpolation table, whose taps sum to 2048 at
every phase. BRR works in fifteen-bit integers; fed floats between -1 and 1 it
quantises everything to nothing. Most of the famous muffle is not the chip but
the 64 KB the whole soundtrack had to fit in, which kept samples far below the
DSP's rate -- stored_hz models that practice, and says so.
Its instruments are DitherEmotion's ExpressiveSNES, a General MIDI SoundFont
of samples ripped from SNES games, cut down by scripts/build_snes_bank.py to
what a game carried: one sample per General MIDI program and one per drum key,
loops on whole 16-sample BRR blocks, kept at the DSP's 32 kHz since they came
out of the chip's own format in the first place (and so not compressed twice).
3.8 MB, app/ytpmv/data/snes_bank.npz, credited in SNES-BANK-SOURCE.txt as
the font's author asks of edited versions. Each keeps the font's volume
envelope, tuning and level, and the bank records which font it came from.
Playback goes through the chip's Gaussian table and the echo, and a note may go
past the two octaves the pitch register allows, since a game chose its samples
to fit and a General MIDI part cannot.
Any General MIDI font can replace it -- python scripts/build_snes_bank.py some.sf2, with --store-hz 32000 --ripped for one ripped from games. Fonts
thrown together from rips are often labelled an octave out; --fix-octaves
corrects a sample whose loop plays a whole octave from its label, but check it
by ear, since an organ's quiet sub-octave can fool it.
app/ytpmv/gb.py -- the Game Boy, which the Color shares unchanged. Formulas
from the gbdev Pan Docs; the output capacitor from SameBoy. Each channel's DAC
idles at +1, so a held square sits off centre and the capacitor drags it back
the whole time -- squares sag, and that is much of the sound. The gbdev wiki
gives the Color a separate capacitor figure that works out to a 700 Hz
high-pass, which would take the bass off entirely; SameBoy uses one figure for
every model, a 28 Hz corner, and so does this. The two machines differ in
their screens: ares's four greens for the Game Boy, and ares's model of the
Color's LCD, which bled each colour into the others and never reached full
brightness.
app/ytpmv/opl.py -- the OPL2 and OPL3. Tables from Nuked-OPL3, whose
log-sine and exponent ROMs turn out to be exact formulas; envelope timing
measured by running Nuked's counter logic rather than derived (attack rate 1
comes out at 2.84 s against a published 2.83). Patches are Freedoom's GENMIDI
(BSD-3-Clause, app/ytpmv/data/), in the format Doom read, so a part gets the
patch for its General MIDI instrument and a drum the patch for its key. DMX,
Doom's music driver, numbers notes an octave under MIDI and the bank's offsets
assume it. The OPL2 plays through a YM3014 DAC -- coarse when loud, fine when
quiet -- and only reads two waveform bits, so patches asking for OPL3
waveforms fall back to an absolute or half sine, as they did on real cards.
app/ytpmv/sid.py -- the SID, from reSID: the envelope rates and sustain
levels, the oscillator and 23-bit noise register, the twelve-bit R-2R DAC (the
6581's is missing a termination resistor and is not linear), and both measured
cutoff curves. The 6581's floors at 220 Hz and steps back down as its register
passes 0x7f; the 8580's is near linear to 12.5 kHz. Filter settings are
register values, so the same patch sounds different on each chip, as tunes
did. Not modelled: combined waveforms, the 6581's filter distortion and its
chip-to-chip variation, and the ADSR delay bug.
The octave a part is moved by to suit a speaking voice is not applied when a chip synthesises it, since a chip has no trouble with 49 Hz.
The last entry in the page's voice dropdown, none: synthesised, leaves the
corpus out entirely. It belongs to that page alone -- choosing it posts
nothing, so the sentence generator and the bot keep whatever voice they had.
Every part is played by the chip (off becomes the Mega Drive, and a -voice
setting becomes its machine playing), no part is muted for want of a word,
and the word controls are hidden. Instruments can still be chosen per part,
from any machine's synthesised ones.
Each tile is then an oscilloscope of its part (app/ytpmv/scope.py): a neon
red trace on black, triggered on a rising zero crossing so a held note stands
still, a few cycles of the part's middle note wide. It is drawn in the
console's own pixels and put through its palette like any other picture, with
colours picked to land on a real red in each -- except the Game Boy, which has
none, and draws in its lightest shade. Over the API it is options.synth.
Every part is measured and moved to where its job says it should sit, rather
than trusting a table that cannot tell a whisper from a brass patch. On by
default; --no-balance, balance=off, or the checkbox.
The measurement is the broadcast one (ITU-R BS.1770), and the part that earns its keep is the gating: a cymbal might be four hits in three minutes, and averaged across the song it looks nearly silent. Anything balancing on plain RMS would crank it until those four hits took the roof off. Gated, the same cymbal played forty times and four times measures within about a decibel.
It nudges rather than levels. Parts inside one role are not meant to be
equally loud -- several decibels between two things both called "rhythm" is
usually somebody's arrangement -- so anything within DEADBAND of its target
is left alone and anything past that is brought back by STRENGTH of the
excess. Flattening them outright was tried, measured tidier, and sounded
worse. The targets in app/ytpmv/master.py are themselves measured, not
guessed: render songs the old way, see what each role came to, take the
median.
app/ytpmv/screen.py. Each machine's resolution, and its colours and no
others.
The Mega Drive's eight levels per channel are not evenly spaced: measured off hardware they run 0, 52, 87, 116, 144, 172, 206, 255. The NES has sixty-four colours burnt in that it could neither mix nor change; there is no single correct palette for it, since the chip emits composite video and the television decodes it, so this uses FCEUX's. The Master System is two bits a channel, taken as even quarters rather than measured. The SNES is five bits a channel on a straight ramp. Console renders are encoded harder than normal, because at the usual setting the codec smears the palette the filter just restricted.
A bot account that makes videos when asked in a Matrix room:
!ytp say nice chocolate cake
!ytp mv the last MIDI file posted in the room
!ytp mv info what the song is and which words it would use
!ytp mv lead=yeah kick=boom max=60 octave:bass=+1 mode:lead=tape
!ytp voices
!ytp help
Post a .mid file, then !ytp mv (or reply to the file with it). In a direct
message the prefix is optional and plain text is said.
It runs as a second compose service against the same image, talking to the
web app over HTTP — so it uses the same queue, and a bot that falls over cannot
take the site with it. Put its settings in a .env beside docker-compose.yml:
MATRIX_HOMESERVER=https://matrix.example.org
MATRIX_USER=@ytp:example.org
MATRIX_PASSWORD=... # used once; the token is saved to the volume
MATRIX_ALLOWED_USERS=*:example.org # or MATRIX_ALLOWED_ROOMS=!abc:example.org
MATRIX_ADMINS=@you:example.org # who may switch the voice for everyone
YTP_PUBLIC_URL=https://ytp.example.org # optional: link to videos too big to uploaddocker compose --profile matrix up -d
docker compose run --rm bot --check-configInvite the bot to a room and it joins. Without either allow-list it answers
anyone who invites it (and logs a warning saying so). Each person gets one
request at a time and a cooldown (MATRIX_COOLDOWN, 20 s); song videos are
capped at MATRIX_MV_MAX_SECONDS (120). Encrypted rooms are not supported —
the bot leaves them rather than sit somewhere it cannot read.
http://localhost:8765/editor.html
The checker below finds words that are labelled wrong. This finds the other half: words whose timings are wrong, which is most of what makes a generation sound bad. An aligner puts an edge where it stops being confident, not where the sound starts and stops — that is why "mrs rosen" lost its N and why "time" once took its T from the word after it.
Pick a source, click a word. You get the video seeked to it, a waveform of the three seconds around it with its edges drawn on, and the neighbouring words' edges in grey. Click the waveform to move the start, shift-click to move the end, or nudge by 10ms and 50ms. Space plays just that clip, so you can hear the edit before saving it.
The optional spectrogram shows measured audio frequencies, using a bounded
STFT calculation when opened. Corpus and source changes discard stale editor
responses. Generation history records its voice, so clip actions apply to the
corpus that produced the result. API clients can send X-Corpus: <slug> to pin
their request; a write for a corpus that is no longer selected returns 409.
Words are flagged when their shape has caused trouble before — very short, very long, or overlapping the next word. On one ASMR source, 335 of 1,011 clips overlap their neighbour.
You can also relabel a word Whisper misheard, delete a clip that is unusable, or add one it skipped entirely. Every edit drops the generator's cache, so the next sentence uses the new timings. Nothing touches the videos — only the corpus database, which is what travels in the bundle.
Whisper's mistakes are invisible from inside. A word it mishears is stored confidently under the wrong spelling, so the corpus looks complete and simply never produces that word — "rupees" was transcribed as "rubies", the most quotable word in that corpus, unreachable by typing it.
python scripts/verify_corpus.py --all
python scripts/verify_corpus.py --source-id 3 --show 40This fetches YouTube's own captions and matches them to the stored labels by timestamp, reporting missed words (caption text with no clip near it, so speech the transcription skipped) and disagreements (both systems heard something at the same moment and spelled it differently).
Captions are not ground truth. Uploader-written subtitles would be, but most
channels do not publish any; what is available is Google's speech recognition,
with its own failures. The value is that it is a different system from
Whisper — the two agreeing means something, and the two disagreeing is worth a
person deciding. Nothing is edited: to act on a finding, put a transcript you
trust in transcripts/<video_id>.txt and run correct.py.
Repeated disagreements matter far more than one-offs. A word both systems disagree about every time it is said is a systematic mishearing; a single one is usually the two of them splitting a phrase differently.
Any channel invents words a dictionary has never held — names, in-jokes, brand names, coinages — and a word with no pronunciation cannot be spliced at all. There are two places to teach it one:
$MRS_DATA_DIR/pronunciations.csv every corpus on this machine
corpora/<name>/pronunciations.csv just that one, and packs with its bundle
Use the global file. Most of what needs teaching is not specific to a speaker — "usb", "wii" and "kilometres" are the same words whoever says them — and a per-corpus copy would have to be maintained again for every channel ingested. The per-corpus file is for a speaker's own coinages; it overrides the global one. Packing includes both files' rules in the bundle's dictionary, so a corpus arrives with its pronunciations on a machine that has never heard of them.
Two columns, and the second can be written whichever way suits:
nug,N AH G # ARPAbet, if you know it
wii,wee # or just a word that already sounds right
mcnug,mick nug # several words are fine
usb,=letters # read it out letter by letter
hevexum,=skip # leave it unsayable on purposeThe second form is the one to reach for. Nobody should have to learn ARPAbet to say that "wii" rhymes with "wee".
To find out what is worth adding:
python scripts/verify_corpus.py --unspliceable --write-templateThat lists every stored word with no pronunciation, worst first, and writes them
into the global CSV commented out, ready to fill in (--per-corpus to keep them
with one corpus instead). It needs no network, and edits take effect on the next
POST /api/reload.
Entries here beat everything built in, and a line that cannot be parsed is reported rather than ignored — silence would leave the word unsayable with nothing to explain why.
A corpus can only say what it has heard. When a word isn't there, the splicer builds it from phonemes taken out of other words — and when even those aren't there, what happens next is a per-corpus setting, stored in the corpus database so it travels inside the bundle:
| mode | behaviour |
|---|---|
strict |
real clips and clean splices only; anything else is reported missing (default) |
loose |
substitutes a near-enough phoneme, and guesses a pronunciation for words the dictionary doesn't have |
desperate |
as loose, and drops sounds nothing can cover — always produces something |
Set it from the effort dropdown next to the voice selector, or:
curl -X POST localhost:8765/api/splice-mode -H 'Content-Type: application/json' -d '{"mode":"loose"}'The modes are additive: substitutions and dropped phonemes are priced far above
any achievable saving, so wherever strict finds a splice, all three modes give
the identical result. Turning it up only changes words that would otherwise come
back missing.
A word that needed a substitution, a dropped sound, or a guessed pronunciation
is marked approx in the API and shown with a dashed border, so an approximation
is never passed off as a faithful splice.
Which you want depends on the corpus. With 7,000 distinct words, strict rarely
gives up on anything and the other modes barely fire. With 30 words, strict
can say almost nothing, and desperate is the difference between a corpus that
works and one that only quotes itself.
Drop a bundle in packs/ and restart — the entrypoint installs anything it
finds there. Or do it directly:
python scripts/corpus.py install my-corpus.tar.zst --name my-corpus
curl -X POST localhost:8765/api/reloadMRS_CORPUS pins which one is served. Without it the first alphabetically
wins, so adding a corpus can silently switch voices.
To add to a corpus that's already running, the container carries the ingest dependencies too:
docker compose run --rm app python scripts/ingest.py <youtube-url>
docker compose run --rm app python scripts/refine_boundaries.py --all --apply
curl -X POST localhost:8765/api/reloadGeneration is queued, not done in the request: a long sentence takes minutes and any proxy in front will 504 first. Submitting returns a job id; poll it.
id=$(curl -s -X POST localhost:8765/api/generate -H 'Content-Type: application/json' -d '{"text":"nice chocolate cake"}' | jq -r .id)
curl -s localhost:8765/api/jobs/$id | jq?wait=1 blocks instead, which is easier from a script.
One job runs at a time. Generation is ffmpeg-bound on a shared box, and two at once exhausted the container's thread ceiling before the queue existed.
| Endpoint | Purpose |
|---|---|
POST /api/generate |
{"text": "..."} → 202 with a job id. ?wait=1 to block |
GET /api/jobs/{id} |
status, stage, progress, result |
GET /api/queue |
queued and running counts |
GET /api/words |
vocabulary with clip counts; the frontend fetches it once |
GET /api/suggest |
autocomplete from real spoken runs |
POST /api/rate |
vote a clip up or down for that word |
GET/POST /api/splice-mode |
read or set how hard the splicer tries |
POST /api/reload |
drop the clip cache after an ingest or correction |
GET /api/stats |
corpus totals |
POST /api/ytpmv/midi |
upload a MIDI file (multipart midi) → key, tempo, parts, suggested sounds |
GET /api/ytpmv/midi/{id}/part/{part_id}?corpus={slug} |
suggested sounds for one part from an installed corpus |
POST /api/ytpmv/sample |
{"text","take","corpus"} → the clip as an instrument, and the note it is on; corpus is optional |
POST /api/ytpmv/preview |
{"midi_id","part"} → a few seconds of one part, audio only |
GET /api/ytpmv/library |
uploaded MIDIs with searchable song metadata |
PATCH /api/ytpmv/midi/{id} |
{"title"} → rename a MIDI and invalidate its cached metadata |
DELETE /api/ytpmv/midi/{id} |
remove the MIDI and sidecars; reject deletion while a queued/running job uses it |
POST /api/ytpmv/preview-mix |
{"midi_id","parts","options","start","seconds","part_id","note","notes"} → queued audio; null seconds plays to song end, part_id isolates a short pattern, note auditions one pitch, notes prepares a bank of 1–24 distinct pitches with offsets in the result |
POST /api/ytpmv/render |
{"midi_id","parts","options"} → 202 with a job id, polled like the rest |
python tests/test_tokenize.py # tokeniser: expansions, markers, boundaries
python tests/test_splice_modes.py # pronunciations, substitutions, cost order
python tests/test_corpus_select.py # which corpus a run writes into
python tests/test_bundle.py # a bundle survives a round trip
python tests/test_corpus_export.py # live WAL snapshots and portable pronunciation rules
python tests/test_corpus_metadata.py # corrections and edits preserve unrelated clip data
python tests/test_numeric_alignment.py # numbers and ordinals keep their spoken meaning
python tests/test_source_formats.py # original and rotated videos concatenate correctly
python tests/test_clip_cache.py # voice/source/window identity and actual cached audio
python tests/test_pitch.py # YTPMV: a wobbling vowel lands on the note
python tests/test_music.py # YTPMV: tempo, key and parts from a MIDI file
python tests/test_matrix_commands.py # what the Matrix bot hears in a messageThese run in CI and need nothing but num2words, nltk, zstandard,
numpy and mido.
python tests/test_end_to_end.py # ~30s: one real video, start to finishThat one is not in CI — it downloads a 63-second video, transcribes it, refines the boundaries and generates sentences from it, so it needs the network, ffmpeg, torch and a Whisper model. Run it before shipping anything that touches ingest, clip selection or the ffmpeg builder. It is the only test that exercises those together, and the bugs that have escaped so far escaped between the parts rather than inside any one of them.
main.py FastAPI app; API, frontend, output/
app/generate.py sentence -> clips -> ffmpeg
app/phonemes.py CMU pronunciations, phoneme-level splice planning
app/forced_align.py torchaudio CTC alignment for sub-word timings
app/database.py SQLite schema, corpus selection, portable paths
app/device.py CPU or CUDA, chosen once for every model
scripts/build_corpus.py channel link -> packed corpus, one command
scripts/ingest.py download, transcribe, store word clips
scripts/ingest_channel.py the same, for a whole channel
scripts/realign.py better word timestamps via stable-ts
scripts/second_opinion.py checks Whisper's words against Qwen3-ASR; fixes or marks disagreements
scripts/refine_boundaries.py word boundaries from CTC alignment and the audio; marks noise-words
scripts/measure_tone.py each take's tone (pitch, loudness, pace...), grouped into categories
scripts/correct.py relabel misheard words from a real transcript
scripts/find_noises.py pull non-verbal noises out of the gaps
app/editor.py the corpus editor's API (the only writer
outside an ingest)
frontend/editor.html fix words and timings by hand, with a waveform
app/ytpmv/ MIDI in, pitch-perfect grid video out
scripts/ytpmv.py the same from a terminal, with a tuning check
app/matrix_bot.py the Matrix bot (python -m app.matrix_bot)
frontend/ytpmv.html the YTPMV page
scripts/verify_corpus.py check stored labels against YouTube captions
scripts/corpus.py pack / unpack / migrate / inspect
corpora/<name>/ one voice: corpus.db + downloads/ + transcripts/
packs/ drop bundles here to install on next start
Older installs kept the database loose in the data directory with downloads/
beside it. Still read, but it can't sit next to another corpus — only one
downloads/ to go round.
The container migrates itself on start, so a redeploy is enough. By hand:
python scripts/corpus.py migrate # report only
python scripts/corpus.py migrate --apply # -> corpora/michael-rosen/It moves rather than copies, and is a no-op once done. If you pinned
MRS_CORPUS=default, change it to the migrated name.
The shipped corpus comes from publicly posted recordings of Michael Rosen reading his own children's poetry. Nothing in it is adult or offensive. By default nothing is synthesised either: a word with no clips and no viable splice is reported missing rather than approximated, and the modes above have to be turned up deliberately, per corpus.
It's a toy for making silly videos out of poetry readings. Keep it there, and don't use it to put words in anyone's mouth in a way that misrepresents them.