moonshine stream: cap the final decode budget at the model card's output length - #455
Merged
Merged
Conversation
michalharakal
commented
Sep 25, 2026
michalharakal
commented
Contributor
Two fixes for short utterances, where the German streaming tiny model is
documented as weakest.
Cap the final decode budget at the model card's own limit,
max_new_tokens = samples * 6.5/16000 + 2, instead of a fixed 24. A fixed
budget lets a one-second clip keep decoding long after the audio is spent,
which is when this model repeats itself ("Lauter, lauter"). The cap is
floored at 4 so a one-word command still has room and ceilinged at the
previous fixed budget, so nothing decodes longer than it did before.
End-align the last encoder window. The encoder graph takes features only
and has no attention mask, so a window that runs past the end of the audio
has its tail zero-padded and the encoder attends to that silence as if it
were speech. Pulling the start back so the window ends on the final frame
makes every frame in it real audio. Only applied when the utterance is at
least one window long and the pulled-back start does not run ahead of what
is already finalized; a clip shorter than one window still needs a real mask.
The finish trace line now reports the tokens used against the budget.
michalharakal
marked this pull request as draft
September 25, 2026 13:37
Contributor
Author
|
|
Measurement separates the two changes, and only one of them earns its place. The decode budget can only bind below about 3.4 s of audio; above that the model card's formula exceeds the old fixed ceiling and the code is a no-op. That gives a control group. On 183 German command-and-control recordings, clips below the threshold finish 0.58 s sooner (paired median, faster in 135 of 172) and clips above it are unchanged (+0.09 s, 11 recordings). The cap does exactly what it can do and nothing where it cannot act, so the win is attributable rather than incidental. Median word error rate does not move. The end-aligned window has no measured effect at all. The 11 recordings where it acts alone are byte-identical between the two builds. It also recomputes rows already finalized in e->mem, since the pulled-back start can precede e->win * HOP and the guard permits that overlap, so it carries a correctness risk for no demonstrated benefit. Removed. Per-recording transcript differences between the builds were 17 of 187, all below the cap threshold, and all of them lose the start of the utterance, which a budget applied to a greedy decode cannot cause. They are more likely harness noise than either change, and this measurement does not claim them.
michalharakal
changed the title
michalharakal
marked this pull request as ready for review
September 25, 2026 13:50
Contributor
Author
|
|
Contributor
Author
|
Timing note for whoever picks this up: #456 prepares the 0.57.0 release and was opened before this PR, so its changelog covers #452 and #453 but not this change. As things stand the cap would land in 0.58.0, not 0.57.0. This matters downstream: the Moonshine cartridge is a thin wrapper over If it should go into 0.57.0, this needs to merge before #456 and get a changelog line there. |
Records the cap, the control-group measurement it rests on, and the fact that the end-aligned window was dropped for measuring nothing. #458 keeps the real encoder-mask gap open.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.