Streaming watchdog that detects LLM repetition collapse early and stops the generator before it burns more tokens.
Modern decoders fail in a particular, expensive way: a model that has lost the plot will keep emitting tokens at full speed. It can stay locked in a short n-gram loop ("the model thinks that the model thinks that..."), spam a single high-probability token ("yes yes yes yes..."), or replay a whole paragraph verbatim. Each of those failure modes happily eats your remaining max_tokens budget. Stop sequences don't help, because the model never produces the stop string. Server-side repetition penalties soften the symptom but only after billions of GPU-cycles, and most hosted APIs don't expose them. By the time a human notices, the bill is already paid.
LoopGuard watches the token stream from the client side, recognises the three common collapse patterns within tens of tokens, and tells the caller to abort. No model changes, no extra GPU, no API.
LoopGuard is an online detector. You feed it tokens one at a time as they stream in. It returns a GuardEvent after each token. When triggered is true, you cut the stream.
It runs three independent checks against a sliding window of the last 96 tokens:
- Tail n-gram repetition. The trailing n-gram (for n in 3, 5, 8) appears at least four times in the window. Catches short looping phrases.
- Window diversity collapse. The fraction of distinct tokens in a sufficiently large window drops below 0.28. Catches single-token spam.
- Long verbatim suffix match. The trailing 16-token chunk also appears earlier in the window verbatim. Catches paragraph-scale loops that slip past the small n-gram check.
The defaults are tuned so natural prose and short, intentional repetitions ("note, note, note") do not trigger.
Most repetition mitigations live inside the model: frequency penalties, no-repeat-ngram, top-p tweaks. Those require server access and still let degenerate outputs run to budget when they fire late. LoopGuard is the opposite: a small, explainable client-side guard that any application using a streaming LLM API can drop in. It does not need logits, only tokens. It also reports a structured reason and detail string, which is useful telemetry: you can log which collapse mode each session hit and tune your sampling parameters from real traffic instead of guessing.
The three-signal design is the interesting part. A pure n-gram counter misses paragraph-scale loops. A pure diversity check misses loops that have a few unique tokens cycling. A pure suffix matcher fires too late on tight short loops. Combining all three keeps the false-positive rate at zero on natural prose while detecting all three collapse modes within tens of tokens.
Open the live FigJam board: https://www.figma.com/board/0yfGsJiYTGNQrbMnYZq7rG
The diagram shows the streaming path: each token enters LoopGuard.feed, is appended to the buffer, the last window is sliced, three checks run in parallel, and the first to fire produces a GuardEvent with a reason. If nothing fires, the caller resumes normal streaming.
graph LR
A["streaming token"] --> B["append to buffer"]
B --> C["slice window"]
C --> D["n-gram tail check"]
C --> E["diversity check"]
C --> F["long suffix check"]
D --> G{"triggered?"}
E --> G
F --> G
G -->|"yes"| H["abort generation"]
G -->|"no"| I["continue"]
The metric is tokens emitted before the generator stops, with a per-scenario budget of 800 tokens, averaged over 40 random seeds. "Without LoopGuard" is the baseline: every scenario runs to budget. "With LoopGuard" stops as soon as collapse is detected.
Numbers from the most recent benchmark run:
| Scenario | Baseline | With LoopGuard | Trigger rate |
|---|---|---|---|
| normal text | 800 | 800 | 0.0% |
| mild repetition | 800 | 800 | 0.0% |
| severe n-gram loop | 800 | 44 | 100.0% |
| single-token noise | 800 | 21 | 100.0% |
| paragraph repeat | 800 | 40 | 100.0% |
Across the three pathological scenarios, the baseline emits 2400 tokens per seed and LoopGuard emits 105 tokens per seed: 95.6 percent of the wasted tokens averted. False-positive rate on benign scenarios is 0 percent: neither the normal-prose stream nor the mild-repetition stream triggers a stop in any of the 40 seeds.
The numbers in the chart are produced by examples/benchmark.py from real runs, not hand-picked.
python -m venv .venv
.venv\Scripts\activate # on Windows
# source .venv/bin/activate # on macOS or Linux
pip install -r requirements.txt
# Sanity-check the detector on five synthetic scenarios.
python examples/demo.py
# Reproduce the chart in docs/before_after.png.
python examples/benchmark.py
# Run the unit tests.
pytestTo use it in your own streaming client:
from loop_guard import LoopGuard
guard = LoopGuard()
for token in client.stream(prompt):
print(token, end="")
event = guard.feed(token)
if event.triggered:
print(f"\n[loop-guard stopped: {event.reason} :: {event.detail}]")
client.cancel()
breakfeed accepts whatever your tokenizer hands you (string pieces, token ids, BPE fragments). It only checks equality, so any hashable token type works.
python examples/demo.py on the canonical seed prints:
LoopGuard demo. Budget per scenario: 600 tokens.
normal_text COMPLETED full 600 tokens (no trigger)
mild_repetition COMPLETED full 600 tokens (no trigger)
severe_loop STOPPED at token 44 reason=ngram_repeat_n3 detail=tail 3-gram repeated 4 times in window
token_noise STOPPED at token 21 reason=ngram_repeat_n3 detail=tail 3-gram repeated 4 times in window
drifted_repeat STOPPED at token 40 reason=long_suffix_match detail=16-token suffix matches an earlier window slice
The reason field is a stable enum-like string (ngram_repeat_n3, ngram_repeat_n5, ngram_repeat_n8, low_diversity, long_suffix_match) suitable for grouping in metrics. The detail field is human-readable and intended for logs.
GuardConfig exposes every threshold:
from loop_guard import GuardConfig, LoopGuard
cfg = GuardConfig(
ngram_sizes=(3, 5, 8), # which tail n-grams to count
max_repeats=4, # how many copies in the window to fire
window=96, # ring-buffer size in tokens
min_unique_ratio=0.28, # diversity floor
diversity_min_window=40, # minimum tokens before diversity check arms
min_long_match=16, # suffix length for the long-match check
min_tokens_before_check=12,
)
guard = LoopGuard(config=cfg)Looser thresholds make the guard more permissive (lower false-positive rate, slower detection). Tighter thresholds detect collapse sooner at the cost of occasional false stops on intentionally repetitive output (lists, ASCII art).
MIT.
