Skip to content

About

Streaming watchdog that detects LLM repetition collapse early and stops the generator before it burns more tokens

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

loop-guard

Streaming watchdog that detects LLM repetition collapse early and stops the generator before it burns more tokens.

Problem

Modern decoders fail in a particular, expensive way: a model that has lost the plot will keep emitting tokens at full speed. It can stay locked in a short n-gram loop ("the model thinks that the model thinks that..."), spam a single high-probability token ("yes yes yes yes..."), or replay a whole paragraph verbatim. Each of those failure modes happily eats your remaining max_tokens budget. Stop sequences don't help, because the model never produces the stop string. Server-side repetition penalties soften the symptom but only after billions of GPU-cycles, and most hosted APIs don't expose them. By the time a human notices, the bill is already paid.

LoopGuard watches the token stream from the client side, recognises the three common collapse patterns within tens of tokens, and tells the caller to abort. No model changes, no extra GPU, no API.

What it does

LoopGuard is an online detector. You feed it tokens one at a time as they stream in. It returns a GuardEvent after each token. When triggered is true, you cut the stream.

It runs three independent checks against a sliding window of the last 96 tokens:

  1. Tail n-gram repetition. The trailing n-gram (for n in 3, 5, 8) appears at least four times in the window. Catches short looping phrases.
  2. Window diversity collapse. The fraction of distinct tokens in a sufficiently large window drops below 0.28. Catches single-token spam.
  3. Long verbatim suffix match. The trailing 16-token chunk also appears earlier in the window verbatim. Catches paragraph-scale loops that slip past the small n-gram check.

The defaults are tuned so natural prose and short, intentional repetitions ("note, note, note") do not trigger.

Why it is interesting

Most repetition mitigations live inside the model: frequency penalties, no-repeat-ngram, top-p tweaks. Those require server access and still let degenerate outputs run to budget when they fire late. LoopGuard is the opposite: a small, explainable client-side guard that any application using a streaming LLM API can drop in. It does not need logits, only tokens. It also reports a structured reason and detail string, which is useful telemetry: you can log which collapse mode each session hit and tune your sampling parameters from real traffic instead of guessing.

The three-signal design is the interesting part. A pure n-gram counter misses paragraph-scale loops. A pure diversity check misses loops that have a few unique tokens cycling. A pure suffix matcher fires too late on tight short loops. Combining all three keeps the false-positive rate at zero on natural prose while detecting all three collapse modes within tens of tokens.

Architecture

Open the live FigJam board: https://www.figma.com/board/0yfGsJiYTGNQrbMnYZq7rG

The diagram shows the streaming path: each token enters LoopGuard.feed, is appended to the buffer, the last window is sliced, three checks run in parallel, and the first to fire produces a GuardEvent with a reason. If nothing fires, the caller resumes normal streaming.

graph LR
  A["streaming token"] --> B["append to buffer"]
  B --> C["slice window"]
  C --> D["n-gram tail check"]
  C --> E["diversity check"]
  C --> F["long suffix check"]
  D --> G{"triggered?"}
  E --> G
  F --> G
  G -->|"yes"| H["abort generation"]
  G -->|"no"| I["continue"]
Loading

Results

Before vs After

The metric is tokens emitted before the generator stops, with a per-scenario budget of 800 tokens, averaged over 40 random seeds. "Without LoopGuard" is the baseline: every scenario runs to budget. "With LoopGuard" stops as soon as collapse is detected.

Numbers from the most recent benchmark run:

Scenario Baseline With LoopGuard Trigger rate
normal text 800 800 0.0%
mild repetition 800 800 0.0%
severe n-gram loop 800 44 100.0%
single-token noise 800 21 100.0%
paragraph repeat 800 40 100.0%

Across the three pathological scenarios, the baseline emits 2400 tokens per seed and LoopGuard emits 105 tokens per seed: 95.6 percent of the wasted tokens averted. False-positive rate on benign scenarios is 0 percent: neither the normal-prose stream nor the mild-repetition stream triggers a stop in any of the 40 seeds.

The numbers in the chart are produced by examples/benchmark.py from real runs, not hand-picked.

How to run it

python -m venv .venv
.venv\Scripts\activate          # on Windows
# source .venv/bin/activate     # on macOS or Linux
pip install -r requirements.txt

# Sanity-check the detector on five synthetic scenarios.
python examples/demo.py

# Reproduce the chart in docs/before_after.png.
python examples/benchmark.py

# Run the unit tests.
pytest

To use it in your own streaming client:

from loop_guard import LoopGuard

guard = LoopGuard()
for token in client.stream(prompt):
    print(token, end="")
    event = guard.feed(token)
    if event.triggered:
        print(f"\n[loop-guard stopped: {event.reason} :: {event.detail}]")
        client.cancel()
        break

feed accepts whatever your tokenizer hands you (string pieces, token ids, BPE fragments). It only checks equality, so any hashable token type works.

Example outputs

python examples/demo.py on the canonical seed prints:

LoopGuard demo. Budget per scenario: 600 tokens.

normal_text         COMPLETED full 600 tokens (no trigger)
mild_repetition     COMPLETED full 600 tokens (no trigger)
severe_loop         STOPPED at token   44  reason=ngram_repeat_n3  detail=tail 3-gram repeated 4 times in window
token_noise         STOPPED at token   21  reason=ngram_repeat_n3  detail=tail 3-gram repeated 4 times in window
drifted_repeat      STOPPED at token   40  reason=long_suffix_match  detail=16-token suffix matches an earlier window slice

The reason field is a stable enum-like string (ngram_repeat_n3, ngram_repeat_n5, ngram_repeat_n8, low_diversity, long_suffix_match) suitable for grouping in metrics. The detail field is human-readable and intended for logs.

Configuration

GuardConfig exposes every threshold:

from loop_guard import GuardConfig, LoopGuard

cfg = GuardConfig(
    ngram_sizes=(3, 5, 8),    # which tail n-grams to count
    max_repeats=4,            # how many copies in the window to fire
    window=96,                # ring-buffer size in tokens
    min_unique_ratio=0.28,    # diversity floor
    diversity_min_window=40,  # minimum tokens before diversity check arms
    min_long_match=16,        # suffix length for the long-match check
    min_tokens_before_check=12,
)
guard = LoopGuard(config=cfg)

Looser thresholds make the guard more permissive (lower false-positive rate, slower detection). Tighter thresholds detect collapse sooner at the cost of occasional false stops on intentionally repetitive output (lists, ASCII art).

License

MIT.

About

Streaming watchdog that detects LLM repetition collapse early and stops the generator before it burns more tokens

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages