A real-world, percentile-based benchmark for LLM inference servers. Point it at any
OpenAI-compatible /v1 endpoint (vLLM, llama.cpp server, SGLang, TGI, Ollama, LM Studio, …)
and it measures how the stack actually feels across representative task categories —
prose, reasoning, math, code, JSON, file-edits, summarization, chat — not synthetic filler.
Unlike most tools that print one average tokens/sec at batch=1, BetterBench reports the distribution (1% low / median / average / 99% high) for three layers — time-to-first-token, inter-token latency, and per-run throughput — and is engineered to resolve ~1% differences through paired, drift-cancelled A/B measurement.
Every run also writes a self-contained HTML report beside its results.json — headline
tiles, charts and the full tables in one file with no network assets, so it opens offline and
travels in an email.
→ Open the example report · more screenshots in The HTML report
pip install -e . # editable install works out of the box; needs Python 3.10+, stdlib + numpy only# Benchmark one endpoint (single-stream, prefill and concurrency sweeps).
# Without --out the result lands in a versioned run directory under
# ~/.betterbench/runs/ — see [Where runs live](#where-runs-live)
betterbench run --endpoint http://192.168.12.47:8080/v1 --model Qwen3.8
# …or keep it in a specific place — an explicit --out always wins
betterbench run --endpoint http://192.168.12.47:8080/v1 --model Qwen3.8 \
--out results/radiance-27b.json
# A gateway in front of the server wants a Bearer key
betterbench run --endpoint https://gateway.internal/v1 --model qwen3-8 \
--api-key "$GATEWAY_KEY" # or export BETTERBENCH_API_KEY
# Re-render a saved result later (markdown to stdout, or a charted HTML page)
betterbench report results/radiance-27b.json
betterbench report results/radiance-27b.json --html
# Run just one phase — decode, prompt processing, or the concurrency sweep
betterbench run --endpoint http://192.168.12.47:8080/v1 --model Qwen3.8 --prefill
betterbench run --endpoint http://192.168.12.47:8080/v1 --model Qwen3.8 --decode --concurrency
# Choose how many measured passes per category (default 20), or take the 5-pass shortcut
betterbench run --endpoint http://192.168.12.47:8080/v1 --model Qwen3.8 --passes 10
betterbench run --endpoint http://192.168.12.47:8080/v1 --model Qwen3.8 --quick
# Record what the run actually was, so two results can be compared later
betterbench run --endpoint http://192.168.12.47:8080/v1 --model Qwen3.8 \
--note image=v0.9.3 --note quant=mxfp4 --note kv=fp8 --note tp=2
# Rigorous paired A/B between two endpoints (e.g. two images on ports 8080/8081)
betterbench ab --endpoint-a http://host:8080/v1 --endpoint-b http://host:8081/v1 \
--model Qwen3.8 --mde 1.0Without --out, a run writes into its own versioned directory under
~/.betterbench/runs/ ($BETTERBENCH_HOME/runs/ when that variable is set) —
nothing lands in the directory you ran it from (unless $BETTERBENCH_HOME
itself is a relative path — one-time stderr warning if so), and re-running
can never overwrite an earlier result:
betterbench run --endpoint ... --model Qwen3.8 --name mxfp4-flip
# output: ~/.betterbench/runs/20260211-091234-mxfp4-flip/
ls ~/.betterbench/runs/
# 20260211-091234-qwen3-8/ 20260211-110402-qwen3-8-2/ 20260212-080101-mxfp4-flip/ ...The directory name is a start timestamp plus a label — the model name by
default, --name mxfp4-flip to call a run what it is — so entries sort by time
and ls reads like a changelog. A second run in the same second appends -2;
collisions never clobber. The directory is created before the measuring starts
and its path printed in the opening banner, so a run knows where it is going
before it spends the hours getting there. A run directory holds that run's results.json and
the charted HTML report beside it, and ab writes its ab.json the same way,
so history accumulates instead of churning:
betterbench report ~/.betterbench/runs/20260211-091234-qwen3-8/results.json --html--out PATH always wins over the default, and BETTERBENCH_HOME moves the
base directory. (betterbench report/compare read whatever path you point
them at and write nothing unless told to.)
Two forms of betterbench compare, one keyword:
# two saved results side by side — a per-category decode table in the terminal
betterbench compare results.json another.json
# no args — an interactive session: it scans $BETTERBENCH_HOME/runs,
# opens a browser at a gallery of every saved run, and stops on Ctrl-C
betterbench compareThe no-args form serves every run directory under $BETTERBENCH_HOME/runs/
(~/.betterbench/runs/ when the variable is unset) from a loopback-only
server that stops on Ctrl-C; the URL it prints is the entry point and the
server writes no files. Pick a run to open its report; pick two to open the
pair page.
The pair page's comparison band shows the paired decode statistics — the
per-pass decode_tps series paired by pass index, truncated to the shorter
side, with a paired-t 95% CI and a SIG/noise verdict per category — plus the combined (weighted, each side's own
weights) decode, and median deltas for TTFT/ITL
(stream updates when batched), prefill per depth, and the concurrency
medians per level.
The orange banner is always on, for a reason: cross-file comparisons are
unpaired in time, so drift between the two runs — thermal, cache-
warmth, minutes or days apart — is indistinguishable from the change under
test. For a verdict: run betterbench ab. The mismatch chips next to the
banner — corpus version, sampling, host, GPU — flag which of the deltas
not to trust (a corpus-version mismatch means the prompts differed, so
every Δ in that page is apples-to-oranges); a phase missing on one side
shows as "not measured", never as zero.
A request goes out unauthorised by default — right for a local vLLM or
llama.cpp server, but wrong once a gateway sits in front of the endpoint.
--api-key sends the key as Authorization: Bearer KEY on every request,
including the /v1/models context probe, which would otherwise 401 and read
as "unknown max context":
betterbench run --endpoint https://gateway.internal/v1 --model qwen3-8 --api-key "..."
# A/B boxes behind two gateways get per-endpoint keys
betterbench ab --endpoint-a https://a.internal/v1 --endpoint-b https://b.internal/v1 \
--model qwen3.8 --api-key-a "$KEY_A" --api-key-b "$KEY_B"The key travels on the wire only: it is never written into the results, so a
results.json stays shareable however it was produced. export BETTERBENCH_API_KEY=... fills in for any missing key flag, keeping it out of
history where the flag would otherwise live.
BetterBench checks once a day whether a newer release has been tagged, and prints a line after the run if so — a benchmark a version behind can be measuring something the current release already fixed:
A newer BetterBench is available: v0.7.0 (running 0.6.1) — https://github.com/GGZ14/BetterBench/releases
The check runs on a background thread and is finished before the first
measured request, so it is never on the wire while a request to the endpoint
under test is being timed. Its answer is cached in
$BETTERBENCH_HOME/update-check.json for 24 hours, so the ordinary invocation
makes no network call at all, and every failure — offline, proxied, rate
limited — is silence rather than an error. Switch it off with
--no-update-check on any subcommand, or BETTERBENCH_NO_UPDATE_CHECK=1.
Single-stream table — per category: TTFT p50/p99, decode t/s median ± IQR, a weighted combined score, and a latency pair that depends on what the server actually streams:
- one token per update — ITL as tokens/sec (1% low = slowest tokens / median / 99% high = fastest), exactly as before;
- several tokens per update (speculative decoding — MTP, EAGLE, Medusa, n-gram,
DFlash2) —
update p50 / p99 (ms)andtok/updateinstead. Tokens that arrive together in one network write have no time between them, so BetterBench reports the gap it measured rather than inventing a per-token number. p99 is the stutter.
Reasoning / answer split — on a thinking model, TTFT is time to the first thinking
token, a wait nobody experiences. Where the split is detectable, the report adds the
reasoning share and TTFA — time to the first answer token — always with the k/N runs
it was computed over. A run cut off before </think> is recorded as unknown, not as an
answer.
Prompt-processing (prefill) sweep — prefill throughput (prompt tokens ÷ TTFT) at
increasing input depth (2K → 64K), with a cold prefix cache and tiny decode, so you see how
prompt processing scales for long-context / RAG / big-file workloads. This sweep is the only
place BetterBench reports prefill throughput: on the short prompts of the single-stream
corpus, TTFT is mostly fixed per-request overhead rather than prefill work, so a per-category
PP t/s understates the real figure by more than 2x. That column is gone.
Cold across runs too. The nonce RNG is seeded from OS entropy, so no two invocations send the same prompt, and a server that keeps its cache warm between runs (a RAM/SSD KV tier) has nothing to serve back.
Run length you control — --passes N sets the measured passes per category (default 20)
and --warmup N the discarded ones; both override whatever a --config file says. --quick
is the shorthand for a smoke run — 5 passes after 1 warmup — for checking that an endpoint,
model name and corpus are wired up before you spend an hour on the real thing. Five passes
is too thin a sample to publish or to compare stacks with; the pass count is printed in the
banner and recorded in the report so a quick result can't be mistaken for a full one.
Concurrency sweep — aggregate throughput and per-request TTFT/decode percentiles at
increasing load, revealing the throughput/latency knee. Levels 1, 2, 4 and 8 by default;
deeper is a --config file away ("concurrency_levels": [1, 2, 4, 8, 16, 32]), and
each extra level costs its own pass of concurrency_requests.
Phases you can run one at a time — --decode (single-stream, batch = 1), --prefill
(the prompt-processing depth sweep) and --concurrency (the load sweep). Naming a phase runs
only the phases named, so re-measuring prompt processing after a kernel change costs the
prefill sweep and nothing else; name several to run several. With none named all three run,
as before. --prefill reads no corpus — the sweep builds its own prompts by depth — and a
partial result records which phases it holds, so a prefill-only file can't be mistaken for a
full run.
A charted HTML report — every run also writes a standalone .html beside its
results.json (same basename), charting every phase and carrying the full tables underneath.
See The HTML report below. Skip it with --no-html, or point it somewhere
else with --html-out.
Paired A/B — Δ with a 95% confidence interval and a verdict that refuses to call a winner inside the noise band:
Decode throughput (B vs A)
- Δ = +1.34% (95% CI [+0.91%, +1.77%])
- B is 1.34% faster — SIGNIFICANT
betterbench run writes the report automatically; betterbench report results.json --html
re-renders one from a saved result at any time. It is a single file — the charts are
hand-drawn SVG, so there is no chart library, no CDN, no fonts to fetch and no build step.
Open it offline, attach it to an email, drop it in a PR.
→ Open the example report
· docs/example-report.html
The charts are the summary; the detail is one hover away — here, the per-category IQR, coefficient of variation and pass count behind a single decode bar. A high CV means that category's passes disagreed, which is exactly what you need to know before believing a small difference.
|
|
| Aggregate throughput against load, and the TTFT p50/p99 pair underneath it — where the line flattens and the p99 bar takes off is the knee. | Median prefill t/s at increasing input depth against a cold prefix cache, with the 1%-low → 99%-high band shaded. |
Charts are for reading; the tables are for quoting. Every figure that went into them is under
a Full numbers fold — single-stream per category, the concurrency sweep, the reasoning /
answer split, and the prefill sweep — with the † sample-size marks carried through, so an
under-sampled percentile is labelled wherever it appears.
No toggle to find and no setting to remember — the report renders light or dark from the
reader's own prefers-color-scheme.
The screenshots above are a
--quicksmoke run (5 passes per category), which is why the percentiles carry†marks — see Sample-size honesty below.
Raw run-to-run noise on real stacks is commonly 3–8% — enough to bury a 1% signal. BetterBench
controls for it (see docs/METHODOLOGY.md):
- Paired, interleaved A/B on the same warm box → the CI is on the difference, cancelling thermal/cache drift.
- Greedy + fixed seed (A/B default) → identical token counts, no sampling variance.
- Unique prompt bodies → honest prefill. A nonce prefix alone only defeats a prefix cache; the prefill sweep reshuffles the whole filler every pass, so a cache keyed on block content (a per-layer KV LRU) has nothing to hit either.
- Power analysis / run-to-confidence → runs until the Δ CI is tighter than your target MDE.
- Sample-size honesty → every percentile below
n · tail ≥ 5is marked†and the shortfall is recorded inresults.json. The token-rich gap series is the trustworthy tail; a per-run p99 at 20 passes is labelled for what it is. - Null-test gate → the tool proves it can't manufacture a phantom win before you trust it.
python tools/self_test.py
# checks timing accuracy against a mock SSE server (known TTFT/ITL)
# and the null A/B test (same endpoint vs itself must report "within noise")
pip install -e ".[dev]" && python -m pytest -q
# gap math under batched streams, the sample-size gate, the A/B sign convention,
# the reasoning split, phase selection end-to-end, and re-rendering a v0.2.3 results fileVersioned prompt sets live in corpus/v1/ (one JSONL per category), spanning an
input×output length grid so prefill-heavy (file-edit, summarize) and decode-heavy (story,
essay) shapes are both represented. Results are only comparable within a corpus_version.
See docs/CONTRIBUTING_PROMPTS.md to add prompts.
Apache-2.0.