Measures one number: end of the caller's speech → first byte of audio out, broken down by hop, against the 850 ms budget.
Nothing here is an agent. There is no intelligence on the critical path by design — the point is to find out whether the transport and the providers can reach the budget before anything is built on top of them. Two days of work here either de-risks the whole plan or tells you the stack choices are wrong, which is the cheapest bad news you will ever buy.
python3 -m spike.bench --mock -n 7Synthetic providers with realistic lognormal latency. If these numbers look wrong, the bug is in the harness, not in a vendor — which is a distinction worth being able to make in ten seconds.
Then see what the whole argument is about:
python3 -m spike.bench --mock --semantic-vad -n 7Same pipeline, fused endpointer instead of a fixed silence timeout. That single swap is the difference between missing budget by 200 ms and clearing it by 100.
cp .env.example .env # fill in the keys
pip install -r requirements.txt
# drop a 16-bit mono 16 kHz recording at audio/caller.wav — see audio/README.md
./run.sh real 7Use a recording from the actual queue you are targeting. Narrowband PSTN audio with background noise is what the endpointer has to survive, and a clean studio read will flatter every number in the report.
The largest hop is endpointing, so spike/endpoint.py implements the fused
endpointer from the turn-taking spec: acoustic silence, an energy-decay prosody
cue, and a semantic gate reading STT partials. The semantic signal moves the
threshold rather than casting a vote, so every decision is one explainable
line rather than an opaque score.
python3 -m spike.endpoint_eval # fixed vs fused, across six cases
python3 -m spike.endpoint_eval --trace dangling # why it decided what it decided
python3 -m spike.test_endpoint # lexicon and no-false-cut gatespython3 -m spike.transcribe audio/caller.wav # once — writes caller.words.json
python3 -m spike.endpoint_eval --wav audio/caller.wav \
--words audio/caller.words.jsonGround truth for "the caller finished here" is the last speech-bearing window in the file, so any fire before it is a false cut with nothing to annotate by hand. The report ends with a mid-utterance pause list — every silence gap over 250 ms and what each endpointer did there. That is the whole game on a real clip: a single-run utterance only tests speed, never restraint.
Transcribe once, then iterate on the endpointer for free. Without a transcript the semantic gate has nothing to read and the fused endpointer runs degraded on two signals of three — the report says so, and on a pausing caller it cuts them off just like the fixed timeout does.
case fixed fused
complete +440ms +320ms
dangling CUT at 1,860ms +360ms
digits +500ms +140ms
short_answer +510ms +150ms
hinglish_dangling CUT at 2,000ms +360ms
hinglish_complete +500ms +340ms
────────────────────────────────────────────────────────
false cuts 2 0
mean latency (clean) 488m 278m
Two metrics, always reported together. A fixed timeout can always win on latency by being short, and always wins on false cuts by being long; the only interesting question is what one configuration does across all six cases at once.
Use it in the bench with --endpointer fused --context <ctx>:
| context | endpoint p50 | total p50 |
|---|---|---|
yes_no |
121 ms | 668 ms |
selection |
201 ms | 747 ms |
open |
361 ms | 908 ms |
open is above the 220 ms line and should be — the 220 in the budget table is a
blend, and an open-ended question is inherently the slow case. Optimise the mix
by making the dialogue layer declare a narrower context wherever it honestly can,
not by shortening the open threshold.
dangling_veto and when uncertain, wait. Both encode the same asymmetry:
cutting a caller off costs far more than a moment of hesitation. Everything else
in BASE_MS is a starting point measured on Indian PSTN traffic and should be
re-tuned per market.
English is SVO, so a trailing auxiliary — "it is", "I was" — is mid-clause and
dangles. Hindi is verb-final, so a trailing auxiliary — "mera order kahan hai"
— is how a sentence ends. Merging the two lexicons makes an endpointer that
waits forever on completed Hindi and cuts off unfinished Hindi, which is why
EN_DANGLERS and HI_DANGLERS are separate and why hinglish_complete is in
the case set. The matrix-language sniff counts Hindi function words only —
"order" and "delivery" are loanwords, not a language switch.
python3 -m spike.uiStandard library only — no server framework, binds to localhost, nothing leaves your machine. It opens by explaining what it measures and how to read the timelines, because the numbers are meaningless without knowing which direction is bad.
Each of the six test utterances renders as a timeline: caller speech in green, a
dashed line where they actually stopped, and a marker for each endpointer.
Firing left of the dashed line means you cut the caller off. Change a lexicon
entry in spike/endpoint.py, hit Run case set, and watch which utterances
move. Every case carries its note and a decision trace.
The lower panel runs the same view against your own recording, with mid-utterance pauses outlined — the only places a cut can happen.
This is an engineering tool, not a product UI, and it holds nothing commercial. Plan, decks, contracts, email drafts and the specifications live in the Field Kit, linked from the dashboard header.
Phase 1 — spike/bench.py. Offline. Feeds a WAV through VAD → STT → LLM →
TTS at realtime pace and records the four hops. Runs on a laptop with nothing
but API keys. Start here.
Phase 2 — spike/room_agent.py. The same measurement inside a live LiveKit
room, so the figure includes transport, framework and network. Routinely another
100–200 ms that nobody budgeted for. Only start this once phase 1 looks sane —
debugging a room and a provider simultaneously is how a two-day spike becomes a
two-week one.
pip install 'livekit-agents[deepgram,elevenlabs,openai,silero]'
python3 -m spike.room_agent dev # join the room from a browser client
python3 -m spike.summarise_room # waterfall from room-runs.jsonlhop p50 p95 budget verdict
endpoint ████████████████████ 501m 565m 220 over by 281
stt_final █████··············· 71m 85m 70 over by 1
llm_ttft ████████████████····· 234m 518m 300 ok
tts_ttfb ████████············· 118m 152m 120 ok
network █████················ 80m 80m 80 assumed
total 1,048m 1,239m 850 OVER by 198 ms
- endpoint — true end of speech → the endpointer commits. Measured from ground truth in the WAV, not from end of file. Almost always the largest hop and almost always the cheapest to fix.
- stt_final — endpoint → final transcript. We disable the recogniser's own endpointing so this measures your pipeline rather than their silence timer.
- llm_ttft — transcript → first token.
- tts_ttfb — first token → first audio byte.
- network — assumed, not measured offline. Phase 2 replaces the assumption.
Watch p95, not just p50. Callers forgive a slow median far more readily than an unpredictable one, and p95 is what a supervisor hears when they spot-check.
Exit code is 0 inside budget, 1 over, 2 if nothing completed — so this drops straight into CI as a gate once you have a number worth defending.
| Hop over budget | First thing to try |
|---|---|
| endpoint | Semantic endpointing. Fixed silence timeouts are the default in every stack and the single biggest lever available. |
| stt_final | Ask the provider to finalise on your signal instead of waiting out its own VAD. |
| llm_ttft | Cache the stable prompt prefix; shorten the system prompt; consider a smaller model for the first sentence only. |
| tts_ttfb | Stream from the first sentence rather than the full completion; confirm you are on the flash/turbo model. |
| network | Co-locate everything in one region per geography. A cross-region hop costs more than any optimisation elsewhere. |
spike/clock.py Timeline, hop definitions, drift-free realtime pacing
spike/report.py Percentiles, waterfall, verdict, JSON export
spike/audio.py WAV loading and ground-truth speech-end detection
spike/endpoint.py Fused endpointer: semantic gate, prosody, thresholds
spike/cases.py Six endpointing cases with exact ground truth
spike/endpoint_eval.py Fixed vs fused scoring
spike/test_endpoint.py Lexicon cases and the no-false-cut gate
spike/providers/ Four-method protocols; mock.py and real.py
spike/bench.py Phase 1 entry point
spike/room_agent.py Phase 2 entry point (LiveKit)
spike/summarise_room.py Waterfall from room-runs.jsonl
- Provider wire details in
spike/providers/real.py. Each class carries aVERIFY:comment with its doc URL. A wrong parameter usually shows up as a plausible-but-wrong latency rather than an error. - The LiveKit event names in
spike/room_agent.py. That API has moved more than once; the handlers degrade to an incomplete run rather than crashing, and the report names whichever mark went missing.
Three gates run on every push, on Python 3.11, 3.12 and 3.13, with no dependencies — everything gated here is standard library.
| Gate | Fails when |
|---|---|
spike.test_endpoint |
A lexicon case regresses, or the no-false-cut check trips |
spike.endpoint_eval |
Any case in the set gets cut off. Non-negotiable. |
spike.endpoint_eval --wav |
The real-recording path or the sidecar loader breaks |
The bench runs as a smoke check only. Exit 1 there means over the 850 ms budget, which is a finding to act on rather than a build to block — anything higher means the harness itself broke.
New languages and new failure cases are the most useful contributions, and both are small. See CONTRIBUTING.md — particularly the note on why each language needs its own dangler lexicon rather than being merged into an existing one.
Apache 2.0. See LICENSE. Chosen over MIT for the explicit patent grant, which protects both users and the author.