A re-runnable measurement of AgentDraft's conflict engine under multi-agent contention. It produces the evidence behind the claim that independent AI scheduling agents can fight over one calendar and still resolve to exactly one winner.
- Conflict resolution accuracy: fraction of rounds where the slot ends held by the highest-priority agent that raced for it.
- Highest-priority win rate: fraction of rounds the rank-1 agent wins.
- Latency distribution: p50 and p99 across all attempts.
- Failure modes: rounds with no winner and outright errors — each should
be 0. Several
201s in one round are a legitimate bump chain, not a double-booking.
uvx --from git+https://github.com/ryabinski-labs/agentdraft-benchmark \
agentdraft-benchmark --helpor, from a checkout:
pip install -e .
agentdraft-benchmark --helpdocker compose up -d dynamodb # in the AgentDraft application repo
uvicorn app.main:app --port 8080
agentdraft-benchmark --rounds 100 --agents 5 --base-url http://127.0.0.1:8080A local dev API returns a dev_token from /v1/auth/magic-link, so the harness
provisions ranked agents itself.
Production never returns a dev_token. Issue one agent per rank in your
AgentDraft dashboard first, then pass them in — either a file:
# agent-keys.txt
sales-bot:avs_live_…:1
recruit-bot:avs_live_…:2
focus-blocker:avs_live_…:3
exec-ea:avs_live_…:4
ops-bot:avs_live_…:5
agentdraft-benchmark \
--rounds 1000 --agents 5 \
--base-url https://api.agentdraft.io \
--keys-file ./agent-keys.txt \
--label agentdraft-prodor the environment:
AGENTDRAFT_AGENT_KEYS="sales-bot:avs_live_…:1,recruit-bot:avs_live_…:2" \
AGENTDRAFT_BASE_URL=https://api.agentdraft.io \
agentdraft-benchmark --rounds 1000 --label agentdraft-prodWithout keys, a production run exits with code 2 and prints what to set.
The owner's user_id is resolved with GET /v1/agents/me using the first key.
Reports land in ./out/ (or --out-dir):
benchmark-YYYY-MM-DD.json— full per-round results and aggregates.benchmark-YYYY-MM-DD.md— human-readable report.
Every report records the stack it ran against. A local run is labelled
local-reference-stack; its latency is an order of magnitude, not a service
level. Correctness numbers travel between machines; timings do not.
The committed baseline in out/benchmark-2026-07-27.* is the run quoted on
https://agentdraft.io/benchmark.
--rounds N number of race rounds (default 100)
--agents N agents per round (default 5)
--base-url URL AgentDraft API base URL (default $AGENTDRAFT_BASE_URL)
--keys-file F pre-issued agent keys (or $AGENTDRAFT_AGENT_KEYS)
--out-dir DIR output directory (default ./out)
--label STR label for the stack tested (default 'default-stack')
--label should describe the agent stack being measured, not AgentDraft
itself. Examples: agentdraft-prod, cal-com-handler,
openai-assistants, lindy-demo, custom-claude-tooluse.
Across N rounds, 5 agents — each with a distinct ranked priority (1 = highest)
— fire POST /v1/bookings concurrently at the same 30-minute slot. Every round
uses a fresh future slot so no engine state leaks across rounds. Results come
from the engine's response (201 COMMITTED or 409 OUTRANKED), and the slot is
then read back through GET /v1/availability to confirm it came out of the race
occupied. A round is correct when the slot ends up held by the highest-priority
agent that raced for it.
MIT. See LICENSE.