Skip to content

About

Open, re-runnable harness behind AgentDraft's multi-agent collision benchmark: fire N AI scheduling agents at one calendar slot and measure conflict-resolution accuracy, priority wins, and latency.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

20 Commits

Folders and files

Repository files navigation

AgentDraft collision benchmark

A re-runnable measurement of AgentDraft's conflict engine under multi-agent contention. It produces the evidence behind the claim that independent AI scheduling agents can fight over one calendar and still resolve to exactly one winner.

  • Conflict resolution accuracy: fraction of rounds where the slot ends held by the highest-priority agent that raced for it.
  • Highest-priority win rate: fraction of rounds the rank-1 agent wins.
  • Latency distribution: p50 and p99 across all attempts.
  • Failure modes: rounds with no winner and outright errors — each should be 0. Several 201s in one round are a legitimate bump chain, not a double-booking.

Install

uvx --from git+https://github.com/ryabinski-labs/agentdraft-benchmark \
  agentdraft-benchmark --help

or, from a checkout:

pip install -e .
agentdraft-benchmark --help

Run it against a local stack

docker compose up -d dynamodb          # in the AgentDraft application repo
uvicorn app.main:app --port 8080

agentdraft-benchmark --rounds 100 --agents 5 --base-url http://127.0.0.1:8080

A local dev API returns a dev_token from /v1/auth/magic-link, so the harness provisions ranked agents itself.

Run it against a production deployment

Production never returns a dev_token. Issue one agent per rank in your AgentDraft dashboard first, then pass them in — either a file:

# agent-keys.txt
sales-bot:avs_live_…:1
recruit-bot:avs_live_…:2
focus-blocker:avs_live_…:3
exec-ea:avs_live_…:4
ops-bot:avs_live_…:5
agentdraft-benchmark \
  --rounds 1000 --agents 5 \
  --base-url https://api.agentdraft.io \
  --keys-file ./agent-keys.txt \
  --label agentdraft-prod

or the environment:

AGENTDRAFT_AGENT_KEYS="sales-bot:avs_live_…:1,recruit-bot:avs_live_…:2" \
AGENTDRAFT_BASE_URL=https://api.agentdraft.io \
  agentdraft-benchmark --rounds 1000 --label agentdraft-prod

Without keys, a production run exits with code 2 and prints what to set. The owner's user_id is resolved with GET /v1/agents/me using the first key.

Output

Reports land in ./out/ (or --out-dir):

  • benchmark-YYYY-MM-DD.json — full per-round results and aggregates.
  • benchmark-YYYY-MM-DD.md — human-readable report.

Every report records the stack it ran against. A local run is labelled local-reference-stack; its latency is an order of magnitude, not a service level. Correctness numbers travel between machines; timings do not.

The committed baseline in out/benchmark-2026-07-27.* is the run quoted on https://agentdraft.io/benchmark.

CLI

--rounds   N    number of race rounds          (default 100)
--agents   N    agents per round               (default 5)
--base-url URL  AgentDraft API base URL        (default $AGENTDRAFT_BASE_URL)
--keys-file F   pre-issued agent keys          (or $AGENTDRAFT_AGENT_KEYS)
--out-dir  DIR  output directory               (default ./out)
--label    STR  label for the stack tested     (default 'default-stack')

--label should describe the agent stack being measured, not AgentDraft itself. Examples: agentdraft-prod, cal-com-handler, openai-assistants, lindy-demo, custom-claude-tooluse.

Method, in one paragraph

Across N rounds, 5 agents — each with a distinct ranked priority (1 = highest) — fire POST /v1/bookings concurrently at the same 30-minute slot. Every round uses a fresh future slot so no engine state leaks across rounds. Results come from the engine's response (201 COMMITTED or 409 OUTRANKED), and the slot is then read back through GET /v1/availability to confirm it came out of the race occupied. A round is correct when the slot ends up held by the highest-priority agent that raced for it.

License

MIT. See LICENSE.

About

Open, re-runnable harness behind AgentDraft's multi-agent collision benchmark: fire N AI scheduling agents at one calendar slot and measure conflict-resolution accuracy, priority wins, and latency.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages