Skip to content

About

Flight recorder for LLM agents: trace every LLM call, tool call and decision with tokens, cost and latency - replay any run as an interactive timeline

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

20 Commits

Folders and files

Repository files navigation

AgentTrace 🛰️

CI

A flight recorder for LLM agents. Wrap any agent loop with AgentTrace and every LLM call, tool call and decision is recorded — tokens, cost, latency, inputs and outputs — then replayed as a timeline you can actually debug.

When agents misbehave in production, the daily reality is staring at traces: why did it loop? why did it pick the wrong tool? where did 50k tokens go? AgentTrace is that debugging surface, small enough to read in one sitting.

Together with InterviewOps (evals) and TokenMeter (cost), this completes the three pillars of production LLM engineering: evals → cost → observability.

The tracing API

from app.tracer import AgentTracer

tracer = AgentTracer("research: quantum computing")

with tracer.step("llm_call", "plan", model="claude-sonnet-5") as s:
    text, t_in, t_out = call_the_model(prompt)
    s.log(input=prompt, output=text, tokens_in=t_in, tokens_out=t_out, cost=0.0008)

with tracer.step("tool_call", "web_search") as s:
    results = search(query)
    s.log(input=query, output=results)

tracer.finish()  # trace is now replayable in the viewer
  • Step types: llm_call · tool_call · thinking · decision · final
  • A step that raises is recorded as an error (with the exception message) and the exception still propagates — tracing never changes agent behavior
  • Payloads are stored truncated (4k chars) — traces are for debugging, not archival
  • Everything lands in SQLite; the viewer renders it live

What a trace looks like

research: How does RAG work and when does it fail?     completed · 6 steps · 405 tok · $0.0041 · 5.2s
  1. [llm_call ] plan searches       1150ms   74 tok   claude-sonnet-5
  2. [decision ] select tools           3ms
  3. [tool_call] web_search           340ms
  4. [tool_call] web_search           340ms
  5. [llm_call ] synthesize answer   3400ms  331 tok   claude-opus-4-8
  6. [final    ] deliver answer         1ms

The web viewer shows this as an interactive timeline: color-coded step types, latency bars, expandable inputs/outputs, error highlighting, and per-trace cost/token totals.

Run locally

cd backend
python -m venv .venv && .venv\Scripts\activate   # Windows
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000

Open http://localhost:8000, type a question, and watch the bundled research agent get traced live (plan → tool selection → searches → synthesis).

Two modes: free demo vs real API

Demo mode (default) Real mode
Trigger ANTHROPIC_API_KEY not set key set in backend/.env
Agent's LLM calls Scripted episodes Real Claude calls (Sonnet plans, Opus synthesizes)
Tracing, timing, cost math Real — identical pipeline Real
Cost ₹0 Fractions of a cent per run

API

Endpoint What it does
POST /api/run {question} → runs the traced demo agent, returns trace_id
GET /api/traces Public demo traces; ?project=<id> + JWT for your own project
GET /api/traces/{id} Full trace with ordered steps and payloads (JWT required unless demo)
POST /api/ingest Batched spans from any app; Authorization: Bearer at_... required
POST /api/auth/signup · login {email, password} → {token} (bcrypt + 7-day JWT)
GET/POST /api/projects List / create projects (JWT); PATCH renames or sets retention_days
GET/POST/DELETE /api/projects/{id}/keys List / mint / revoke that project's ingest keys
POST /api/keys Mint a key into the public demo project (pre-account clients)
GET /api/health {status, mock}
POST /api/reset Clear the public demo traces (accounts untouched)

Trace your app — the agenttrace SDK

The sdk/ package ships spans to this server over HTTP — buffered locally, posted asynchronously, and fail-open (a down collector never breaks your agent):

pip install -e sdk
curl -X POST http://localhost:8000/api/keys        # → {"key": "at_..."}
export AGENTTRACE_API_KEY=at_...                   # AGENTTRACE_ENDPOINT defaults to localhost:8000
from agenttrace import AgentTracer, traced, traced_client
import anthropic

tracer = AgentTracer("my agent run")
client = traced_client(anthropic.Anthropic())   # every messages.create logged: model, tokens, cost

@traced("tool_call")                            # tool functions become steps
def web_search(query): ...

client.messages.create(model="claude-opus-5", max_tokens=1024,
                       messages=[{"role": "user", "content": "hi"}])
web_search("agent observability")
tracer.finish()                                 # trace is now replayable in the viewer

See sdk/examples/quickstart.py for a runnable 10-line script.

Tests

38 tests cover the tracer contract (ordering, error capture, truncation, timing), the API, the authenticated ingest path, accounts (signup/login, project isolation, key revocation, cross-project trace hijack, retention pruning), and the SDK (buffering, fail-open, decorator, auto-instrumentation, pricing). CI runs on every push:

pip install -r backend/requirements-dev.txt
cd backend && python -m pytest tests -q
cd ../sdk && python -m pytest tests -q

Deploy (Render, free tier)

render.yaml blueprint included: render.com → New → Blueprint → connect this repo → deploy. Runs in demo mode (zero API cost) until you add ANTHROPIC_API_KEY.

Storage is SQLite by default (ephemeral on Render's free tier — fine for the demo). For durable accounts data, create a free Neon Postgres and set DATABASE_URL in the Render dashboard; Alembic migrates the schema on startup.

Roadmap: demo → agent-debugging platform

AgentTrace is evolving from a single-app demo into a hosted observability platform any agent can send traces to. Phases ship in order; each is usable on its own.

Phase 1 — Core tracer & viewer ✅ (shipped)

  • AgentTracer context-manager API (5 step types, error capture, payload truncation)
  • SQLite persistence with per-trace aggregates (steps, tokens, cost, latency, errors)
  • Traced demo research agent (mock + real Claude modes)
  • Interactive timeline viewer (latency bars, expandable payloads, error highlighting)
  • 11-test suite + GitHub Actions CI, Render deploy blueprint

Phase 2 — Trace any app, not just the demo ✅ (shipped)

The jump from "demo" to "tool people can adopt": tracing is now a client library that ships spans to this server over HTTP.

  • Ingest API: POST /api/ingest accepting batched spans, authenticated with at_... ingest keys (minted via POST /api/keys, stored hashed; out-of-order batches can't regress a finished trace)
  • Pip-installable SDK (sdk/, zero runtime deps): same AgentTracer API, but spans buffer locally and POST asynchronously — fail-open (a down collector warns once and never breaks the user's agent)
  • Anthropic SDK auto-instrumentation: wrap a client once (traced_client(anthropic.Anthropic())) and every messages.create call is logged with model, tokens and cost automatically
  • @traced("tool_call") decorator for tool functions (works bare or with a step type)

Phase 3 — Accounts & projects (multi-user) ✅ (shipped)

  • Email + password auth (bcrypt + JWT) with login/signup pages in the existing design system
  • Projects: traces group under projects; each project mints its own ingest keys (label, revoke, shown-once reveal)
  • SQLAlchemy + Alembic; SQLite locally / Postgres (Neon free tier) via DATABASE_URL — a pre-Alembic database is stamped and upgraded in place, existing traces/keys land in a public Legacy / Demo project so nothing breaks
  • Retention policy: auto-prune traces older than N days per project (enforced on ingest and listing)
    • Done when: two users see only their own projects; a revoked key stops ingesting — both are tests now

Phase 4 — Debugging superpowers (the problem-solving layer)

The features that turn a trace store into a debugging product:

  • Nested spans: sub-agent trees rendered as collapsible branches, not flat timelines
  • Trace diff: run the same task twice, see steps/costs/outputs side by side — the fastest way to answer "what changed?"
  • Search & filter: by status, tool name, model, min cost, min latency, text-in-payload
  • Budget alerts: per-project thresholds (tokens / cost / latency / error count) — breaching runs get flagged in the list and surfaced on a dedicated Alerts view
  • Failure patterns: group error traces by exception type + failing step to spot repeat offenders
  • "Explain this trace": one-click LLM summary of what the agent did and where it went wrong (BYOK — uses the viewer's own Anthropic key)

Phase 5 — Production & ecosystem

  • Live tailing: watch a running agent's steps stream into the timeline (WebSocket)
  • Public share links for individual traces (read-only, expiring)
  • OpenTelemetry export so traces flow into Grafana/Datadog alongside app telemetry
  • Rate limiting per ingest key, structured logging
  • Docs site: 5-minute integration guides (raw API, SDK, auto-instrumentation)

About

Flight recorder for LLM agents: trace every LLM call, tool call and decision with tokens, cost and latency - replay any run as an interactive timeline

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages