Catch silent cost surges and broken agent loops before they ship.
pip install agent-trajectory-diffWebsite & Interactive Docs · Cookbooks · Live Demo Repo · Changelog
AgentDiff is a developer-first Python library, pytest plugin, and CLI for regression testing multi-turn, tool-using AI agents by comparing execution paths (trajectories) head-to-head.
When you change a prompt, tweak a system instruction, or upgrade an LLM, traditional assertions only verify that the final string matches. They miss the silent failures: the agent took 5 extra tool calls, burned 3× the tokens, entered an infinite retry loop, or drifted from the verified execution path.
AgentDiff aligns candidate execution DAGs against committed golden baselines in <10ms without calling any paid LLM judges.
- Deterministic Graph Diffing: Topological DAG alignment and Longest Common Subsequence (LCS) step comparison in
<10mswith zero paid LLM-judge calls. - Statistical Baselines & Variance Bands: Capture N-run envelopes (
record --runs 3) so non-deterministic agents don't flake CI on harmless jitter. - Zero-Config Setup (
agentdiff init): Auto-detects LangGraph, CrewAI, OpenAI Agents SDK, or OpenTelemetry and writesagentdiff.toml+ CI workflow in seconds. - In-PR Interactive Blessings (
/agentdiff approve): Reviewers bless intended trajectory improvements from PR comments asagentdiff-ci[bot]. - 100% Local & Air-Gapped: Zero telemetry, no cloud accounts, no outbound network calls during diffs. Raw prompts and tool outputs stay local.
- Drop-in CI Merge Gate: Native exit codes (
0pass /1regression fail) and automated GitHub Action PR comments with human-first verdicts. - Universal Telemetry Adapters: Seamlessly diff traces from LangGraph, CrewAI, OpenAI Agents SDK, Langfuse, LangSmith, OpenInference / OpenTelemetry, or generic JSON.
Auto-detect your agent framework and generate your configuration + CI workflow:
agentdiff init --scenario customer_support --runs 3 --with-approveRecord an N-run baseline envelope from any agent function without writing boilerplate telemetry:
agentdiff record my_agent:run \
--input '{"query": "summarize repo"}' \
--runs 3 \
--out baselines/customer_support.envelope.jsonCompare candidate runs against your baseline envelope:
agentdiff diff baselines/customer_support.envelope.json traces/candidate.json --fail-on-regressionEnforce trajectory parity directly in your test suite:
import pytest
from agentdiff import load_trace, compare
from agentdiff.testing import assert_no_regressions
def test_agent_refactor_efficiency():
# Load traces (auto-detects telemetry source format)
baseline = load_trace("tests/baselines/golden.json")
candidate = load_trace("tests/traces/candidate.json")
# Run sub-10ms deterministic comparison
report = compare(baseline, candidate)
# Assert no structural drift, cost surges, or tool loops
assert_no_regressions(
report,
max_divergence=0.25, # Max Trajectory Divergence Index [0.0 - 1.0]
max_cost_increase_pct=5.0, # Max 5% token cost increase
allow_loops=False, # Reject repetitive tool call cycles
max_wasted_effort=0.10, # Max 10% error/retry/abandoned steps
max_recovery_step_ratio=1.5 # Max recovery steps relative to baseline
)Block broken agent PRs before they land in production using the official composite action:
name: AgentDiff Regression Gate
on:
pull_request:
permissions:
contents: read
pull-requests: write # Allows posting automated root-cause PR comments
jobs:
agent-regression-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0
with:
baseline: baselines/customer_support.envelope.json
candidate: traces/pr_candidate.json
pr: ${{ github.event.pull_request.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}When a regression occurs, the gate fails with exit code 1 and comments on the PR with culprit identification and a collapsed divergence tree:
### AgentDiff Gate: REGRESSION DETECTED
| Metric | Baseline | Candidate | Threshold | Status |
| :--- | :--- | :--- | :--- | :--- |
| **Divergence (TDI)** | 0.00 | 0.42 | ≤ 0.25 | FAIL |
| **Cost Surge** | $0.0042 | $0.0138 (+228%) | ≤ +5.0% | FAIL |
| **Loops (LBI)** | 0 | 3 loops | 0 | FAIL |
| **Wasted Effort (WEI)**| 0.00 | 0.38 | ≤ 0.10 | FAIL |
**Culprit Step:** Step 4 `execute_sql` entered a 3× retry loop after schema refactor.| Metric | Target / Range | Algorithmic Definition | Description |
|---|---|---|---|
| Trajectory Divergence Index (TDI) |
0.0 (Identical) to 1.0 (Divergent) |
Structural distance between baseline and candidate execution DAGs using Longest Common Subsequence. | |
| Wasted Effort Index (WEI) |
0.0 (Optimal) to 1.0 (Total Waste) |
Fraction of execution steps spent in failed, retried, or aborted tool operations. | |
| Loop Buster Index (LBI) | Integer ( |
Stagnant State Cycle Detection | Counts repeating consecutive tool call patterns where inputs/outputs show no state progression. |
| Recovery Step Ratio (RSR) |
1.0 = Parity; |
$$\text{RSR} = \frac{\text{Recovery Steps}{\text{candidate}}}{\text{Recovery Steps}{\text{baseline}}}$$ | Measures the number of steps required to return to the verified golden trajectory path after encountering an error. |
| Resource Deltas ( |
Percentage ( |
$\frac{\text{Val}{\text{candidate}} - \text{Val}{\text{baseline}}}{\text{Val}_{\text{baseline}}} \times 100$ | Exact percentage deltas for |
| Telemetry Framework / Format | Adapter Spec | Ingestion Guide |
|---|---|---|
| LangGraph / LangChain | --adapter langgraph |
cookbooks/langgraph |
| CrewAI | --adapter crewai |
cookbooks/crewai |
| OpenAI Agents SDK | --adapter openai_agents |
cookbooks/openai_agents |
| Langfuse | --adapter langfuse |
cookbooks/langfuse |
| LangSmith | --adapter langsmith |
cookbooks/langsmith |
| OpenInference / OpenTelemetry | --adapter openinference |
cookbooks/openinference |
| Generic JSON Schema | --adapter generic |
schema/v0.1.0/trace.json |
Agent trajectories often contain proprietary prompts, sensitive tool payloads, and customer data. AgentDiff is engineered with strict local-first principles:
- Zero Outbound Network Traffic: Parsing, DAG diffing, metric calculations, and reporting run 100% locally.
- Air-Gapped & Firewall Friendly: Run tests on laptops, in air-gapped VPCs, or under strict enterprise egress policies.
- Repo-Committed Baselines: Your golden trajectories live in Git next to the code they protect.
- No Third-Party APM Lock-In: Switch tracing providers at any time; AgentDiff normalizes all schemas to a unified specification.
- Official Documentation: https://agentdiff.app
- Live Demo Repository: github.com/lostmartian/agentdiff-demo
- Engine Specification: Under the Hood
- Interactive Visualizer: Diff Playground
- Adapters Guide: Framework Integration
This repository uses uv for lightning-fast environment and dependency management.
# Clone the repository
git clone https://github.com/kerrshift/agentdiff.git
cd agentdiff
# Install dependencies and sync virtualenv
uv sync
# Run linting and code formatting checks
make lint
# Run the test suite
make test
# Build package distributions
make build
# Website & docs (separate repo, deploys agentdiff.app)
# → github.com/kerrshift/agentdiff-websiteDistributed under the MIT License. See LICENSE for more information.