Skip to content

docs(sync): update guides, quickstart, and README for 0.5.0 release #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lostmartian merged 1 commit into from
Sep 1, 2026
Merged

docs(sync): update guides, quickstart, and README for 0.5.0 release #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 22 additions & 83 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,70 +32,40 @@ AgentDiff aligns candidate execution DAGs against committed golden baselines in
## Highlights

- **Deterministic Graph Diffing:** Topological DAG alignment and Longest Common Subsequence (LCS) step comparison in `<10ms` with zero paid LLM-judge calls.
- **100% Local & Air-Gapped:** Zero telemetry, no cloud accounts, no network calls during diffs. Raw prompts and tool outputs never leave your machine or CI runner.
- **Drop-in CI Merge Gate:** Native exit codes (`0` pass / `1` regression fail) and automated GitHub Action PR comments with collapsed divergence trees and culprit attribution.
- **Universal Telemetry Adapters:** Seamlessly diff traces exported from **LangGraph**, **CrewAI**, **OpenAI Agents SDK**, **Langfuse**, **LangSmith**, **OpenInference / OpenTelemetry**, or generic JSON.
- **Config-as-Code & Goodhart Guard:** Commit thresholds in `agentdiff.toml` right next to your code. Flag drift when baseline gate definitions change.
- **First-Class Pytest Plugin:** Native `agentdiff_trace` fixture and `assert_no_regressions` assertion helper.

## Architecture

```mermaid
flowchart LR
subgraph Ingestion["1. Ingestion"]
A1[LangGraph / CrewAI] --> T[Normalized AgentTrace]
A2[Langfuse / LangSmith] --> T
A3[OpenInference / OTel] --> T
A4[OpenAI Agents SDK] --> T
end

subgraph DiffEngine["2. Diff Engine (<10ms)"]
T --> DAG[Topological DAG Align]
DAG --> Metrics[TDI · WEI · LBI · RSR · ΔCost]
end

subgraph Enforcement["3. Merge Gate"]
Metrics --> Gate{Thresholds Violated?}
Gate -- No --> Pass[Exit 0 · Update Baseline]
Gate -- Yes --> Fail[Exit 1 · Block PR & Post Root-Cause Comment]
end
```

## Installation

```bash
# Using pip
pip install agent-trajectory-diff
- **Statistical Baselines & Variance Bands:** Capture N-run envelopes (`record --runs 3`) so non-deterministic agents don't flake CI on harmless jitter.
- **Zero-Config Setup (`agentdiff init`):** Auto-detects LangGraph, CrewAI, OpenAI Agents SDK, or OpenTelemetry and writes `agentdiff.toml` + CI workflow in seconds.
- **In-PR Interactive Blessings (`/agentdiff approve`):** Reviewers bless intended trajectory improvements from PR comments as `agentdiff-ci[bot]`.
- **100% Local & Air-Gapped:** Zero telemetry, no cloud accounts, no outbound network calls during diffs. Raw prompts and tool outputs stay local.
- **Drop-in CI Merge Gate:** Native exit codes (`0` pass / `1` regression fail) and automated GitHub Action PR comments with human-first verdicts.
- **Universal Telemetry Adapters:** Seamlessly diff traces from **LangGraph**, **CrewAI**, **OpenAI Agents SDK**, **Langfuse**, **LangSmith**, **OpenInference / OpenTelemetry**, or generic JSON.

# Using uv
uv add agent-trajectory-diff
## Quickstart

# Global CLI tool (isolated environment)
uv tool install agent-trajectory-diff
```
### 1. Initialize with `agentdiff init`
Auto-detect your agent framework and generate your configuration + CI workflow:

Enable tab completion for bash, zsh, fish, or powershell:
```bash
agentdiff --install-completion
agentdiff init --scenario customer_support --runs 3 --with-approve
```

## Quickstart

### 1. Record a Golden Baseline
Record a canonical execution trace from any agent function without writing boilerplate telemetry:
### 2. Record a Statistical Baseline Envelope
Record an N-run baseline envelope from any agent function without writing boilerplate telemetry:

```bash
agentdiff record my_agent:run --input '{"query": "summarize repo"}' --out baselines/golden.json
agentdiff record my_agent:run \
--input '{"query": "summarize repo"}' \
--runs 3 \
--out baselines/customer_support.envelope.json
```

### 2. Compare Traces in CLI
Compare candidate runs against your golden baseline:
### 3. Compare Traces in CLI
Compare candidate runs against your baseline envelope:

```bash
agentdiff baselines/golden.json traces/candidate.json --fail-on-regression --max-divergence 0.25
agentdiff diff baselines/customer_support.envelope.json traces/candidate.json --fail-on-regression
```

### 3. Pytest Regression Testing
### 4. Pytest Regression Testing
Enforce trajectory parity directly in your test suite:

```python
Expand All @@ -122,34 +92,6 @@ def test_agent_refactor_efficiency():
)
```

## Config-as-Code (`agentdiff.toml`)

Commit your gate policy directly to your repository. AgentDiff auto-discovers `agentdiff.toml` in your working directory tree:

```toml
[compare]
detect_loops = true
strict_tool_signatures = false

[adapter]
name = "auto" # auto, generic, openinference, langfuse, langsmith, openai_agents

[cli]
format = "terminal" # terminal, json, markdown, pr
baseline = "baselines/golden.json"
max_loops = 0
max_divergence = 0.25
max_cost_delta = 5.0
max_recovery_ratio = 1.5

[assertions] # Defaults for assert_no_regressions / pytest plugin
max_divergence = 0.25
max_cost_increase_pct = 5.0
allow_loops = false
max_wasted_effort = 0.10
max_recovery_step_ratio = 1.5
```

## GitHub Actions CI Gate

Block broken agent PRs before they land in production using the official composite action:
Expand All @@ -174,13 +116,10 @@ jobs:
with:
python-version: "3.11"

- uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.2.2
- uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0
with:
baseline: baselines/golden.json
baseline: baselines/customer_support.envelope.json
candidate: traces/pr_candidate.json
max-divergence: "0.25"
max-cost-delta: "5.0"
max-loops: "0"
pr: ${{ github.event.pull_request.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}
```
Expand Down
55 changes: 31 additions & 24 deletions website/docs/01-Getting Started/02-Quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,51 +12,58 @@ uv add agent-trajectory-diff

This installs the `agentdiff` CLI and the `agentdiff` Python package.

## 2. No trace yet? Record one
## 2. Initialize with `agentdiff init`

Point `record` at any callable (your agent's entry function) and AgentDiff runs
it once and captures a canonical trace — no telemetry or framework needed:
Run the zero-config setup wizard in your project root. It auto-detects your agent framework (LangGraph, CrewAI, OpenAI Agents SDK, OpenTelemetry, etc.) and generates your config plus CI gate workflow:

```bash
agentdiff record my_agent:run --input '{"question": "What is AgentDiff?"}' --out traces/run.json
agentdiff init --scenario customer_support --runs 3 --with-approve
```

- `--input` takes a JSON object (passed to the callable as kwargs) or `@file.json`
- A failed run is still recorded — diff it to see exactly what broke
## 3. Record a statistical baseline envelope

## 3. Compare two traces
Capture an N-run baseline envelope so variance bands prevent false-positive CI failures:

The CLI takes a baseline trace and a candidate trace. Run the same task twice
(e.g. on `main` and on your branch), export the traces, then:
```bash
agentdiff record my_agent:run \
--input '{"question": "What is AgentDiff?"}' \
--runs 3 \
--out baselines/customer_support.envelope.json
```

- `--input` takes a JSON object or `@file.json`
- `--runs 3` captures a statistical envelope with empirical mean ± k·sigma bands

## 4. Compare candidate runs

Run candidate executions against your baseline envelope:

```bash
agentdiff baseline.json candidate.json
agentdiff diff baselines/customer_support.envelope.json traces/candidate.json
```

AgentDiff auto-detects the telemetry format (`generic`, `openinference`,
`langfuse`, `langsmith`, `openai_agents`) and prints a terminal report with
the divergence metrics:
AgentDiff compares the candidate with min-TDI-of-N matching and variance bands:

```text
Trajectory Divergence Index (TDI): 0.33
Loops Detected: 1
Candidate Wasted Effort (WEI): 0.00
Cost Delta: +41.0%
Status: REGRESSION
Baseline: customer_support.envelope.json (3 runs)
Candidate: traces/candidate.json
TDI (min-of-3): 0.00 [PASS]
Step Count: 6 (band: 6.3 ± 0.9) [PASS]
Cost Delta: +2.1% [PASS]
Loops: 0 [PASS]
Status: PASSED
```

**Why did it diverge?** Add `--explain` for a human-readable breakdown and
`--tree` for a collapsed, visual comparison of the two paths:
**Why did it diverge?** Add `--explain` for a breakdown and `--tree` for a visual comparison:

```bash
agentdiff baseline.json candidate.json --explain --tree
agentdiff diff baselines/customer_support.envelope.json traces/candidate.json --explain --tree
```

**Gate it in CI.** Add `--fail-on-regression` to exit non-zero when thresholds
are exceeded:
**Gate it in CI.** Add `--fail-on-regression` to exit non-zero when hard invariants or thresholds are breached:

```bash
agentdiff baseline.json candidate.json --fail-on-regression
agentdiff diff baselines/customer_support.envelope.json traces/candidate.json --fail-on-regression --pr 12
```

The full CLI surface - including baseline rotation and PR comments - is covered
Expand Down
141 changes: 141 additions & 0 deletions website/docs/01-Getting Started/03-Init Wizard.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# `agentdiff init` — Zero-Config Setup Wizard

`agentdiff init` is the fastest way to add AgentDiff to an existing AI agent codebase. It inspects your project, auto-detects which agent framework or telemetry format you use, and generates a tailored `agentdiff.toml` policy file alongside a production-ready GitHub Actions regression gate workflow.

---

## 1. Quick Usage

Run in your project root:

```bash
agentdiff init
```

The wizard scans your environment, identifies installed packages, and outputs:

```text
Detected framework: LangGraph (StateGraph checkpoint parser)
Wrote agentdiff.toml
Wrote .github/workflows/agentdiff.yml

Next steps:
1. Record a baseline envelope:
agentdiff record <your_agent:run> --runs 3 --out baselines/default.envelope.json
2. Commit agentdiff.toml, the workflow(s), and the baseline.
3. Open a PR — AgentDiff gates it automatically.
4. Reviewers bless accepted drift with: /agentdiff approve
```

---

## 2. Auto-Detection Matrix

`agentdiff init` automatically detects the following frameworks by checking installed packages in your virtual environment:

| Framework / Adapter | Package Checked | Generated Config Adapter | Generated Workflow Step |
|---|---|---|---|
| **LangGraph** | `langgraph` | `adapter.name = "langgraph"` | Ingests native LangGraph StateGraph snapshots |
| **CrewAI** | `crewai` | `adapter.name = "crewai"` | Ingests multi-agent task hierarchy & CrewOutput dumps |
| **OpenAI Agents SDK** | `agents` | `adapter.name = "openai_agents"` | Ingests official OpenAI Agents SDK run trees |
| **OpenTelemetry / OpenInference** | `opentelemetry-api` | `adapter.name = "openinference"` | Standard GenAI span ingestion |
| **Generic Python** | *(fallback)* | `adapter.name = "auto"` | Ingests canonical AgentTrace JSON |

### Overriding Detection with `--adapter`

If you are using multiple frameworks or want to specify an adapter explicitly, pass `--adapter`:

```bash
agentdiff init --adapter crewai
```

---

## 3. CLI Flags & Options

```bash
agentdiff init [OPTIONS]
```

| Flag | Default | Description |
|---|---|---|
| `--scenario <name>` | `default` | Scenario name written into `agentdiff.toml` and workflow files. |
| `--runs <int>` | `3` | Number of sample runs configured for statistical baseline envelopes. |
| `--adapter <name>` | *(auto-detected)* | Override framework detection (`langgraph`, `crewai`, `openai_agents`, `openinference`, `generic`). |
| `--with-approve` | `false` | Also generate `.github/workflows/agentdiff-approve.yml` for in-PR `/agentdiff approve` re-baselining. |
| `--force` | `false` | Overwrite existing `agentdiff.toml` or workflow files if they already exist. |

---

## 4. Generated Artifacts

### 1. `agentdiff.toml` (v0.5 Spec)

```toml
[scenario.customer_support]
mode = "statistical"
sample_runs = 3
max_cost_increase_pct = 5.0

[scenario.customer_support.hard_invariants]
fail_on_identical_loops = true
max_tool_repeats = 3

[scenario.customer_support.tolerances]
step_count_std_dev = 2.0
divergence_ceiling = 0.35
```

### 2. `.github/workflows/agentdiff.yml`

```yaml
name: AgentDiff Gate

on:
pull_request:

permissions:
contents: read
pull-requests: write

jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install dependencies
run: |
pip install agent-trajectory-diff

- name: Record Candidate Run
run: |
agentdiff record my_agent:run \
--input '{"query": "sample query"}' \
--out traces/candidate.json

- name: Run AgentDiff Gate
uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0
with:
baseline: baselines/customer_support.envelope.json
candidate: traces/candidate.json
pr: ${{ github.event.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}
```

### 3. `.github/workflows/agentdiff-approve.yml` (when `--with-approve` is used)

Writes the command listener workflow that reacts to `/agentdiff approve` comments by repository maintainers, authenticating with the hosted `token.agentdiff.app` identity service and flipping the Checks API result.

---

## Next Steps

- Learn how [Statistical Baselines](08-Statistical%20Baselines.md) prevent flaking on non-deterministic agents.
- Explore the interactive [Approve Bot](09-Approve%20Bot.md) workflow.
- Review [Regression Gates](../02-Core%20Concepts/04-Regression%20Gates.md) for hard invariant details.
Loading
Loading