diff --git a/README.md b/README.md index 3e5805ce..895d7b65 100644 --- a/README.md +++ b/README.md @@ -32,70 +32,40 @@ AgentDiff aligns candidate execution DAGs against committed golden baselines in ## Highlights - **Deterministic Graph Diffing:** Topological DAG alignment and Longest Common Subsequence (LCS) step comparison in `<10ms` with zero paid LLM-judge calls. -- **100% Local & Air-Gapped:** Zero telemetry, no cloud accounts, no network calls during diffs. Raw prompts and tool outputs never leave your machine or CI runner. -- **Drop-in CI Merge Gate:** Native exit codes (`0` pass / `1` regression fail) and automated GitHub Action PR comments with collapsed divergence trees and culprit attribution. -- **Universal Telemetry Adapters:** Seamlessly diff traces exported from **LangGraph**, **CrewAI**, **OpenAI Agents SDK**, **Langfuse**, **LangSmith**, **OpenInference / OpenTelemetry**, or generic JSON. -- **Config-as-Code & Goodhart Guard:** Commit thresholds in `agentdiff.toml` right next to your code. Flag drift when baseline gate definitions change. -- **First-Class Pytest Plugin:** Native `agentdiff_trace` fixture and `assert_no_regressions` assertion helper. - -## Architecture - -```mermaid -flowchart LR - subgraph Ingestion["1. Ingestion"] - A1[LangGraph / CrewAI] --> T[Normalized AgentTrace] - A2[Langfuse / LangSmith] --> T - A3[OpenInference / OTel] --> T - A4[OpenAI Agents SDK] --> T - end - - subgraph DiffEngine["2. Diff Engine (<10ms)"] - T --> DAG[Topological DAG Align] - DAG --> Metrics[TDI · WEI · LBI · RSR · ΔCost] - end - - subgraph Enforcement["3. Merge Gate"] - Metrics --> Gate{Thresholds Violated?} - Gate -- No --> Pass[Exit 0 · Update Baseline] - Gate -- Yes --> Fail[Exit 1 · Block PR & Post Root-Cause Comment] - end -``` - -## Installation - -```bash -# Using pip -pip install agent-trajectory-diff +- **Statistical Baselines & Variance Bands:** Capture N-run envelopes (`record --runs 3`) so non-deterministic agents don't flake CI on harmless jitter. +- **Zero-Config Setup (`agentdiff init`):** Auto-detects LangGraph, CrewAI, OpenAI Agents SDK, or OpenTelemetry and writes `agentdiff.toml` + CI workflow in seconds. +- **In-PR Interactive Blessings (`/agentdiff approve`):** Reviewers bless intended trajectory improvements from PR comments as `agentdiff-ci[bot]`. +- **100% Local & Air-Gapped:** Zero telemetry, no cloud accounts, no outbound network calls during diffs. Raw prompts and tool outputs stay local. +- **Drop-in CI Merge Gate:** Native exit codes (`0` pass / `1` regression fail) and automated GitHub Action PR comments with human-first verdicts. +- **Universal Telemetry Adapters:** Seamlessly diff traces from **LangGraph**, **CrewAI**, **OpenAI Agents SDK**, **Langfuse**, **LangSmith**, **OpenInference / OpenTelemetry**, or generic JSON. -# Using uv -uv add agent-trajectory-diff +## Quickstart -# Global CLI tool (isolated environment) -uv tool install agent-trajectory-diff -``` +### 1. Initialize with `agentdiff init` +Auto-detect your agent framework and generate your configuration + CI workflow: -Enable tab completion for bash, zsh, fish, or powershell: ```bash -agentdiff --install-completion +agentdiff init --scenario customer_support --runs 3 --with-approve ``` -## Quickstart - -### 1. Record a Golden Baseline -Record a canonical execution trace from any agent function without writing boilerplate telemetry: +### 2. Record a Statistical Baseline Envelope +Record an N-run baseline envelope from any agent function without writing boilerplate telemetry: ```bash -agentdiff record my_agent:run --input '{"query": "summarize repo"}' --out baselines/golden.json +agentdiff record my_agent:run \ + --input '{"query": "summarize repo"}' \ + --runs 3 \ + --out baselines/customer_support.envelope.json ``` -### 2. Compare Traces in CLI -Compare candidate runs against your golden baseline: +### 3. Compare Traces in CLI +Compare candidate runs against your baseline envelope: ```bash -agentdiff baselines/golden.json traces/candidate.json --fail-on-regression --max-divergence 0.25 +agentdiff diff baselines/customer_support.envelope.json traces/candidate.json --fail-on-regression ``` -### 3. Pytest Regression Testing +### 4. Pytest Regression Testing Enforce trajectory parity directly in your test suite: ```python @@ -122,34 +92,6 @@ def test_agent_refactor_efficiency(): ) ``` -## Config-as-Code (`agentdiff.toml`) - -Commit your gate policy directly to your repository. AgentDiff auto-discovers `agentdiff.toml` in your working directory tree: - -```toml -[compare] -detect_loops = true -strict_tool_signatures = false - -[adapter] -name = "auto" # auto, generic, openinference, langfuse, langsmith, openai_agents - -[cli] -format = "terminal" # terminal, json, markdown, pr -baseline = "baselines/golden.json" -max_loops = 0 -max_divergence = 0.25 -max_cost_delta = 5.0 -max_recovery_ratio = 1.5 - -[assertions] # Defaults for assert_no_regressions / pytest plugin -max_divergence = 0.25 -max_cost_increase_pct = 5.0 -allow_loops = false -max_wasted_effort = 0.10 -max_recovery_step_ratio = 1.5 -``` - ## GitHub Actions CI Gate Block broken agent PRs before they land in production using the official composite action: @@ -174,13 +116,10 @@ jobs: with: python-version: "3.11" - - uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.2.2 + - uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0 with: - baseline: baselines/golden.json + baseline: baselines/customer_support.envelope.json candidate: traces/pr_candidate.json - max-divergence: "0.25" - max-cost-delta: "5.0" - max-loops: "0" pr: ${{ github.event.pull_request.number }} github-token: ${{ secrets.GITHUB_TOKEN }} ``` diff --git a/website/docs/01-Getting Started/02-Quickstart.md b/website/docs/01-Getting Started/02-Quickstart.md index 6752b1f7..6cfd08d3 100644 --- a/website/docs/01-Getting Started/02-Quickstart.md +++ b/website/docs/01-Getting Started/02-Quickstart.md @@ -12,51 +12,58 @@ uv add agent-trajectory-diff This installs the `agentdiff` CLI and the `agentdiff` Python package. -## 2. No trace yet? Record one +## 2. Initialize with `agentdiff init` -Point `record` at any callable (your agent's entry function) and AgentDiff runs -it once and captures a canonical trace — no telemetry or framework needed: +Run the zero-config setup wizard in your project root. It auto-detects your agent framework (LangGraph, CrewAI, OpenAI Agents SDK, OpenTelemetry, etc.) and generates your config plus CI gate workflow: ```bash -agentdiff record my_agent:run --input '{"question": "What is AgentDiff?"}' --out traces/run.json +agentdiff init --scenario customer_support --runs 3 --with-approve ``` -- `--input` takes a JSON object (passed to the callable as kwargs) or `@file.json` -- A failed run is still recorded — diff it to see exactly what broke +## 3. Record a statistical baseline envelope -## 3. Compare two traces +Capture an N-run baseline envelope so variance bands prevent false-positive CI failures: -The CLI takes a baseline trace and a candidate trace. Run the same task twice -(e.g. on `main` and on your branch), export the traces, then: +```bash +agentdiff record my_agent:run \ + --input '{"question": "What is AgentDiff?"}' \ + --runs 3 \ + --out baselines/customer_support.envelope.json +``` + +- `--input` takes a JSON object or `@file.json` +- `--runs 3` captures a statistical envelope with empirical mean ± k·sigma bands + +## 4. Compare candidate runs + +Run candidate executions against your baseline envelope: ```bash -agentdiff baseline.json candidate.json +agentdiff diff baselines/customer_support.envelope.json traces/candidate.json ``` -AgentDiff auto-detects the telemetry format (`generic`, `openinference`, -`langfuse`, `langsmith`, `openai_agents`) and prints a terminal report with -the divergence metrics: +AgentDiff compares the candidate with min-TDI-of-N matching and variance bands: ```text -Trajectory Divergence Index (TDI): 0.33 -Loops Detected: 1 -Candidate Wasted Effort (WEI): 0.00 -Cost Delta: +41.0% -Status: REGRESSION +Baseline: customer_support.envelope.json (3 runs) +Candidate: traces/candidate.json +TDI (min-of-3): 0.00 [PASS] +Step Count: 6 (band: 6.3 ± 0.9) [PASS] +Cost Delta: +2.1% [PASS] +Loops: 0 [PASS] +Status: PASSED ``` -**Why did it diverge?** Add `--explain` for a human-readable breakdown and -`--tree` for a collapsed, visual comparison of the two paths: +**Why did it diverge?** Add `--explain` for a breakdown and `--tree` for a visual comparison: ```bash -agentdiff baseline.json candidate.json --explain --tree +agentdiff diff baselines/customer_support.envelope.json traces/candidate.json --explain --tree ``` -**Gate it in CI.** Add `--fail-on-regression` to exit non-zero when thresholds -are exceeded: +**Gate it in CI.** Add `--fail-on-regression` to exit non-zero when hard invariants or thresholds are breached: ```bash -agentdiff baseline.json candidate.json --fail-on-regression +agentdiff diff baselines/customer_support.envelope.json traces/candidate.json --fail-on-regression --pr 12 ``` The full CLI surface - including baseline rotation and PR comments - is covered diff --git a/website/docs/01-Getting Started/03-Init Wizard.md b/website/docs/01-Getting Started/03-Init Wizard.md new file mode 100644 index 00000000..6e1a5187 --- /dev/null +++ b/website/docs/01-Getting Started/03-Init Wizard.md @@ -0,0 +1,141 @@ +# `agentdiff init` — Zero-Config Setup Wizard + +`agentdiff init` is the fastest way to add AgentDiff to an existing AI agent codebase. It inspects your project, auto-detects which agent framework or telemetry format you use, and generates a tailored `agentdiff.toml` policy file alongside a production-ready GitHub Actions regression gate workflow. + +--- + +## 1. Quick Usage + +Run in your project root: + +```bash +agentdiff init +``` + +The wizard scans your environment, identifies installed packages, and outputs: + +```text +Detected framework: LangGraph (StateGraph checkpoint parser) +Wrote agentdiff.toml +Wrote .github/workflows/agentdiff.yml + +Next steps: + 1. Record a baseline envelope: + agentdiff record --runs 3 --out baselines/default.envelope.json + 2. Commit agentdiff.toml, the workflow(s), and the baseline. + 3. Open a PR — AgentDiff gates it automatically. + 4. Reviewers bless accepted drift with: /agentdiff approve +``` + +--- + +## 2. Auto-Detection Matrix + +`agentdiff init` automatically detects the following frameworks by checking installed packages in your virtual environment: + +| Framework / Adapter | Package Checked | Generated Config Adapter | Generated Workflow Step | +|---|---|---|---| +| **LangGraph** | `langgraph` | `adapter.name = "langgraph"` | Ingests native LangGraph StateGraph snapshots | +| **CrewAI** | `crewai` | `adapter.name = "crewai"` | Ingests multi-agent task hierarchy & CrewOutput dumps | +| **OpenAI Agents SDK** | `agents` | `adapter.name = "openai_agents"` | Ingests official OpenAI Agents SDK run trees | +| **OpenTelemetry / OpenInference** | `opentelemetry-api` | `adapter.name = "openinference"` | Standard GenAI span ingestion | +| **Generic Python** | *(fallback)* | `adapter.name = "auto"` | Ingests canonical AgentTrace JSON | + +### Overriding Detection with `--adapter` + +If you are using multiple frameworks or want to specify an adapter explicitly, pass `--adapter`: + +```bash +agentdiff init --adapter crewai +``` + +--- + +## 3. CLI Flags & Options + +```bash +agentdiff init [OPTIONS] +``` + +| Flag | Default | Description | +|---|---|---| +| `--scenario ` | `default` | Scenario name written into `agentdiff.toml` and workflow files. | +| `--runs ` | `3` | Number of sample runs configured for statistical baseline envelopes. | +| `--adapter ` | *(auto-detected)* | Override framework detection (`langgraph`, `crewai`, `openai_agents`, `openinference`, `generic`). | +| `--with-approve` | `false` | Also generate `.github/workflows/agentdiff-approve.yml` for in-PR `/agentdiff approve` re-baselining. | +| `--force` | `false` | Overwrite existing `agentdiff.toml` or workflow files if they already exist. | + +--- + +## 4. Generated Artifacts + +### 1. `agentdiff.toml` (v0.5 Spec) + +```toml +[scenario.customer_support] +mode = "statistical" +sample_runs = 3 +max_cost_increase_pct = 5.0 + +[scenario.customer_support.hard_invariants] +fail_on_identical_loops = true +max_tool_repeats = 3 + +[scenario.customer_support.tolerances] +step_count_std_dev = 2.0 +divergence_ceiling = 0.35 +``` + +### 2. `.github/workflows/agentdiff.yml` + +```yaml +name: AgentDiff Gate + +on: + pull_request: + +permissions: + contents: read + pull-requests: write + +jobs: + gate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install dependencies + run: | + pip install agent-trajectory-diff + + - name: Record Candidate Run + run: | + agentdiff record my_agent:run \ + --input '{"query": "sample query"}' \ + --out traces/candidate.json + + - name: Run AgentDiff Gate + uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0 + with: + baseline: baselines/customer_support.envelope.json + candidate: traces/candidate.json + pr: ${{ github.event.number }} + github-token: ${{ secrets.GITHUB_TOKEN }} +``` + +### 3. `.github/workflows/agentdiff-approve.yml` (when `--with-approve` is used) + +Writes the command listener workflow that reacts to `/agentdiff approve` comments by repository maintainers, authenticating with the hosted `token.agentdiff.app` identity service and flipping the Checks API result. + +--- + +## Next Steps + +- Learn how [Statistical Baselines](08-Statistical%20Baselines.md) prevent flaking on non-deterministic agents. +- Explore the interactive [Approve Bot](09-Approve%20Bot.md) workflow. +- Review [Regression Gates](../02-Core%20Concepts/04-Regression%20Gates.md) for hard invariant details. diff --git a/website/docs/02-Core Concepts/04-Regression Gates.md b/website/docs/02-Core Concepts/04-Regression Gates.md index 10f1a5ac..1785d1b9 100644 --- a/website/docs/02-Core Concepts/04-Regression Gates.md +++ b/website/docs/02-Core Concepts/04-Regression Gates.md @@ -28,28 +28,38 @@ AssertionError: AgentDiff Regression Verification Failed: - Candidate Wasted Effort Index (WEI) of 0.2500 exceeded threshold of 0.1000. ``` -## CLI: `--fail-on-regression` +## Hard Invariants vs. Soft Findings + +In AgentDiff 0.5.0, regression gates decouple **fatal architectural bugs** (hard invariants) from **evaluative drift** (soft findings): + +| Gate Category | Rule | Severity | Exit Code | Blessable in PR? | +|---|---|---|---|---| +| **Hard Invariant** | Identical Cyclical Loops (`fail_on_identical_loops = true`) | `BLOCK` | Exit `1` | ❌ Never | +| **Hard Invariant** | Tool Repeat Cap (`max_tool_repeats = 3`) | `BLOCK` | Exit `1` | ❌ Never | +| **Hard Invariant** | Error Recovery Cascade (Recovery ratio $\ge 3\times$) | `BLOCK` | Exit `1` | ❌ Never | +| **Soft Finding** | Trajectory Divergence Index ($> \text{ceiling}$) | `WARN` / `FAIL` | Exit `1` | ✅ Yes (`/agentdiff approve`) | +| **Soft Finding** | Token Cost Delta ($> \text{max\_cost\_delta}$) | `WARN` / `FAIL` | Exit `1` | ✅ Yes (`/agentdiff approve`) | + +## CLI: `agentdiff diff` ```bash -agentdiff baseline.json candidate.json --fail-on-regression +agentdiff diff baselines/default.envelope.json traces/candidate.json --fail-on-regression ``` -Exits with code `1` when a regression is detected. The default thresholds are: +Exits with code `1` when a regression is detected. The CLI default thresholds are: | Flag | Default | Meaning | | --- | --- | --- | | `--max-divergence` | `0.3` | Max TDI before regression. | | `--max-loops` | `0` | Max loop count before regression. | | `--max-cost-delta` | `10.0` | Max cost increase % before regression. | -| `--max-recovery-ratio` | *(off)* | Max Recovery Step Ratio before regression (opt-in — the gate is disabled unless set). | +| `--max-recovery-ratio` | `3.0` | Max Recovery Step Ratio before blocking (default hard gate at 3.0×). | ```bash -agentdiff baseline.json candidate.json \ +agentdiff diff baselines/default.envelope.json traces/candidate.json \ --fail-on-regression \ - --max-divergence 0.2 \ - --max-loops 1 \ - --max-cost-delta 5.0 \ - --max-recovery-ratio 1.5 + --max-divergence 0.25 \ + --max-cost-delta 5.0 ``` These defaults can also be committed in an `agentdiff.toml` (see diff --git a/website/docs/03-Guides/05-Configuration.md b/website/docs/03-Guides/05-Configuration.md index e691b8c7..7c4c493c 100644 --- a/website/docs/03-Guides/05-Configuration.md +++ b/website/docs/03-Guides/05-Configuration.md @@ -5,7 +5,7 @@ traces instead of repeating them as CLI flags. AgentDiff auto-discovers an `agentdiff.toml` file from the current directory upward, or you can point at it explicitly with `--config`. -## Example +## Example (`agentdiff.toml` v0.5) ```toml [compare] @@ -13,22 +13,29 @@ detect_loops = true strict_tool_signatures = false [adapter] -name = "auto" # auto, generic, openinference, langfuse, langsmith, openai_agents +name = "auto" # auto, generic, langgraph, crewai, openai_agents, openinference, langfuse, langsmith + +# Default scenario configuration +[scenario.default] +mode = "statistical" # "statistical" (envelope mode) or "strict" (single run) +sample_runs = 3 # Rolling window size for envelopes +max_cost_increase_pct = 5.0 + +[scenario.default.hard_invariants] +fail_on_identical_loops = true +max_tool_repeats = 3 + +[scenario.default.tolerances] +step_count_std_dev = 2.0 +divergence_ceiling = 0.35 [cli] format = "terminal" # terminal, json, markdown, pr -baseline = "baselines/current.json" +baseline = "baselines/default.envelope.json" max_loops = 0 max_divergence = 0.3 max_cost_delta = 10.0 -max_recovery_ratio = 1.5 # opt-in Recovery Step Ratio gate (omit to disable) - -[assertions] # defaults used by assert_no_regressions / pytest plugin -max_divergence = 0.25 -max_cost_increase_pct = 5.0 -allow_loops = false -max_wasted_effort = 0.1 -max_recovery_step_ratio = 1.5 +max_recovery_ratio = 3.0 ``` ## Precedence diff --git a/website/docs/03-Guides/08-Statistical Baselines.md b/website/docs/03-Guides/08-Statistical Baselines.md new file mode 100644 index 00000000..0c84d7ed --- /dev/null +++ b/website/docs/03-Guides/08-Statistical Baselines.md @@ -0,0 +1,98 @@ +# Statistical Baselines & N-Run Envelopes + +Modern AI agents often exhibit non-deterministic execution paths: step orders can vary slightly, tool responses may require extra or fewer tokens, and latency fluctuates. + +If CI diffs a candidate run against a single strict baseline trace, harmless non-determinism can trigger false-positive gate failures. **Statistical Baselines** (introduced in AgentDiff 0.5.0) solve this with **N-run baseline envelopes** and empirical variance bands. + +--- + +## 1. How It Works + +Instead of capturing a single execution trace, you record $N$ representative runs (e.g. $N=3$ or $N=5$) into a versioned **Baseline Envelope** (`agentdiff_baseline_envelope` artifact, schema 2.0.0): + +```bash +agentdiff record my_agent:run \ + --input '{"query": "Generate Q3 sales analysis"}' \ + --runs 3 \ + --out baselines/sales_analysis.envelope.json +``` + +When gating in CI, AgentDiff performs: +1. **Min-TDI-of-$N$ Matching**: If *any* of the $N$ recorded baseline runs explains the candidate's trajectory within tolerance, the sequence is considered valid. +2. **Variance Band Boundaries**: Calculates the empirical mean and standard deviation ($\mu \pm k\sigma$) across step count, latency, and token consumption. +3. **Hard Invariant Gating**: Infinite retry loops and repetitive stagnant failures are still strictly blocked regardless of variance bands. + +--- + +## 2. Baseline Envelope Structure + +The envelope JSON holds all $N$ canonical traces alongside computed statistical boundaries: + +```json +{ + "schema_version": "2.0.0", + "artifact_type": "agentdiff_baseline_envelope", + "scenario": "sales_analysis", + "runs": 3, + "statistics": { + "step_count": { "mean": 6.33, "std_dev": 0.47, "min": 6, "max": 7 }, + "cost_usd": { "mean": 0.0142, "std_dev": 0.0011 }, + "total_latency_ms": { "mean": 1820.0, "std_dev": 140.5 } + }, + "traces": [ ... ] +} +``` + +--- + +## 3. Configuration in `agentdiff.toml` + +Declare statistical tolerances in `agentdiff.toml`: + +```toml +[scenario.sales_analysis] +mode = "statistical" # Enables envelope gating mode +sample_runs = 3 # Target envelope size +max_cost_increase_pct = 5.0 + +[scenario.sales_analysis.hard_invariants] +fail_on_identical_loops = true # Zero-tolerance: loops always block +max_tool_repeats = 3 + +[scenario.sales_analysis.tolerances] +step_count_std_dev = 2.0 # Candidate must be within mean ± 2.0 * sigma +divergence_ceiling = 0.35 # Max acceptable sequence TDI +``` + +--- + +## 4. Comparing Against an Envelope + +Run the comparator explicitly against the envelope: + +```bash +agentdiff diff baselines/sales_analysis.envelope.json traces/candidate.json --fail-on-regression +``` + +Output: + +```text +Baseline: sales_analysis.envelope.json (3 runs, mode: statistical) +Candidate: traces/candidate.json + +TDI (min-of-3): 0.00 (matched Run 2) [PASS] +Step Count: 7 (envelope: 6.3 ± 0.9) [PASS] +Cost Delta: +2.1% (band: ≤ +5.0%) [PASS] +Loops: 0 loops [PASS] + +Verdict: PASSED (Candidate within baseline envelope variance bands) +``` + +--- + +## 5. Rolling Window Baseline Rotation + +When you update a baseline envelope (via CLI `--update-baseline` or `/agentdiff approve`), AgentDiff rotates the rolling window of `sample_runs`: the oldest run drops off and the candidate run joins, automatically recalculating the empirical $\mu$ and $\sigma$ bands. + +### Backward Compatibility +Existing single-trace baselines (`agent_trace.schema.json` v1.0.0) remain fully supported. When AgentDiff loads a single-run baseline, it wraps it as an envelope with $N=1$ in `strict` mode. diff --git a/website/docs/03-Guides/09-Approve Bot.md b/website/docs/03-Guides/09-Approve Bot.md new file mode 100644 index 00000000..0c2c4a17 --- /dev/null +++ b/website/docs/03-Guides/09-Approve Bot.md @@ -0,0 +1,84 @@ +# The Interactive Approve Bot (`/agentdiff approve`) + +When an engineer improves an agent's execution path (e.g. optimizing a 5-step workflow into 2 steps, or changing a prompt's tool call order), the trajectory intentionally diverges from the golden baseline. + +Rather than checking out the branch locally, recording a new baseline, and committing manual JSON files, reviewers can approve the candidate run directly from the GitHub Pull Request thread by commenting: + +```text +/agentdiff approve +``` + +--- + +## 1. How the Flow Works + +```mermaid +sequenceDiagram + autonumber + actor Reviewer as PR Reviewer + participant GitHub as GitHub PR Thread + participant Bot as AgentDiff Approve Bot + participant Token as token.agentdiff.app + participant Repo as Git Repository + + Reviewer->>GitHub: Comments "/agentdiff approve" + GitHub->>Bot: Triggers approve workflow + Bot->>Token: Authenticates as agentdiff[bot] + Bot->>Bot: Verifies D3 Invariants (No loops) + Bot->>Repo: Commits candidate trace to baseline envelope + Bot->>GitHub: Flips Check status to PASSED (Checks API) + Bot->>GitHub: Posts approval confirmation comment +``` + +--- + +## 2. Setting Up the Approve Bot + +### Option A: `agentdiff init --with-approve` (Recommended) + +When initializing your repository, pass `--with-approve`: + +```bash +agentdiff init --with-approve +``` + +This generates `.github/workflows/agentdiff-approve.yml` configured with permissions, commenter write-access checks, concurrency guards, and artifact handoff. + +### Option B: Three Identity Tiers + +The approve bot supports three authentication tiers that degrade gracefully: + +1. **Hosted Identity (Zero Config)**: When the [AgentDiff CI GitHub App](https://github.com/apps/agentdiff-ci) is installed on your repository, the workflow mints a short-lived (≤1 hour) token from `token.agentdiff.app`. Comments appear branded as **`agentdiff[bot]`**. +2. **Self-Managed GitHub App**: Set `AGENTDIFF_APP_ID` and `AGENTDIFF_APP_PRIVATE_KEY` repository secrets for dedicated organization-owned bots. +3. **Default `GITHUB_TOKEN`**: If no App is installed, the bot operates seamlessly using the repository's native `GITHUB_TOKEN` and posts comments as `github-actions[bot]`, flipping the check result green via the Checks API. + +--- + +## 3. D3 Safety Policy: Invariants Are Never Blessable + +AgentDiff enforces a strict separation between soft variations and structural bugs: + +| Condition | Example | Blessable via `/agentdiff approve`? | +|---|---|---| +| **Path Drift** | New tool sequence, alternative valid route | ✅ **Yes** (Human judgment) | +| **Cost / Token Increase** | +15% token usage due to better context | ✅ **Yes** (Human judgment) | +| **Cyclical Loops** | Same tool repeated with stagnant arguments | ❌ **NEVER** (Strictly blocked) | +| **Error Cascades** | ≥ 3× recovery steps spent looping on errors | ❌ **NEVER** (Strictly blocked) | + +If a PR contains an infinite loop, commenting `/agentdiff approve` will output an error refusing to bless the broken run until the underlying bug is fixed. + +--- + +## 4. CLI Command Reference + +The approve bot workflow executes the `agentdiff approve` CLI command behind the scenes: + +```bash +agentdiff approve [--scenario ] [--runs ] [--pr ] +``` + +Example: + +```bash +agentdiff approve baselines/customer_support.envelope.json traces/pr_candidate.json --pr 42 +``` diff --git a/website/docs/04-CI CD Integration/01-GitHub Actions Setup.md b/website/docs/04-CI CD Integration/01-GitHub Actions Setup.md index 08747054..24b55631 100644 --- a/website/docs/04-CI CD Integration/01-GitHub Actions Setup.md +++ b/website/docs/04-CI CD Integration/01-GitHub Actions Setup.md @@ -107,13 +107,10 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.11" - - uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.2.2 + - uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0 with: - baseline: traces/baseline.json + baseline: baselines/customer_support.envelope.json candidate: traces/candidate.json - max-divergence: "0.3" - max-loops: "0" - max-cost-delta: "10.0" pr: ${{ github.event.pull_request.number }} # optional github-token: ${{ secrets.GITHUB_TOKEN }} # required with pr ``` @@ -122,10 +119,10 @@ jobs: | Input | Default | Description | | --- | --- | --- | -| `baseline` | *(required)* | Path to the stored baseline trace JSON. | +| `baseline` | *(required)* | Path to the stored baseline trace JSON or statistical envelope. | | `candidate` | *(required)* | Path to the candidate trace JSON. | | `package` | `agent-trajectory-diff` | Package spec (PyPI name, `git+https://…`, or a local path). | -| `adapter` | `auto` | `auto`, `generic`, `openinference`, `langfuse`, `langsmith`, `openai_agents`. | +| `adapter` | `auto` | `auto`, `generic`, `langgraph`, `crewai`, `openinference`, `langfuse`, `langsmith`, `openai_agents`. | | `max-divergence` | `0.3` | Maximum Trajectory Divergence Index (TDI). | | `max-loops` | `0` | Maximum loop count. | | `max-cost-delta` | `10.0` | Maximum cost increase percentage. | @@ -133,7 +130,19 @@ jobs: | `pr` | *(empty)* | PR number to post the report comment to. | | `github-token` | *(empty)* | Token for the PR comment (required when `pr` is set). | -Pin the action to a release tag (`@v0.2.2`) for reproducible gates. +Pin the action to a release tag (`@v0.5.0`) for reproducible gates. + +## In-PR Approvals (`/agentdiff approve`) + +When developers alter an agent's expected trajectory, reviewers can re-baseline directly on GitHub by commenting `/agentdiff approve`. + +Generate the approve bot workflow with: + +```bash +agentdiff init --with-approve +``` + +The bot authenticates via the hosted [AgentDiff CI GitHub App](https://github.com/apps/agentdiff-ci) (or repository `GITHUB_TOKEN`), commits the blessed candidate run to the baseline envelope, and flips the GitHub Check result to green. ## Baseline rotation in CI diff --git a/website/docs/04-CI CD Integration/02-CI CD in Action.md b/website/docs/04-CI CD Integration/02-CI CD in Action.md index 15f37261..267e55b0 100644 --- a/website/docs/04-CI CD Integration/02-CI CD in Action.md +++ b/website/docs/04-CI CD Integration/02-CI CD in Action.md @@ -44,13 +44,11 @@ jobs: env: GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} run: python scripts/run_agent.py --prompt "${{ github.event.inputs.prompt }}" --out run.json - - uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.2.2 + - uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0 with: - baseline: traces/gemini_baseline.json - candidate: run.json - max-divergence: "0.3" - max-loops: "0" - pr: ${{ github.event.pull_request.number || github.event.inputs.pr }} + baseline: baselines/customer_support.envelope.json + candidate: traces/candidate.json + pr: ${{ github.event.pull_request.number }} github-token: ${{ secrets.GITHUB_TOKEN }} ```