Auditing AI agents: a curated list of papers, tools, datasets, benchmarks, and standards covering reliability, monitoring, failure attribution, and decision records.
-
Updated
Nov 16, 2026 - Python
Auditing AI agents: a curated list of papers, tools, datasets, benchmarks, and standards covering reliability, monitoring, failure attribution, and decision records.
SAGE: self-correcting autonomous research via Multi-Hypothesis Failure Attribution (MHFA). Diagnose many causes of a failed experiment, verify the most critical, and route the fix to the right level.
A curated list of research on LLM-agent traces: evidence tracing, execution provenance, failure attribution, observability, runtime safety, memory provenance, and learning from traces. Companion list of arXiv:2606.04990.
Agent Trace Analysis Platform (atap) — pluggable pipeline for trace representation, analysis, error classification, failure attribution and recovery for LLM agents
What must be true of a diagnosis before it is allowed to change an agent? The article, its PDF, full derivations, the gate code and a script that recomputes every figure.
Observability and a transparent, auditable agent graph for Massive Intelligence (IM) swarms: OpenTelemetry-compatible span tracing, incremental drift detection, failure attribution, loop detection, thirteen alert channels, and SHA-256-chained, Ed25519-signed trace logs. Zero dependencies, node-free core, TypeScript-first.
Did My Agents Hand Off? A deterministic, no-model scorer for multi-agent traces: five checks, 18 explainable rules, every finding cites the events it fired on.
Executable contract-based reliability evaluation for long-horizon LLM agents
An explainable small judge model & data pipeline for LLM Agent trajectory evaluation, failure attribution, and experience extraction.
Record, replay, and automatically attribute failures in multi-agent LLM systems — deterministic replay, counterfactual + learned root-cause attribution, and verified fixes. Offline by default.
Telemetry-grounded, calibrated failure attribution for agent oversight (OTel GenAI + Who&When).
When the agent breaks, which layer dropped the ball? Operator-facing failure-attribution taxonomy for AI agent estates: ten-way dictionary, MAST + AgentRx crosswalks, postmortem template. Part of the Spine catalog.
Evidence and reproducibility tools for measuring admission changes and downstream fates in ordered validator pipelines.
Counterfactual replay for LLM agent failures — rewind, intervene, re-run, blame the step
FALCON: Failure Attribution and Localized Causal Interventions in Federated Learning — localizes which FL pipeline stage (selection / local training / compression / aggregation) originated a failure via matched record-replay and restore/inject/sham interventions
Failure attribution for agent pipelines — find which span caused the failure and what kind of fix it needs.
To associate your repository with the failure-attribution topic, visit your repo's landing page and select "manage topics."