Proposal: optional ResultSet reporter for EvalPort interop (no new deps, doesn't touch trace schema)

[adhabnr-ux]

Summary

A small, opt-in reporter — agentdiff.reporters.openeval — that renders a DiffReport as an EvalPort ResultSet (JSON), so a CI regression-gate run from AgentDiff can land in the same results pipeline/dashboard as evals from other frameworks. No new dependency, no change to the trace schema or the comparison engine — same shape as the existing reporters/pr.py / reporters/markdown.py / reporters/terminal.py modules, just another sink for a DiffReport.

Why I think this fits (and where it doesn't)

I read through engine/comparator.py, models/report.py, and the three existing reporters before writing this, so I want to be precise about the actual overlap, not a generic "let's integrate!" pitch.

AgentDiff's DiffReport isn't a per-input correctness grade — it's a deterministic structural diff between two trajectories (TDI via LCS alignment, WEI, loop detection, RSR, resource deltas). EvalPort's TestCase/expected_output/llm_judge grader machinery genuinely doesn't apply here, and I'm not proposing any of that. EvalPort's spec is explicit that it "does not define a trace format" and only references trace IDs — so I'm not proposing AgentDiff adopt EvalPort's AgentTrace shape either; schema/agent_trace.schema.json stays exactly as-is.

The part that does line up is the result layer, not the input layer. EvalPort's ResultSet is just: one result per evaluated unit, each carrying grader_results (id/type/score/passed/reason), rolling up to a summary. That maps cleanly onto one DiffReport if each of AgentDiff's checks (TDI, WEI, loop count, RSR, cost/latency/token delta) becomes one grader_result. The spec's grader type system is explicitly open for this: any non-standard type string is valid as long as params.handler is set (spec/SPEC.md §"Type openness (normative)"), which is exactly the "framework-native, not a fake standard grader" case — same pattern the repo's own crewai-openeval-adapter uses for gr_tool_selection (type: "custom", params.handler: "crewai:tools_subset").

Concretely, one comparison run (baseline_id vs candidate_id) becomes one results[] entry:

# agentdiff/reporters/openeval.py  (sketch — not a working PR yet)
from agentdiff.models.report import DiffReport

def generate_openeval_resultset(
    report: DiffReport,
    *,
    suite_id: str = "agentdiff",
    run_id: str,
    started_at: str,
    completed_at: str,
) -> dict:
    def grader(gid, gtype, score, passed, reason):
        return {
            "grader_id": gid,
            "type": gtype,
            "score": score,
            "passed": passed,
            "reason": reason,
            "params": {"handler": f"agentdiff:{gtype}"},  # required for non-standard types
        }

    grader_results = [
        grader("gr_trajectory_divergence", "agentdiff_tdi",
                1.0 - report.trajectory_divergence_index,
                report.trajectory_divergence_index <= 0.3,  # DEFAULT_MAX_DIVERGENCE in reporters/pr.py
                f"TDI={report.trajectory_divergence_index:.4f}"),
        grader("gr_wasted_effort", "agentdiff_wei",
                1.0 - report.candidate_wei, report.candidate_wei <= report.baseline_wei,
                f"candidate WEI={report.candidate_wei:.4f} (baseline {report.baseline_wei:.4f})"),
        grader("gr_loop_detection", "agentdiff_loops",
                0.0 if report.loops_detected else 1.0, not report.loops_detected,
                f"{len(report.loops_detected)} loop(s) detected"),
        grader("gr_recovery_ratio", "agentdiff_rsr",
                None, report.recovery_step_ratio <= 1.5,
                f"RSR={report.recovery_step_ratio:.2f} "
                f"({report.candidate_recovery_steps}/{report.baseline_recovery_steps})"),
        grader("gr_resource_delta", "agentdiff_resource_delta", None,
                report.cost_delta_percentage <= 10.0,
                f"cost {report.cost_delta_percentage:+.2f}%, "
                f"latency {report.latency_delta_percentage:+.2f}%, "
                f"tokens {report.token_delta_percentage:+.2f}%"),
    ]
    passed = sum(1 for g in grader_results if g["passed"])

    return {
        "version": "1.0.0",
        "suite_id": suite_id,
        "run_id": run_id,
        "started_at": started_at,
        "completed_at": completed_at,
        "results": [{
            "test_case_id": f"{report.baseline_id}__vs__{report.candidate_id}",
            "grader_results": grader_results,
            "passed": report.passed,
            "metadata": {
                "gate_provenance": report.gate_provenance,
                "violations": [v.message for v in report.violations],
                "warnings": [w.message for w in report.warnings],
            },
        }],
        "summary": {
            "total": len(grader_results),
            "passed": passed,
            "failed": len(grader_results) - passed,
            "pass_rate": passed / len(grader_results),
        },
    }

Honest caveats

• Narrow, real value today. EvalPort's own adoption notes (spec/ADOPTION.md) are candid that native framework support hasn't landed anywhere yet — this only helps if you're already aggregating results from multiple eval tools into one EvalPort-shaped store/dashboard. I'm not claiming demand exists inside this repo's userbase; I genuinely don't know if it does.
• One-way, and deliberately so. No TestCase import direction — AgentDiff doesn't consume single graded inputs, so there's nothing sensible to import. This is export-only, same direction as reporters/pr.py.
• Threshold values in the sketch above are illustrative defaults, not what a real PR would hardcode — a real implementation would thread through whatever thresholds the caller already passed to assert_no_regressions/generate_pr_markdown rather than re-deriving them.
• Zero new dependency: the sketch above is a plain dict, no evalport-sdk import required, so it doesn't trip the "adds a dependency" review gate in CONTRIBUTING.md. It also doesn't touch schema/ at all.

Since CONTRIBUTING.md asks to open an issue first for anything that touches public API surface, I'm posting this as a proposal rather than a PR — happy to build it as feat/openeval-resultset-reporter against current main (0.5.0) if this is a direction you'd want, or to hear why it isn't.

— Sahi, independent contributor (not affiliated with this project)

[lostmartian]

Thanks @adhabnr-ux, this is the highest-effort first issue this repo has received, and it shows: you clearly read comparator.py, the reporter modules, and the contributor gates before writing. The framing ("result layer, not input layer") is exactly the right seam, and respecting the no-LLM-judge non-goal is appreciated.

Before deciding, a few questions, mostly because your own caveats are the honest part of this proposal, and they deserve to be the deciding factor:

1. What's your real setup today? Are you running AgentDiff on a repo (gated CI, committed baseline)? And what consumes the ResultSet on your side: which store/dashboard, and what else in your pipeline already emits EvalPort-shaped results?
2. Timing: spec/ADOPTION.md is candid that no native framework support has landed yet. What makes an AgentDiff exporter now valuable vs. once EvalPort has 2-3 emitting tools? (Not a gotcha: we gate features on evidence, and "first external ask" is evidence, just not all of it.)
3. Scope: if this lands, it ships as an opt-in, zero-dependency module. Your "no evalport-sdk import" note is a hard requirement, not a preference.

One design note if we proceed: the sketch recomputes pass/fail from illustrative thresholds (e.g. RSR <= 1.5, loops via report.loops_detected), but 0.5.0's gate is now centralized in GateResult/GateFinding (hard violations vs soft warnings, with the actual thresholds the caller used). A real implementation should map grader_results from those findings directly, which is both more correct and less code than the sketch. (It also means your reporter inherits the [scenario.*] config for free.)

Happy with the export-only, one-way direction; that matches how we think about pr.py. Tell us more about (1)/(2) and we'll make the call.

[adhabnr-ux]

Appreciate the careful read, and I want to answer (1) and (2) honestly rather than shade them toward "yes, build it."

1. My real setup: I don't have a gated CI pipeline running AgentDiff against a committed baseline today. I'm not an AgentDiff user proposing this from my own pain point — I maintain EvalPort and I'm doing ecosystem-side outreach: reading real codebases and proposing exporters where the result shape genuinely maps, the same way I've approached every other framework this round (TruLens, an ares adapter now merged upstream, BenchFlow, CrewAI, AutoGen, and others). So there's no store/dashboard on my end consuming your ResultSets specifically — I don't want to imply otherwise. What is true and checkable: EvalPort's adapters/ directory currently has 48 packages, and several of the more recent ones (ares, BenchFlow) went through exactly this same propose-then-build process with their maintainers reviewing the actual mapping, not a rubber stamp. That's real but it's still "the format exists and a few tools emit it," not "AgentDiff users are asking for this."

2. Timing: Given (1), I think your read is right — "first external ask" is evidence, not sufficient evidence on its own, and I don't have a second data point to offer you today. I'd rather say that plainly than manufacture urgency. If you'd rather wait until EvalPort has more adopters with actual dashboards consuming ResultSets cross-tool before spending review time on this, that's a completely reasonable call and I won't push back on it.

3. Scope: Agreed as a hard requirement, not a preference — opt-in, zero-dependency, export-only, reporters/openeval.py alongside pr.py/markdown.py/terminal.py, no evalport-sdk import, no touch to schema/.

On your design note — you're right, and I went and read testing/assertions.py/models/report.py to redo it properly rather than defend the sketch. evaluate_gate() → GateResult(violations: list[GateFinding], warnings: list[GateFinding]) is exactly the real source of truth, and it's a cleaner mapping than what I sketched:

# agentdiff/reporters/openeval.py (revised sketch)
from agentdiff.models.report import DiffReport, GateFinding
from agentdiff.testing.assertions import GateResult

def generate_openeval_resultset(
    report: DiffReport,
    gate_result: GateResult,
    *,
    suite_id: str = "agentdiff",
    run_id: str,
    started_at: str,
    completed_at: str,
) -> dict:
    def grader_result(finding: GateFinding) -> dict:
        return {
            "grader_id": f"gr_{finding.code}",
            "type": "custom",
            "score": None,  # GateFinding carries no numeric score, only pass/fail + message
            "passed": finding.severity.value != "hard",
            "reason": finding.message,
            "params": {"handler": f"agentdiff:{finding.code}"},
        }

    # Only findings that actually fired get a grader_result -- evaluate_gate()
    # doesn't emit a "passing" GateFinding for a check that found nothing wrong,
    # so an empty violations+warnings list means "every configured gate passed,"
    # not "zero gates were configured." That's worth being explicit about in
    # metadata rather than silently implying a fixed, always-present check set.
    grader_results = [grader_result(f) for f in gate_result.violations + gate_result.warnings]

    return {
        "version": "1.0.0",
        "suite_id": suite_id,
        "run_id": run_id,
        "started_at": started_at,
        "completed_at": completed_at,
        "results": [{
            "test_case_id": f"{report.baseline_id}__vs__{report.candidate_id}",
            "grader_results": grader_results,
            "passed": gate_result.passed,  # GateResult.passed == not violations
            "metadata": {
                "gate_provenance": report.gate_provenance,
                "checks_evaluated": len(grader_results),
                "trajectory_divergence_index": report.trajectory_divergence_index,
                "candidate_wei": report.candidate_wei,
                "recovery_step_ratio": report.recovery_step_ratio,
            },
        }],
        "summary": {
            "total": 1,
            "passed": 1 if gate_result.passed else 0,
            "failed": 0 if gate_result.passed else 1,
            "pass_rate": 1.0 if gate_result.passed else 0.0,
        },
    }

This drops the illustrative-threshold recomputation entirely (no more re-deriving <= 0.3/<= 1.5 from constants that duplicate what the caller already passed to evaluate_gate), inherits [scenario.*] config for free as you said, and is meaningfully shorter than the original. I also switched summary to count the one comparison as the unit (matching test_case_id), rather than counting individual grader checks in summary.total — a ResultSet.summary describes result-level pass/fail, and the checks-within-one-comparison count now lives in metadata.checks_evaluated instead, which felt like a more honest fit once I looked at how the other adapters in this ecosystem populate summary.

I'll leave this with you rather than push toward a PR — happy to write it if/when you'd rather revisit, and equally happy if the honest answer here is "not now."

[lostmartian]

This is about as honest as an ecosystem pitch gets, and it makes the decision easy: not now, with a designed landing path.

The blocker was never quality. The revised mapping is correct (verified against GateSeverity/GateFinding), shorter than your first sketch, and the summary re-framing to result-level units is the right call. The blocker is exactly the one you named: no consuming dashboard yet, and no AgentDiff user asking for cross-tool aggregation. This repo gates features on evidence, and we'd be adding public API surface ahead of both.

So here's the trigger, written down so nobody has to re-litigate it: when EvalPort has 2-3 emitting tools and at least one dashboard/store consuming ResultSets cross-tool, or an AgentDiff user asks for this, the blueprint in this thread becomes a PR we'll review fast. Your revised sketch is the spec.

Labeling this proposal + waiting-on-evidence and leaving it open as the design of record. And for what it's worth: the fact that we could hand this to a contributor who read the gate internals and came back with the right abstraction on the first revision says good things about how composable 0.5.0 left the reporter layer. If you're ever gating agents with AgentDiff for real, that's the evidence that trumps all of this.