Add release readiness (Go/No-Go) to agent test reports

README.md

@@ -308,6 +308,52 @@ aggregate.assert_responses_stable()
 aggregate.assert_traces_stable()
 ```
 
+### Behavior scoring (no single expected result)
+
+AI output can vary between runs and still be correct. Score the behavior against a threshold instead of comparing to one exact answer.
+
+| API | What it checks |
+|-----|----------------|
+| `await result.assert_score(scorer, *, min_score)` | One run scores ≥ `min_score` (0.0–1.0); score and reason appear in the report |
+| `aggregate.assert_pass_rate(check, *, min_rate)` | `check(result)` passes on ≥ `min_rate` of repeated runs; `AssertionError` counts as a failed run |
+| `await aggregate.assert_score_rate(scorer, *, min_score, min_rate)` | Scorer reaches `min_score` on ≥ `min_rate` of repeated runs |
+
+Built-in scorers are deterministic and dependency-free:
+
+| Scorer | Score |
+|--------|-------|
+| `RequiredTermsScorer(terms)` | Fraction of terms present in the response |
+| `ForbiddenTermsScorer(terms)` | 1.0 if none present, else 0.0 |
+| `RequiredFieldsScorer(fields)` | Fraction of fields present and non-null (dotted paths: `"order.status"`) |
+
+```python
+from agent_test_kit import ForbiddenTermsScorer, RequiredTermsScorer, run_repeatedly
+
+await result.assert_score(RequiredTermsScorer(["order 123", "pending"]), min_score=1.0)
+await result.assert_score(ForbiddenTermsScorer(["system prompt", "api key"]), min_score=1.0)
+
+aggregate = await run_repeatedly(agent_client, input={"order_id": 123}, runs=5, idempotency_key="k")
+aggregate.assert_pass_rate(lambda r: r.success, min_rate=0.8)
+await aggregate.assert_score_rate(RequiredTermsScorer(["pending"]), min_score=1.0, min_rate=0.8)
+```
+
+Judgment-based scoring (LLM judge, embeddings, rubric) lives in **your** agent repo. Implement the `Scorer` protocol; `score` may be sync or async:
+
+```python
+from agent_test_kit import Score, response_text
+
+
+class GroundednessJudge:
+    name = "groundedness"
+
+    async def score(self, result) -> Score:
+        verdict = await my_llm_judge(response_text(result))
+        return Score(verdict.value, reason=verdict.rationale)
+
+
+await result.assert_score(GroundednessJudge(), min_score=0.8)
+```
+
 ### Regression cases (YAML-driven)
 
 Load cases with `load_regression_cases(path)`; evaluate with `case.evaluate(result)`.
@@ -335,6 +381,45 @@ for case in cases:
 
 Mark regression tests with `@pytest.mark.agent_regression`.
 
+**Goal-based cases.** Describe what the user must achieve and what must never happen, then allow behavior to vary across repeated runs. All fields are optional; existing cases are unchanged.
+
+| Case field | Meaning |
+|------------|---------|
+| `goal` | What the user must accomplish (shown in the evaluation) |
+| `persona` | Who is asking, e.g. `frustrated_customer` |
+| `runs` | How many times `case.run()` executes the input (default `1`) |
+| `behavior.min_pass_rate` | Fraction of runs that must pass (default `1.0`) |
+| `behavior.required_terms` / `forbidden_terms` / `required_fields` | Built-in checks applied to every run, including `case.evaluate()` |
+| `behavior.scores` | `scorer_name: min_score` for scorers you pass to `case.run()` |
+
+```yaml
+- id: refund_status_frustrated_customer
+  source: PROD-412
+  description: Customer asks angrily where their refund is
+  goal: User learns the current refund status for order 123
+  persona: frustrated_customer
+  runs: 5
+  input: {message: "Where is my refund for order 123?"}
+  expected_outcome:
+    success: true
+    read_only: true
+    forbidden_tools: [{server: billing, name: issue_refund}]
+  behavior:
+    min_pass_rate: 0.8
+    required_terms: ["123"]
+    forbidden_terms: ["system prompt", "api key"]
+    scores: {groundedness: 0.8}
+  created_at: 2026-09-24T00:00:00Z
+```
+
+```python
+for case in load_regression_cases("tests/data/behavior_cases.yaml"):
+    evaluation = await case.run(agent_client, scorers=[GroundednessJudge()])
+    evaluation.assert_passed()  # fails with pass rate and per-run failures
+```
+
+`case.run()` executes `runs` times with one idempotency key (`regression:<id>` by default) and checks that every scorer named in `behavior.scores` was provided before calling the agent. To score results you already have, use `await case.evaluate_runs(results, scorers=...)`. Full example: [`examples/behavior_regression_cases.yaml`](examples/behavior_regression_cases.yaml).
+
 ### Typical PR-review workflow (all together)
 
 ```python
@@ -551,6 +636,25 @@ pytest tests/ \
 
 Tool-flow data appears when you attach execution results — via fixtures, `agent_scenario.attach_execution_result(result)`, or `store_execution_result(...)`.
 
+**Release readiness.** Every report opens with a **GO / NO-GO** block: evidence for a human release decision, not a replacement for it. Failed or errored scenarios, security failures, and duplicate writes block. Everything else is a note unless you set a threshold.
+
+| Option | Effect |
+|--------|--------|
+| `--agent-baseline=reports/main.json` | Compare with a previous JSON report: regressions block; pass-rate drops, scenarios not run, token growth above 20%, and fingerprint changes (`model`, `prompt_version`, `agent_version`, `knowledge_version`) are notes. A missing baseline is a note. |
+| `--agent-min-persona-pass-rate=0.8` | Block when any persona's pass rate is below the value |
+| `--agent-enforce-readiness` | Fail the pytest session on NO-GO (default: report only) |
+
+Attach goal-based evaluations so the report can segment by persona:
+
+```python
+async def test_refund_case(agent_client, agent_scenario):
+    evaluation = await case.run(agent_client)
+    agent_scenario.attach_regression_evaluation(evaluation)
+    evaluation.assert_passed()
+```
+
+Set `AGENT_TEST_KNOWLEDGE_VERSION` (RAG index or knowledge-base version) next to model and prompt version so answer drift can be traced to its source. To evaluate outside pytest: `evaluate_readiness(load_report(path), baseline=load_report(base), policy=ReadinessPolicy(max_pass_rate_drop=0.1))`.
+
 ### Step 12 — Wire into CI
 
 ```bash
@@ -621,14 +725,24 @@ from agent_test_kit import (
     AgentExecutionResult,
     AnyOfStep,
     CleanupManager,
+    ForbiddenTermsScorer,
     HtmlReportWriter,
     JsonReportWriter,
     OptionalStep,
     PromptReviewClient,
+    ReadinessPolicy,
+    RegressionRunEvaluation,
     RepeatedExecutionResult,
+    RequiredFieldsScorer,
+    RequiredTermsScorer,
+    Score,
+    Scorer,
     ToolStep,
+    evaluate_readiness,
     get_profile,
     load_regression_cases,
+    load_report,
+    response_text,
     run_repeatedly,
     run_verifiers,
     SideEffectVerifier,

examples/behavior_regression_cases.yaml

@@ -0,0 +1,48 @@
+# Goal-based regression cases: describe what the user must achieve and what must
+# never happen, then let behavior vary across repeated runs within a pass rate.
+#
+# Load with load_regression_cases(path) and execute with:
+#   evaluation = await case.run(agent_client, scorers=[GroundednessJudge()])
+#   evaluation.assert_passed()
+
+cases:
+  - id: refund_status_frustrated_customer
+    source: PROD-412
+    description: Customer asks angrily where their refund is
+    goal: User learns the current refund status for order 123
+    persona: frustrated_customer
+    runs: 5
+    input:
+      message: "It's been two weeks. Where is my refund for order 123?"
+    expected_outcome:
+      success: true
+      read_only: true
+      forbidden_tools:
+        - server: billing
+          name: issue_refund
+    behavior:
+      min_pass_rate: 0.8
+      required_terms: ["123"]
+      forbidden_terms: ["system prompt", "api key"]
+      scores:
+        groundedness: 0.8
+    tags: [billing, nondeterministic]
+    created_at: 2026-09-24T00:00:00Z
+
+  - id: order_lookup_structured
+    source: PROD-398
+    description: Order lookup returns a structured summary
+    goal: User receives order status and delivery date
+    persona: new_customer
+    runs: 3
+    input:
+      order_id: 123
+    expected_outcome:
+      success: true
+      required_tools:
+        - server: billing
+          name: get_order
+    behavior:
+      required_fields: [order.status, order.delivery_date]
+    tags: [billing]
+    created_at: 2026-09-24T00:00:00Z

examples/consumer_test_example.py

@@ -4,6 +4,8 @@
 
 import pytest
 
+from agent_test_kit import ForbiddenTermsScorer, RequiredFieldsScorer, run_repeatedly
+
 
 @pytest.mark.asyncio
 @pytest.mark.agent_integration
@@ -21,3 +23,29 @@ async def test_agent_flow(agent_client: object) -> None:
     result.assert_tool_called("bitbucket_get_diff", server="bitbucket")
     result.assert_tool_called("jira_create_issue", server="jira")
     result.assert_tool_not_called("bitbucket_merge_pull_request", server="bitbucket")
+
+
+@pytest.mark.asyncio
+@pytest.mark.agent_integration
+@pytest.mark.agent_nondeterministic
+async def test_agent_behavior_is_reliable(agent_client: object) -> None:
+    """Accept output variance, but require reliable behavior across runs."""
+    aggregate = await run_repeatedly(
+        agent_client,  # type: ignore[arg-type]
+        input={"repository": "test-repository", "pull_request_id": 125},
+        runs=5,
+        idempotency_key="pr-125-review",
+    )
+
+    aggregate.assert_no_duplicate_writes()
+    aggregate.assert_pass_rate(lambda result: result.success, min_rate=0.8)
+    await aggregate.assert_score_rate(
+        RequiredFieldsScorer(["summary", "risk_level"]),
+        min_score=1.0,
+        min_rate=0.8,
+    )
+    await aggregate.assert_score_rate(
+        ForbiddenTermsScorer(["api key", "system prompt"]),
+        min_score=1.0,
+        min_rate=1.0,
+    )

src/agent_test_kit/__init__.py

@@ -23,10 +23,12 @@
     ResultPublishError,
 )
 from agent_test_kit.regression import (
+    RegressionBehavior,
     RegressionCase,
     RegressionCaseLoadError,
     RegressionEvaluation,
     RegressionExpectedOutcome,
+    RegressionRunEvaluation,
     RegressionToolExpectation,
     load_regression_cases,
 )
@@ -38,6 +40,15 @@
 from agent_test_kit.reporting.html_report import HtmlReportWriter
 from agent_test_kit.reporting.json_report import JsonReportWriter
 from agent_test_kit.reporting.markdown_status import MarkdownStatusReportWriter
+from agent_test_kit.reporting.readiness import ReadinessPolicy, evaluate_readiness, load_report
+from agent_test_kit.scoring import (
+    ForbiddenTermsScorer,
+    RequiredFieldsScorer,
+    RequiredTermsScorer,
+    Score,
+    Scorer,
+    response_text,
+)
 from agent_test_kit.verifiers import (
     SideEffectVerifier,
     VerificationContext,
@@ -57,6 +68,7 @@
     "CleanupManager",
     "CursorAgentClient",
     "FlowAssertionEngine",
+    "ForbiddenTermsScorer",
     "HtmlReportWriter",
     "HttpResultPublisher",
     "JsonReportWriter",
@@ -65,24 +77,34 @@
     "PromptReviewClient",
     "PromptAiHelperProfile",
     "PublishResult",
+    "ReadinessPolicy",
+    "RegressionBehavior",
     "RegressionCase",
     "RegressionCaseLoadError",
     "RegressionEvaluation",
     "RegressionExpectedOutcome",
+    "RegressionRunEvaluation",
     "RegressionToolExpectation",
     "RepeatedExecutionResult",
+    "RequiredFieldsScorer",
+    "RequiredTermsScorer",
     "ResultPublishError",
     "ResultPublisher",
     "ResultPublisherConfig",
     "RunContext",
+    "Score",
+    "Scorer",
     "SideEffectVerifier",
     "ToolStep",
     "VerificationContext",
     "VerificationResult",
     "VerificationSummary",
     "__version__",
+    "evaluate_readiness",
     "load_regression_cases",
+    "load_report",
     "get_profile",
+    "response_text",
     "run_repeatedly",
     "run_verifiers",
 ]

src/agent_test_kit/client/config.py

@@ -31,6 +31,10 @@ class AgentTestConfig(BaseSettings):
     environment: str = Field(default="local")
     model: str | None = None
     prompt_version: str | None = None
+    knowledge_version: str | None = Field(
+        default=None,
+        description="Version of the RAG index or knowledge base the agent answered from",
+    )
     repository: str | None = None
     branch: str | None = None
     commit: str | None = None

src/agent_test_kit/models/execution.py

@@ -19,6 +19,7 @@
         WorkflowStep,
     )
     from agent_test_kit.models.enums import ToolCallStatus
+    from agent_test_kit.scoring import Score, Scorer
 
 
 class AssertionOutcome(BaseModel):
@@ -335,6 +336,29 @@ def assert_write_tool_called_once(
             ),
         )
 
+    async def assert_score(self, scorer: Scorer, *, min_score: float) -> Score:
+        """Score this result and require ``score.value >= min_score``."""
+        from agent_test_kit.scoring import evaluate_score, validate_unit_interval
+
+        validate_unit_interval("min_score", min_score)
+        score = await evaluate_score(scorer, self)
+
+        def evaluate() -> None:
+            if score.value < min_score:
+                detail = f" ({score.reason})" if score.reason else ""
+                raise AssertionError(
+                    f"Expected {scorer.name} score >= {min_score:.2f}. "
+                    f"Observed: {score.value:.2f}{detail}"
+                )
+
+        self._capture_assertion(
+            "assert_score",
+            expected={"scorer": scorer.name, "min_score": min_score},
+            actual={"score": score.value, "reason": score.reason, "details": score.details},
+            assertion=evaluate,
+        )
+        return score
+
     @classmethod
     def from_api_response(
         cls,