Repository navigation
Add release readiness (Go/No-Go) to agent test reports - #4
Merged
Merged
Conversation
Adds a Scorer protocol (sync or async), deterministic built-in scorers (required terms, forbidden terms, required fields), assert_score on a single execution, and assert_pass_rate / assert_score_rate on repeated runs. Score outcomes flow into the existing JSON and HTML report assertion tables. Co-authored-by: Dor Ingber <Doringber@users.noreply.github.com>
Co-authored-by: Dor Ingber <Doringber@users.noreply.github.com>
Regression cases accept optional goal, persona, runs and behavior fields. Built-in behavior checks (required/forbidden terms, required fields) apply in evaluate(); named scores and min_pass_rate apply in evaluate_runs() and run(), which executes a case repeatedly and reports per-run failures. Co-authored-by: Dor Ingber <Doringber@users.noreply.github.com>
Co-authored-by: Dor Ingber <Doringber@users.noreply.github.com>
…on, and opt-in Go/No-Go enforcement Co-authored-by: Dor Ingber <Doringber@users.noreply.github.com>
Co-authored-by: Dor Ingber <Doringber@users.noreply.github.com>
cursor
Bot
changed the base branch from
cursor/behavior-regression-cases-083e
to
main
September 24, 2026 18:34
Doringber
marked this pull request as ready for review
September 24, 2026 18:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3 (goal-based regression cases).
What
Every JSON/HTML report now carries a
readinessblock: evidence for a human release decision.RegressionRunEvaluationare grouped by persona (weakest first).--agent-min-persona-pass-rateturns a weak persona into a blocking reason.--agent-baseline=<previous report.json>lists regressions (blocking), pass-rate drops, scenarios not run, token growth >20%, and fingerprint changes (model,prompt_version,agent_version,knowledge_version) as notes. A missing or unreadable baseline is a note, never a crash.--agent-enforce-readinessfails the session on NO-GO. Default is report-only.knowledge_versionconfig (AGENT_TEST_KNOWLEDGE_VERSION) for RAG / knowledge-base version.agent_scenario.attach_regression_evaluation(evaluation)fills case id, goal, persona, pass rate, and runs on the scenario.ReadinessPolicy,evaluate_readiness,load_reportfor use outside pytest.No new dependencies. All new report fields are optional, so existing reports and callers are unaffected.
Tests
tests/test_readiness.py(39 tests, parametrized) covers policy validation, blocking reasons, persona segments, baseline rules, JSON round-trip, HTML rendering/escaping, plugin session handling, and an end-to-endpytesterrun with enforcement.test_version.pyx2 andtest_cursor_agent_client_normalizes_malformed_tool_payloads[malformed_event1]).readiness.py100% coverage; total 88%.json_report.pyexpected_mcp_serversand long HTML/CSS lines) are unchanged.