Skip to content

Add goal-based regression cases evaluated across repeated runs - #3

Merged
cursor[bot] merged 4 commits into
mainfrom
cursor/behavior-regression-cases-083e
Sep 24, 2026
Merged

cursor[bot] merged 4 commits into
mainfrom
cursor/behavior-regression-cases-083e

Conversation

@Doringber

Copy link
Copy Markdown
Owner

Stacked on #2, which adds scorers and pass rates. Merge that one first.

Why

Today a regression case describes one exact path: required tools, order, counts. For AI behavior the useful contract is different. It says what the user must achieve, what must never happen, and how often the agent must get it right. This lets the same YAML files express "expected behavior", not only "expected result".

What

A regression case gains optional fields. Existing cases load and evaluate exactly as before.

Field Meaning
goal What the user must accomplish
persona Who is asking
runs Executions per case in case.run(); default 1, max 100
behavior.min_pass_rate Fraction of runs that must pass; default 1.0
behavior.required_terms / forbidden_terms / required_fields / case_sensitive Built-in deterministic checks
behavior.scores scorer_name: min_score for consumer-provided scorers

Evaluation works at three levels:

  • case.evaluate(result) stays sync and deterministic. It now also applies the built-in behavior checks and returns scores per check.
  • await case.evaluate_runs(results, scorers=[...]) evaluates each run, applies named scores, and compares the pass rate to min_pass_rate. It returns a RegressionRunEvaluation with the goal, persona, pass rate, and each run's failures and scores.
  • await case.run(executor, scorers=[...]) runs evaluate_runs over run_repeatedly with one idempotency key, regression:<id> by default. It rejects missing or duplicate scorer names before calling the agent, so a config mistake does not spend agent calls.

RegressionRunEvaluation.assert_passed() failures report the pass rate and the failures of each failing run.

Also included:

  • examples/behavior_regression_cases.yaml, which a test loads to keep it valid.
  • A README section and public API exports: RegressionBehavior and RegressionRunEvaluation.

Testing

  • tests/test_regression_behavior.py adds 21 tests. They cover schema validation and loader error context, built-in checks, pass-rate boundaries, named scores, missing or duplicate scorers, idempotency key propagation, and default behavior for cases without the new fields.
  • The existing tests/test_regression_cases.py passes unchanged.
  • Coverage for regression.py is 93%.
  • Full suite: 225 passed, 19 skipped, 3 failed. The 3 failures are the same ones that fail on main: the version check (0.1.1 vs 0.1.2) and one malformed-payload case in the cursor client.
  • ruff check and ruff format are clean. mypy --strict shows only the error in reporting/json_report.py:117 that already exists on main.
Open in Web Open in Cursor 

cursoragent and others added 4 commits September 24, 2026 17:35
Adds a Scorer protocol (sync or async), deterministic built-in scorers
(required terms, forbidden terms, required fields), assert_score on a single
execution, and assert_pass_rate / assert_score_rate on repeated runs. Score
outcomes flow into the existing JSON and HTML report assertion tables.

Co-authored-by: Dor Ingber  <Doringber@users.noreply.github.com>
Co-authored-by: Dor Ingber  <Doringber@users.noreply.github.com>
Regression cases accept optional goal, persona, runs and behavior fields.
Built-in behavior checks (required/forbidden terms, required fields) apply in
evaluate(); named scores and min_pass_rate apply in evaluate_runs() and run(),
which executes a case repeatedly and reports per-run failures.

Co-authored-by: Dor Ingber  <Doringber@users.noreply.github.com>
Co-authored-by: Dor Ingber  <Doringber@users.noreply.github.com>
@cursor
cursor Bot changed the base branch from cursor/behavior-scorers-083e to main September 24, 2026 18:34
@Doringber
Doringber marked this pull request as ready for review September 24, 2026 18:34
@cursor
cursor Bot merged commit dc8de3e into main Sep 24, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants