Skip to content

Add behavior scorers and pass-rate assertions - #2

Merged
cursor[bot] merged 2 commits into
mainfrom
cursor/behavior-scorers-083e
Sep 24, 2026
Merged

cursor[bot] merged 2 commits into
mainfrom
cursor/behavior-scorers-083e

Conversation

@Doringber

Copy link
Copy Markdown
Owner

Why

AI agents can give different correct answers to the same input. Today the kit can only check exact expectations: tool order, exact response equality, and exact trace equality. This PR adds the first step from "expected result" to "expected behavior". A run passes when its score reaches a threshold, and a repeated run passes when enough of its runs pass.

The package stays lightweight. There are no new dependencies and no built-in LLM provider. Judgment-based scoring, such as an LLM judge or embeddings, lives in the consumer repo behind a Scorer protocol. This follows the same pattern as SideEffectVerifier.

What

  • Scorer protocol and Score. score(result) can be sync or async and returns a value in [0.0, 1.0] with an optional reason and details. Out-of-range values, NaN, booleans, and wrong return types are rejected.
  • Built-in deterministic scorers:
    • RequiredTermsScorer scores the fraction of terms found in the response.
    • ForbiddenTermsScorer scores 1.0 or 0.0.
    • RequiredFieldsScorer scores the fraction of fields present and non-null, with dotted paths.
  • await result.assert_score(scorer, min_score=...) is recorded in assertion_outcomes. The score, threshold, and reason appear in the existing JSON and HTML assertion tables with no reporter changes.
  • RepeatedExecutionResult.assert_pass_rate(check, min_rate=...) checks a pass rate across repeated runs. A check that raises AssertionError counts as a failed run, so existing result.assert_* calls can be reused as checks.
  • await RepeatedExecutionResult.assert_score_rate(scorer, min_score=..., min_rate=...) checks that a scorer reaches its threshold on enough runs.
  • README section, public API list, and a repeated-run example in examples/consumer_test_example.py.

Existing APIs are unchanged, including assert_responses_stable and assert_traces_stable.

Testing

  • tests/test_scoring.py has 44 parametrized unit tests covering the scorers, threshold boundaries, validation, sync and async scorers, pass-rate failure messages, and the path of score outcomes into the JSON report.
  • Coverage is 99% for scoring.py, 94% for repetition.py, and 95% for models/execution.py.
  • Full suite: 204 passed, 19 skipped, 3 failed. The 3 failures also fail on main before this change and are unrelated:
    • test_version.py expects 0.1.1, but VERSION is 0.1.2.
    • One case in test_cursor_agent_client.py for malformed payloads.
  • ruff check and ruff format are clean on all touched files. mypy --strict shows no new errors; the one existing error in reporting/json_report.py:117 is untouched.
Open in Web Open in Cursor 

cursoragent and others added 2 commits September 24, 2026 17:35
Adds a Scorer protocol (sync or async), deterministic built-in scorers
(required terms, forbidden terms, required fields), assert_score on a single
execution, and assert_pass_rate / assert_score_rate on repeated runs. Score
outcomes flow into the existing JSON and HTML report assertion tables.

Co-authored-by: Dor Ingber  <Doringber@users.noreply.github.com>
Co-authored-by: Dor Ingber  <Doringber@users.noreply.github.com>
@Doringber
Doringber marked this pull request as ready for review September 24, 2026 18:34
@cursor
cursor Bot merged commit 1390607 into main Sep 24, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants