Repository navigation
Add behavior scorers and pass-rate assertions - #2
Merged
Merged
Conversation
Adds a Scorer protocol (sync or async), deterministic built-in scorers (required terms, forbidden terms, required fields), assert_score on a single execution, and assert_pass_rate / assert_score_rate on repeated runs. Score outcomes flow into the existing JSON and HTML report assertion tables. Co-authored-by: Dor Ingber <Doringber@users.noreply.github.com>
Co-authored-by: Dor Ingber <Doringber@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
AI agents can give different correct answers to the same input. Today the kit can only check exact expectations: tool order, exact response equality, and exact trace equality. This PR adds the first step from "expected result" to "expected behavior". A run passes when its score reaches a threshold, and a repeated run passes when enough of its runs pass.
The package stays lightweight. There are no new dependencies and no built-in LLM provider. Judgment-based scoring, such as an LLM judge or embeddings, lives in the consumer repo behind a
Scorerprotocol. This follows the same pattern asSideEffectVerifier.What
Scorerprotocol andScore.score(result)can be sync or async and returns a value in[0.0, 1.0]with an optional reason and details. Out-of-range values, NaN, booleans, and wrong return types are rejected.RequiredTermsScorerscores the fraction of terms found in the response.ForbiddenTermsScorerscores 1.0 or 0.0.RequiredFieldsScorerscores the fraction of fields present and non-null, with dotted paths.await result.assert_score(scorer, min_score=...)is recorded inassertion_outcomes. The score, threshold, and reason appear in the existing JSON and HTML assertion tables with no reporter changes.RepeatedExecutionResult.assert_pass_rate(check, min_rate=...)checks a pass rate across repeated runs. A check that raisesAssertionErrorcounts as a failed run, so existingresult.assert_*calls can be reused as checks.await RepeatedExecutionResult.assert_score_rate(scorer, min_score=..., min_rate=...)checks that a scorer reaches its threshold on enough runs.examples/consumer_test_example.py.Existing APIs are unchanged, including
assert_responses_stableandassert_traces_stable.Testing
tests/test_scoring.pyhas 44 parametrized unit tests covering the scorers, threshold boundaries, validation, sync and async scorers, pass-rate failure messages, and the path of score outcomes into the JSON report.scoring.py, 94% forrepetition.py, and 95% formodels/execution.py.mainbefore this change and are unrelated:test_version.pyexpects 0.1.1, butVERSIONis 0.1.2.test_cursor_agent_client.pyfor malformed payloads.ruff checkandruff formatare clean on all touched files.mypy --strictshows no new errors; the one existing error inreporting/json_report.py:117is untouched.