Skip to content

Add Jev semantic assertions on agent replies - #1

Draft
Doringber wants to merge 1 commit into
mainfrom
cursor/jev-semantic-assertions-d3d2
Draft

Doringber wants to merge 1 commit into
mainfrom
cursor/jev-semantic-assertions-d3d2

Conversation

@Doringber

Copy link
Copy Markdown
Owner

Summary

Agent replies change wording even when the meaning stays the same. This adds AgentExecutionResult.assert_means, which sends the response text to pytest-jev and records each claim's probability on assertion_outcomes so the HTML report shows it.

How to use it

pip install "agent-test-kit[jev]"
export OPENROUTER_API_KEY=...   # or TYPESAFE_API_KEY
async def test_refund_reply(agent_scenario, jev):
    result = await agent_client.execute({"ticket": "I was charged twice"})
    agent_scenario.attach_execution_result(result)
    result.assert_success()
    result.assert_means(
        jev,
        holds=[
            "apologizes to the customer",
            "says the duplicate payment was refunded",
        ],
        lacks=["asks for a password or a full card number"],
    )
  • String responses are judged as-is. Dict responses use the first non-empty field among text, summary, message, content, reply, and suggested_prompt (result.response_text).
  • Tests that request the jev fixture are marked jev. pytest -m "not jev" stays offline. Without an API key, pytest-jev skips those tests; --jev-require fails CI instead.
  • Copy-paste example: examples/jev_semantic_example.py.

Tests

tests/test_semantic_assertions.py uses a stand-in judge, so the suite does not call Jev or need an API key.

Open in Web Open in Cursor 

assert_means sends the response text through the pytest-jev fixture so tests can check what a reply means, and records each claim probability on the HTML report.

Co-authored-by: Dor Ingber  <Doringber@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants