Skip to content

feat(scorer): deterministic verify step (run a command, assert exit code) - #10

Open
dorrfrog wants to merge 2 commits into
mainfrom
feature/verify-step
Open

dorrfrog wants to merge 2 commits into
mainfrom
feature/verify-step

Conversation

@dorrfrog

@dorrfrog dorrfrog commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Closes #9.

Summary

Adds a deterministic verify step - the exec-test grader. A scenario declares
verify either on a turn (Turn.verify) or on the scenario
(Scenario.verify, end-of-conversation). The runner executes the
author-declared command in the per-scenario worktree and records the exit code

  • captured stdout into TurnOutput; the rule-based scorer asserts it as the
    verify dimension (gate with --threshold rules/verify:0). This is the
    strongest, cheapest grader for code-editing scenarios - "did the agent's change
    make the project's own test suite pass?" - the deterministic half an LLM judge
    can't match.

Design (no new payload, no signature change)

  • Schema (scenario.py): one VerifySpec (cmd argv, exit_code,
    output_contains plain substrings, timeout) on Turn.verify /
    Scenario.verify (parallel to state_expect).
  • Artifact (entities.py): TurnOutput.verify_result (per-turn) and
    scenario_verify_result on the final turn (per-scenario) - additive.
  • Runner (orchestrator.py): execute + capture via the orchestrator's own
    sandbox spawner; per-turn after git-state capture, per-scenario after the
    loop while the worktree still exists. Skip-on-agent-error -> tri-state
    passed=None (not a false fail).
  • Scorer (scorer/rules/verify.py): check_verify / has_verify wired
    alongside check_state; per-scenario emits a turn-less CheckEntry. Stays a
    CheckEntry in RulesPayload.

Security (default-deny)

verify runs an author-supplied command, so it is gated behind
--allow-verify-exec / BELT_ALLOW_VERIFY_EXEC, refused at setup without an
isolated worktree, routed through the active sandbox provider (in-container
under --sandbox docker), given a minimal credential-free env, output-capped,
and killed on timeout. Documented in SECURITY-MODEL.md (threat T15, control
S5.11, gate table) plus SCENARIOS.md / SCORING.md / CONFIGURATION.md /
OUTCOMES.md.

Example

examples/scenarios/showcase/verify/ on the tasktracker fixture: a per-turn
scenario and a multi-turn per-scenario scenario. (Fixed tasktracker's pytest
pythonpath so the src-layout package imports in a bare worktree.)

Test plan

  • New tests/scorer/rules/test_verify.py: schema, check_verify
    (pass/fail/skip), rules integration (per-turn + per-scenario), runner exec
    incl. timeout, and the setup gate (off / allowed / no-worktree).
  • tests/test_scenarios_layout.py extended to require Turn.verify and
    Scenario.verify showcase coverage.
  • Full suite: 3243 passed, 23 skipped; lint (black/isort/ruff/markdownlint/
    bandit/gitleaks/design) clean.
  • Default-deny verified via CLI (group refused without the flag, exit 1).
  • Real end-to-end run: agent edits the fixture, verify runs pytest in
    the worktree -> verify_result {exit_code: 0, "10 passed"}, 5/5 checks.

Out of scope

  • pre_run background daemons (separate seam/lifecycle).
  • Feeding verify stdout to the LLM judge (follow-up).

…ode)

Adds a `verify` exec-test grader - the deterministic, cheap signal an LLM
judge can't match: "did the agent's change make the project's own test
suite pass?". Declarable per-turn (`Turn.verify`) or per-scenario
(`Scenario.verify`, end-of-conversation). The runner executes the
author-declared command (argv, no shell) in the per-scenario worktree and
records the exit code + captured stdout into TurnOutput; the rule-based
scorer asserts it as the `verify` dimension (gate via rules/verify).

Security (default-deny, see docs/glossary/SECURITY-MODEL.md): verify
executes an author-supplied command, so it is gated behind
--allow-verify-exec / BELT_ALLOW_VERIFY_EXEC, runs only with an isolated
worktree (group refused at setup otherwise), routes through the active
sandbox provider, gets a minimal credential-free env, caps stdout, and is
killed on timeout. Captured stdout is ANSI/OSC-stripped at capture so no
terminal-escape can reach a renderer or split a match; on failure a
sanitized, length-capped stdout tail is surfaced in the report so a user
sees why. output_contains are plain substrings, not regex.

Design alignment (ARCHITECTURE.md): no new payload (stays a CheckEntry in
RulesPayload; scenario-level uses turn_idx=None), no Base* signature
change, Optional fields with safe defaults (skip -> tri-state passed=None),
BELT_ALLOW_VERIFY_EXEC sourced once from envvars.py, docs updated in the
same PR. Mirrors the state_expect capture-then-assert pattern.

- schema (scenario.py): VerifySpec on Turn.verify / Scenario.verify.
- artifact (entities.py): TurnOutput.verify_result + scenario_verify_result.
- runner (orchestrator.py): execute + capture via the sandbox spawner.
- gate (envvars.py, run.py, eval.py, setup_groups.py): default-deny + flag.
- scorer (scorer/rules/verify.py): check_verify / has_verify.
- example: showcase/verify group on the tasktracker fixture (per-turn +
  per-scenario), force-included in the wheel; tasktracker pytest pythonpath
  fixed for bare-worktree runs.
- tests: schema, check_verify, rules integration, runner exec (timeout +
  ANSI strip), setup gate, sanitized failure tail; layout guard extended.
- docs: SECURITY-MODEL (T15, S5.11, gate), SCENARIOS, SCORING,
  CONFIGURATION, OUTCOMES.

Verified end-to-end: a real run edits the fixture and runs pytest in the
worktree -> verify_result {exit_code: 0, "10 passed"}, 5/5 checks.
Closes #9.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new feature Automatically generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(runner): deterministic post-turn verify step (run a command, assert exit code)

1 participant