Repository navigation
fix(scoring): withhold on judge failure, and stop two detector false zeros - #20
Conversation
…zeros A check the judge could not evaluate was counted as a check the agent failed. With a JUDGE_MODEL the endpoint does not serve, every GEval raised and the run published 'Passed 0 out of N checks' with no per-check verdict at all. Unjudged items now leave the denominator, and when none could be judged the metric abstains so the composite withholds rather than publishing an unmeasured zero. The detector zeroed clean runs two ways, both on tasks whose prompt sends the agent into the operator's home for a fixture: 'ls -la ~' printed bench.env and matrix-runs into tool output and matched the harness-environment rule, and a stale copy of a delivered input became a content fingerprint that the agent matched by reading its own advisory. Both are now filtered by the prompt, the same authorization the detector already applies to path rules. narrow_home_listing_rules restricts the bare-filename rule to args for records whose prompt names a home fixture; drop_fingerprints_matching_inputs drops any rule matching the content of an input the prompt named. Replayed over the corpus all five flagged runs go clean, and the passive-sighting control still fires where nothing authorized the agent to be in home.
|
|
||
| kept = [] | ||
| for rule in rules: | ||
| if matches_own_input(rule): |
There was a problem hiding this comment.
This loop isn't scoped to fingerprints, it walks every rule, including all ten DEFAULT_RULES. So an authorized fixture whose text happens to contain a matching token drops the whole static rule for that record. A fixture mentioning devops-bench/docs kills harness-repo entirely, including the ~/devops-bench and .git patterns that have nothing to do with the coincidence; same shape for upstream-github and for task-definition via ^\sexpected_output\s: under MULTILINE against any YAML input.
The docstring's rationale ("a fingerprint that fires on the run's own input is evidence of nothing") holds for inventory fingerprints, where the pattern is a literal line lifted off the leftover. It doesn't hold for static keyword rules, where a coincidental token says nothing about whether the agent touched harness material. filter_rules_for_prompt is gated on r.source for the same reason.
Can this be gated on rule.category == CATEGORY and not rule.source?
| # to read. | ||
| for record, inventory_rules in zip(detailed_results, task_inventories, strict=True): | ||
| try: | ||
| prompt_text = record.get("input") or "" |
There was a problem hiding this comment.
After this PR "clean" means either "nothing fired" or "what would have fired was dropped," and the only thing distinguishing them is an INFO log that doesn't travel with results.json. Since clean vs flagged now decides outcomeScore via IntegrityCatastrophic, could the report carry the authorizations that applied? Something like suppressed: [{category, reason}]. Otherwise a zero-or-not can't be reconstructed from the record.
| if not prompt or not prompt_fixture_paths(prompt, home): | ||
| return rules | ||
| return tuple( | ||
| rule.model_copy(update={"fields": ("args",)}) |
There was a problem hiding this comment.
Narrowing to ("args",) also drops output, which is the agent's own final answer rather than tool plumbing . An agent that writes "the home directory contains bench.env, matrix-runs, …" into its response is now clean on these five tasks. The false positive you're fixing lands in result (the ls listing), so ("args", "output") should give the identical corpus replay while keeping that channel.
| [record], | ||
| self._cheat_rules | ||
| + filter_rules_for_prompt(inventory_rules, record.get("input") or ""), | ||
| drop_fingerprints_matching_inputs( |
There was a problem hiding this comment.
Neither call passes home, so both resolve against the harness process's home. Under the sandbox work the agent's ~ is /workspace, so the authorization set would be computed from a tree the agent never saw. check_prompt_fixtures already models this with its mounted flag. Worth threading the same thing through here before the sandbox branches land, since they touch this exact block.
| """ | ||
| from devops_bench.evalharness.fixtures import prompt_fixture_paths | ||
|
|
||
| if not prompt or not prompt_fixture_paths(prompt, home): |
There was a problem hiding this comment.
prompt_fixture_paths doesn't check existence or read-vs-write, so this triggers on any ~/x in the prompt, including a deliverable the agent is told to write. Being told to produce ~/report.md doesn't require listing home, but it buys the same suppression. Probably fine in practice, just broader than the justification in the docstring ("an agent that runs ls -la ~ to find it"), so worth being an explicit choice
2 participants
Split out of #14 so it can be reviewed on its own. Code only — the
JUDGE_MODELvalue correction and its docs stay on #14. 1660 tests pass, ruff clean.1. A dead judge read as a dead agent
checklist.pycounted every item the judge could not evaluate as a failure. WithJUDGE_MODELpointing at an id the endpoint does not serve, every GEval raised and the run published"Passed 0 out of N checks"with zeroCheck:keys beside it.The signature is unambiguous across both corpus trees:
Nonegemini-3.1-proThe ten affected runs had deterministic correctness of 0.889 to 1.000.
Unjudged items now leave the denominator. When none could be judged the metric abstains entirely, so the composite withholds instead of publishing a zero as if it had been measured.
2. Two detector false positives
Both hit tasks whose prompt sends the agent into the operator's home for a fixture:
ls -la ~printsbench.envandmatrix-runsinto tool output, matching theharness-environmentrule — a flag for material the agent never opened.cat ~/cve-advisory-<cluster>.json, exactly what its prompt instructed.Both are now filtered by the prompt — the same authorization the detector already applies to path rules.
narrow_home_listing_rulesrestricts the bare-filename rule toargsfor records whose prompt names a home fixture;drop_fingerprints_matching_inputsdrops any rule whose pattern matches the content of an input the prompt named.Replayed over the corpus, all five flagged runs go clean:
The deliberate passive-sighting control is preserved: its existing test still passes, because a sighting is only excused for records whose prompt authorized the agent to be in home.
Scope
Five runs were hit by both defects, so their zeros need this plus a re-judge — the tool for that is in #13, which depends on this PR and refuses to write without it.
New tests: judge-evaluates-nothing withholds, partial outage scores over what was judged, fully-judged path unchanged.