feat(metrics): zero the outcome when a run cheats - #3
Open
isadominguez314 wants to merge 10 commits into
Open
Conversation
Detection has been flag-only: a run that read the benchmark's own material scored exactly as if it had not. Add an always-on, deterministic IntegrityMetric that turns a flagged cheating_report into a catastrophic zero, so the run stays on the leaderboard as a visible zero rather than disappearing from it the way validated=False would have done. The gate emits IntegrityCatastrophic rather than reusing VerificationCatastrophic: the scores map is last-write-wins, so a clean integrity check sharing that key would erase a real task catastrophic. The two keys live together in core.score_keys and are read from there by both the pipeline and the normalizer, so the row's catastrophic flag cannot drift from the zero applied to outcomeScore. A no_data report (errored run, or detection disabled) emits nothing -- having seen nothing is not innocence. The outcome finalizer no longer returns early on a gated run whose correctness sources all abstained, because a null outcomeScore drops the row out of leaderboard aggregatebecause a null outcomeScore drops the row out of la so it is neverbecause a null outcomeScore drops the row out of leaderboard aggregatebeonbecause a null outcomeScore drops the row out of leaderboard aggregatebt its result is now consulted by scoring.
There was a problem hiding this comment.
🟡 Changes recommended
Judge initialization failures and malformed flagged reports can bypass the intended integrity zero.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds an integrity metric that converts detected cheating into a visible zero outcome.
Changes:
- Adds and registers
IntegrityCatastrophic. - Integrates integrity gates into outcome scoring and normalization.
- Adds tests and documentation for integrity scoring.
File summaries
| File | Description |
|---|---|
devops_bench/core/score_keys.py |
Defines shared catastrophic score keys. |
devops_bench/detection/detector.py |
Documents scoring consequences. |
devops_bench/evalharness/default.py |
Connects reports to later scoring. |
devops_bench/metrics/integrity.py |
Implements the integrity metric. |
devops_bench/metrics/pipeline.py |
Applies integrity gates to outcomes. |
devops_bench/results/normalize.py |
Normalizes all catastrophic gates. |
devops_bench/results/row.py |
Updates catastrophic-field documentation. |
docs/components/detection.md |
Documents detection and limitations. |
docs/components/metrics.md |
Documents integrity scoring. |
tests/unit/evalharness/test_default_harness.py |
Updates detector contract coverage. |
tests/unit/metrics/test_metrics_integrity.py |
Tests integrity verdict handling. |
tests/unit/metrics/test_metrics_pipeline.py |
Tests outcome gating behavior. |
tests/unit/results/test_results_normalize.py |
Tests normalized integrity flags. |
Review details
- Files reviewed: 13/13 changed files
- Comments generated: 2
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Two fail-open paths let a flagged run keep a passing score. _score built the judge before running any metric and let the failure propagate, so a bad JUDGE_PROVIDER or missing key aborted scoring for the whole batch -- including the deterministic catastrophic gates, which need no judge. It now falls back to a null judge and scores what it can. This was live rather than theoretical: get_judge_model() raises for lack of an OPENAI_API_KEY under test, and the harness test had the resulting empty scores map frozen in as an assertion. _reason type-checked the elements of cheating_report.categories but not the container, so a persisted non-list raised on iteration; the pipeline's per-metric guard swallowed that and dropped the gate entirely.
jessie1111101
left a comment
jessie1111101
left a comment
There was a problem hiding this comment.
Ran this against 8 published run records: all clean, every OutcomeScore
unchanged. Gate is inert on honest runs. 225 tests pass, ruff clean.
One concern inline on integrity.py. Separate-key design is right, and sharing
CATASTROPHIC_SCORE_KEYS between pipeline and normalize is the correct call.
PR body is corrupted from "leaderboard aggregate" onward — a clause repeats four
times and truncates mid-word. Worth fixing before it's the merge commit message.
# Conflicts: # docs/components/detection.md
Orphaned when the catastrophic flag moved to _CATASTROPHIC_KEYS; nothing imports it from this module (metrics/verification.py's copy is separate and still live). Flagged in review on PR #3.
…itive override Review on PR #3 asked for both to be explicit: emitting nothing on no_data means no gate, so its OutcomeScore matches a clean run's and the distinction lives only in the per-metric map; and a wrongly flagged record can only be overturned today by hand-editing its stored cheating_report, since the deterministic gate re-fires on rescore and BENCH_CHEAT_DETECT is all-or-nothing at construction.
itssimrank
reviewed
Sep 2, 2026
itssimrank
reviewed
Sep 2, 2026
itssimrank
approved these changes
Sep 2, 2026
…on point Review on PR #3 asked for the implications of the shared tuple: a key added there automatically zeroes OutcomeScore and flips the row's catastrophic flag with no further wiring, and everything in it must be deterministic because the pipeline applies these gates without a judge.
# Conflicts: # docs/components/cheat-detection.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detection has been flag-only: a run that read the benchmark's own material scored exactly as if it had not. Add an always-on, deterministic IntegrityMetric that turns a flagged cheating_report into a catastrophic zero, so the run stays on the leaderboard as a visible zero rather than disappearing from it the way validated=False would have done.
The gate emits IntegrityCatastrophic rather than reusing VerificationCatastrophic: the scores map is last-write-wins, so a clean integrity check sharing that key would erase a real task catastrophic. The two keys live together in core.score_keys and are read from there by both the pipeline and the normalizer, so the row's catastrophic flag cannot drift from the zero applied to outcomeScore.
A no_data report (errored run, or detection disabled) emits nothing -- having seen nothing is not innocence. The outcome finalizer no longer returns early on a gated run whose correctness sources all abstained, because a null outcomeScore drops the row out of leaderboard aggregates -- exactly the erasure a visible zero exists to prevent -- so outcomeScore is never left null once a gate has fired: the composite takes a synthesized correctness of 0.0 and the reason string reports c=n/a. Detection itself stays flag-only and never mutates a record; but its report is now consulted by scoring.