Reproducible code and metrics for measuring exploitability and contamination in LLM evaluation harnesses.
-
Updated
Aug 6, 2026 - Python
Reproducible code and metrics for measuring exploitability and contamination in LLM evaluation harnesses.
What a benchmark scores with nothing in it: content-free answers against each task's own majority baseline, plus a judged arm. Preregistered. Closed: 45 tasks on inspect_evals 0.24.0, 1 beats its baseline outright (bbh, a task property).
Reference tasks and a tech report on catching fake deltas in agent evals: shortcut-flip + frozen verifier-recomputed control. One-command repro.
To associate your repository with the evaluation-integrity topic, visit your repo's landing page and select "manage topics."