feat(benchmark): verified-retry orchestrator — production-legal best-of-N with a layered verifier

[eric-liu-nvidia]

What

A new standalone module benchmark/verified_retry/ implementing a best-of-N retry harness whose pass/fail decisions come from its own layered verifier — no benchmark answer key anywhere in the loop:

1. Executable checks, derived blind (checks.py) — an LLM writes a check script from the task statement before any attempt exists (prevents inheriting a solution's misreading of ambiguous requirements), and the script is executed in the attempt's workspace. Execution outranks opinion.
2. Confidence-gated LLM judge (judge.py::judge_attempt) — absolute verdict on demonstrated evidence, accepted only at confidence ≥ 0.85; acts as a veto over passing checks and as the sole verifier when nothing is machine-checkable. Unparseable verdicts fail closed.
3. Comparative pick (judge.py::judge_compare) — when no attempt verifies within budget, picks the most-likely-correct candidate from archived attempts; result is explicitly flagged accepted=False. Skipped when only one candidate exists.

The orchestrator (orchestrator.py) runs up to max_attempts executor attempts, each in a pristine copy of the task workspace (attempt independence), stops at the first verified success, and archives every attempt.

Why (measured motivation)

From a TB-2.1 study with Kimi-K3-Max under the Claude Code harness (4 replicate runs per config):

• Retries are the dominant accuracy lever: single-attempt 70.5% vs oracle pass@4 83.1%; no advisor/prompt configuration moved the single-attempt mean (all arms 61–67/89, within the ±4–5 task per-run noise).
• An LLM judge alone is not verifier-strength: on 120 blind verdicts vs hidden ground truth it scored 82.5% accuracy with a 35.6% false-pass rate — every false-pass came from trusting narrated rather than executed evidence.
• Judge confidence separates truth from error (true-pass mean 0.83 vs false-pass 0.69); gating at ≥0.85 removed 14/16 false-passes while keeping 43/70 true passes.
• Comparative picking is far stronger than absolute judging: 87% correct picks (26/30) vs 62.5% random — implying ~78.7%/89 for judge-selected best-of-4, ≈ a frontier model's single-attempt score with no oracle.
• Early stopping makes the economics work: measured solve distributions give ~1.7 expected attempts per task at N=4 (~43% of naive 4× cost).

Design boundaries

• Not a Switchyard route/strategy: the gateway never touches the workspace, and executing checks requires the workspace — this wraps the executor where the sandbox lives.
• Executor and LLM are injected callables (ExecutorFn, LlmFn): the core is stdlib-only, no provider SDK, no docker, no network.
• Scores produced through this harness are pass@N-with-own-verifier and must be reported as such, never as single-attempt accuracy (README states this explicitly).

Testing

uv run pytest benchmark/verified_retry/tests -q — 10 tests, all passing: early stop on first verified success, retry-until-checks-pass, exhausted-budget comparative pick, crash never accepted, low-confidence veto, judge-only fallback when nothing is checkable, workspace-pristineness assertion per attempt, fail-closed on unparseable verdicts, single-candidate compare skip.

Follow-ups (not in this PR)

• Production LlmFn adapter (Anthropic-format client) and a container-backed ExecutorFn for real agent runs (needs a docker-capable host).
• Integration run against TB task images to measure the harness end-to-end (legal: checks derive from public statements only).
• Optional: wire as a harbor custom agent for platform-side evaluation.

🤖 Generated with Claude Code