feat(cli): coder-eval execute + detached grading via evaluate <run_dir>

[akshaylive]

Splits running from grading, so an external harness can own the verdict — and closes the loop so a run executed now can be graded later.

The motivating case is Harbor (Terminal-Bench 2.0), which builds its own container, calls coder-eval as the agent, and grades with its own tests/test.sh. Grading twice there would be worse than not grading at all: coder-eval's verdict would be reported alongside Harbor's without being the one that counts.

This is Part A, phases 1–5 of the Harbor interop plan (tmp/harborframework.md).

coder-eval execute  tasks/hello.yaml --run-dir ./r   # run, capture, score nothing
coder-eval evaluate ./r/default/hello/00             # supply the verdict later
coder-eval aggregate ./r                             # run.json now reports it

---

1. coder-eval execute

coder-eval run with the grading half removed. The agent runs and the full trajectory lands in task.json as usual, but no criterion is checked, weighted_score stays None, and the row finalizes as the new FinalStatus.NOT_GRADED.

NOT_GRADED is a fourth reporting category

category == "ungraded" — not a fold into the existing three. Folding into failed would depress every pass rate, into succeeded would invent verdicts, into error would report a healthy run as broken.

Ungraded rows leave both sides of every rate: RunSummary / VariantAggregate pass_rate and error_share now divide by tasks_graded (tasks_run - tasks_not_graded), identical to tasks_run for any graded run. tasks_not_graded is part of the sum-to-tasks_run invariant, not a tasks_failed sub-counter, and is defaulted so existing run.json / experiment.json still parse.

weighted_score is set to None explicitly rather than left to calculate_weighted_score, which writes 0.0 for an empty results list — indistinguishable from "graded and scored zero", and every downstream score or 0.0 would launder it into a real-looking failure.

Only SUCCESS/FAILURE collapse into it. ERROR, TIMEOUT, BUILD_FAILED, MAX_TURNS_EXHAUSTED and the budget stops are facts about the run, not about grading — they still apply, and execute still exits non-zero on a crash.

The switch

BatchRunConfig.grade → Orchestrator(grade=...), gating all four grading call sites. It crosses the docker boundary in context.json, defaulting to True in-container so a host predating execute keeps grading.

Deliberately not a task-config field: no 5-layer merge, no -D path. A task YAML must never declare itself ungraded; only the invoking command decides.

run and execute share one body (run_pipeline) — no third code path. Only the Typer signature is restated, and a test keeps the two option sets in step.

Refused rather than degraded

Not supported	Why
--junit-xml	A report of verdicts, and there are none. (reports_junit still emits <skipped> for an ungraded row met elsewhere.)
Simulation tasks	The dialog loop reads criteria results to decide whether to keep talking; an ungraded dialog would silently change its own stopping behavior.
stop_early:	Goes inert — it exists to cut a run once the criteria decide, and here the full trajectory is the deliverable.

--resume is supported — see part 4.

---

2. Detached grading — evaluate <run_dir>

evaluate now takes two shapes, told apart by a pure resolver (cli/evaluate_target.py) on one probe: a target holding task.json is a run directory. Passing a task file over a run directory re-grades it with different criteria, reusing the trajectory and workspace of a run you already paid for.

A re-grade must describe the run that happened

Run-dir mode rebuilds the task from the run's own task_config.resolved, not by re-reading the YAML. resolved is post-merge, so variant overrides, -D flags and dataset row expansion are already baked in — re-loading the source would silently grade a different task. Fallback to source_file happens only when resolved no longer validates, and says so loudly.

Orchestrator(prior_result=...) seeds the fresh result, carrying:

• the trajectory — every derived figure (tokens, cost, command_stats, model_used) recomputes from iterations, so seeding it reproduces them exactly;
• iteration_count, which evaluate-only used to flatten to 1;
• early_stop — load-bearing. Gate selection is FIRED-ONLY: when set, the checker gates on the weighted armed subset instead of strict-AND. Dropping it would re-grade a truncated trajectory under the full-run gate and flip the verdict;
• execution facts (max_turns_exhausted, error_message, sdk_options).

Grader-host environment_info is preserved under a graded_by sub-dict rather than overwriting the run's.

Two further parity fixes, both closing gaps the code already knew about:

• command_base_path is now persisted and restored in the evaluate-only branch. _sync_sandbox_command_path_with_agent's docstring already named "evaluate-only mode" as a known PATH gap; without this a detached grade resolves run_command binaries against ambient PATH and can disagree with the run it grades.
• _join_litellm_actual_cost skips when prior_result is set. It keys on a per-Orchestrator nonce the prior turns never carried, so it would match nothing and overwrite already-correct per-turn costs.

A re-grade refuses outright on a reference_digest mismatch — grading then would score the agent's old work against a new answer key.

The verdict is written back into the run's task.json, keeping the pre-grade record as task.execute.json. That in-place write is what makes plain coder-eval aggregate <run_dir> rebuild a graded run.json with zero new code.

---

3. Sandbox.adopt — and the pre-existing bug it fixes

adopt(workspace) reuses setup's adoption half but skips every materializing step (_setup_template, _generate_cli_recorders, venv/package installs, the destructive $HOME remediation), running only non-mutating derivation: mock-dir +x, venv discovery, plugin-tools pin. _cleanup_on_exit stays False, so an adopted tree is never moved or deleted.

In-place is more correct, not merely faster. _setup_template filters its copy through _should_ignore_template_file, whose default list drops node_modules, dist, build, .venv, .git. So evaluate today scores a file that is plainly there as missing:

copy path:  Score 0.00   "File 'node_modules/x/a.js' does not exist"
in place:   Score 1.00   "File 'node_modules/x/a.js' exists"

That is a defect independent of execute — it breaks grading for any task that builds something.

Defaults: in-place for a run directory (it is the run's own output), copy for a bare work directory (criteria can mutate it and it is the user's tree). --in-place / --copy override. adopt hard-errors on driver: docker: a container workspace is unreachable from the host, so adopting one would grade whatever happens to sit at that host path.

---

4. --resume now distinguishes "executed" from "graded"

--resume decided a task was finished by asking "does task.json carry any final_status". NOT_GRADED is a final status, so run --resume over an executed run reported the tasks complete, graded nothing, and exited 0:

after execute:            NOT_GRADED
$ coder-eval run --run-dir tmp/res --resume
↻ Resume: 1 task(s) already complete, running 0 remaining
Results: 1/1 executed, not graded
real exit code: 0
after run --resume:       NOT_GRADED

"Finished" is relative to the resuming command. partition_for_resume(tasks, *, grade) returns a four-way ResumePartition:

On disk	run --resume	execute --resume
No task.json, unreadable, or no final_status	re-run	re-run
NOT_GRADED	grade in place	already complete
Any other status, incl. FAILURE / ERROR	already complete	already complete

A NOT_GRADED row owes execute nothing but owes run a grade, so run --resume runs the criteria against the trajectory and workspace already on disk rather than paying for the agent twice — the entire reason the two commands are separate.

The carve-out is only for NOT_GRADED. Resume has never retried failures, and a parametrized test pins that so this cannot grow into a general "retry bad rows" rule. clear_rerun_artifacts skips to_grade, whose artifacts are the very thing being graded. A per-task grading failure is warned and folded back in with its original ungraded result, so one bad row neither aborts the resume nor vanishes from run.json.

grade is exempt from the config-drift warning: execute → run --resume is a supported flow, and that warning's "already-finalized tasks keep their original-config results" text is actively wrong for it. execute --resume is consequently supported and no longer refused.

orchestration/regrade.py is the single implementation shared by this path and evaluate's run-dir mode — two copies of "how to re-grade" would drift into two verdicts for the same run.

A fidelity bug the test caught

A re-graded row was reporting the grading pass's clock. A 10-minute agent run re-graded in 2 seconds would record 2 seconds — and duration_seconds feeds average_duration, the report tables and the evalboard, so harness comparisons would have been quietly wrong. A task row describes the task, so it now keeps the agent run's started_at and duration_seconds; the grading cost is preserved separately as environment_info["grading_duration_seconds"].

---

Ripple

The explicit-mapping guards did their job — every surface below failed loudly rather than silently mis-bucketing the new status: pyright on reports_junit._category_of, the _status_badge category tests, the published-action "every FinalStatus must be classified" test, and CE018's enum-parity check.

• reports_junit — ungraded → <skipped> (already counted by _set_counts).
• reports_html — neutral badge; the "no member falls through to neutral" guard now allows it for ungraded only.
• reports / reports_experiment — a Not Graded line; the pass rate reads n/a for a fully ungraded run instead of 0.0%. An ordinary empty run keeps its original 0/0 rendering — different facts.
• experiment aggregation — average_score means over graded rows only; _pick_worst_status ranks ungraded least-urgent so any real verdict wins.
• verify-published-action.yml — NOT_GRADED hard-fails. That job runs the published action, which always grades, so reaching it means the action dispatches the wrong command and every score gate in the job is measuring nothing.
• evalboard statusCategory — NOT_GRADED → "unknown", the category every consumer already treats as "no verdict here".

Also: evaluate's Typer command is now a thin wrapper over run_evaluation(...) with real Python defaults — the same split run/execute use. Calling a Typer command in-process hands unspecified options an OptionInfo sentinel, which silently made in_place=None truthy.

---

Verification

make verify green (4612 passed, 92.07%) and make evalboard-verify green (608 tests).

The headline test asserts execute + evaluate reaches the same status, score and per-criterion results as a single run — compared against a real run rather than hardcoded values, so a change breaking both paths still fails. Alongside it:

• an end-to-end execute that asserts pre_run's file is written, so a merely-skipped task cannot pass;
• a negative control proving run still scores that same task 1.0;
• aggregate rebuilds a graded run.json unaided; the trajectory survives the re-grade; the adopted workspace is not moved or deleted; task.execute.json preserves the ungraded record;
• adopt writes nothing, deletes nothing, and exposes the filtered directories;
• the target resolver, table-tested over every (one arg / two args) × (run dir / plain dir / file / missing) combination;
• the docker context.json round-trip, and run/execute signature parity.

Scoped out

Relaxing the non-empty success_criteria validator. execute on an existing task YAML needs no such change; it is only needed for a foreign task format with no criteria to declare, and belongs with that work.

🤖 Generated with Claude Code

[uipreliga]

Review: coder_eval — pr:154

Scope: pr:154 · branch akshaya/coder_eval_execute · 9e479ae · 2026-09-09T04:29Z · workflow variant

Change class: complex — introduces a new execute command, a detached-grading path over run dirs (evaluate <run_dir>), a NOT_GRADED FinalStatus with a fourth ungraded rate bucket, Sandbox.adopt, in-place regrade, and a four-way resume partition; correctness requires reasoning about control flow, status semantics, trust boundaries, and score-affecting metric denominators

The detached-grading design is sound and the security posture is deliberate (7.7/10 overall, no criticals, an explicit untrusted-run-dir threat model, and a real "None, never 0.0" ungraded contract), but a small cluster of defects still lets identical agent output receive a different score or final_status depending on which command graded it — a re-graded MAX_TURNS_EXHAUSTED row pinned at exit 1 while holding score 1.000, a symlinked artifacts/ that escapes the run directory and grades a foreign tree, a reference_digest anti-cheat write that is wiped before it reaches disk, and fabricated 0.0 pass rates on three surfaces — and the weakest axis, Test Health at 6/10 (a surviving mutant on the docker grade guard, zero coverage on the evaluate <task.yaml> <run_dir> shape), is exactly what let them ship; fix the four verdict-changing defects, then pin each one with a test.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	7.6 / 10	0	1	2	4	orchestrator.py keeps absorbing the detached-grade/grading-switch concern: god class + three functions past the complexity bar
2. Type Safety	8.8 / 10	0	1	0	2	TaskTrend.passRate: number cannot express "never measured": with the new isGraded denominator gate (trends.ts:158), a fully-ungraded task falls into the : 0 fallback at trends.ts:196 and publishes a fabricated 0% that sorts to the top of Trends
3. Test Health	6 / 10	0	2	4	0	_assert_grade_honored's evidence check is an unpinned surviving mutant, and its on-disk quarantine has zero coverage
4. Security	8.8 / 10	0	0	2	2	Grading workspace escapes the run directory: default_workspace roots its containment check at the attacker-supplied artifacts/ dir and does not check the single-child / flat-artifacts returns at all
5. Architecture & Design	7.4 / 10	0	2	1	1	evaluate <run_dir> --copy writes the grading copy's sandbox_path back into the run's task.json, breaking every later re-grade
6. Error Handling & Resilience	8 / 10	0	1	2	0	reference_digest is written during _stage_reference and then wiped by the wholesale environment_info reassignment, so verify_reference_unchanged is still a permanent no-op on every real run
7. API Surface & Maintainability	8.2 / 10	0	1	1	3	_owes_a_grade routes docker-death BUILD_FAILED/ERROR rows into to_grade, overwriting the container diagnostic and contradicting the resume contract
8. Evaluation Harness Quality	6.8 / 10	0	2	2	2	_nothing_was_measured misses the failed-category execution facts, so an execute night with one TIMEOUT or budget-stopped row publishes a fabricated pass_rate = 0.0

Overall Score: 7.7 / 10 · Weakest Axis: Test Health at 6 / 10
Totals: 🔴 0 · 🟠 10 · 🟡 14 · 🔵 14 across 8 axes.

Blockers

1. [Axis 1] orchestrator.py keeps absorbing the detached-grade/grading-switch concern: god class + three functions past the complexity bar (src/coder_eval/orchestrator.py:1033) — Verified by diffing radon cc -s between origin/main and pr-154. Members of this cluster, all worsened by this diff:

• src/coder_eval/orchestrator.py:1033 Orchestrator._finalize_result — D (26) → D (28). The new branches are if self.prior_result is not None: ... environment_info["grading_duration_seconds"] = ... plus the if self.grade: calculate_weighted_score(...) else: self.result.weighted_score = None fork inside the already-wrapped try.
• src/coder_eval/orchestrator.py:1468 Orchestrator._setup — C (19) → D (22), i.e. this PR is what pushed it across the anchor's >20 in a hot module line. Added: the if self.grade: ... else: logger.info(...) early-stop fork, and the restored_path = self.result.environment_info.get("command_base_path"); if isinstance(restored_path, str) and restored_path: PATH-restore block.
• src/coder_eval/orchestrator.py:3122 Orchestrator._cleanup — C (19) → C (20), from the new if self.sandbox.was_adopted: arm inside the PreservationMode.NONE branch.
• (enumerated, lower-severity member) src/coder_eval/cli/run_command.py:887 _run_with_experiment — C (19) → D (24), from the if not grade: simulation rejection block and the if not grade: return summary, 0 rollup skip.

Credit where due: the same diff reduced _evaluation_loop 26→23 and run() 18→16 by extracting _select_gate() and _terminal_status(), and left _simulation_dialog_loop E(40) and _check_run_limits D(24) untouched. Apply the same extraction treatment to the three regressions: lift the prior_result timing/score fork out of _finalize_result into a _finalize_regrade_timing() sibling of _seed_from_prior_result; lift _setup's two new blocks into _arm_early_stop() and _restore_recorded_command_path(); and in _run_with_experiment move the if not grade: simulation rejection next to the other resolution-time validations.
2. [Axis 2] TaskTrend.passRate: number cannot express "never measured": with the new isGraded denominator gate (trends.ts:158), a fully-ungraded task falls into the : 0 fallback at trends.ts:196 and publishes a fabricated 0% that sorts to the top of Trends (evalboard/lib/trends.ts:196) — The PR made the : 0 fallback reachable for the first time. evalboard/lib/trends.ts:196 reads passRate: b.totalCount > 0 ? b.successCount / b.totalCount : 0, while line 155 now gates the denominator — if (isGraded(t.status)) b.totalCount += 1;. Before this change b.totalCount += 1 was unconditional, so a task that appeared at all had totalCount >= 1 and the : 0 arm was dead; now a task whose every row in the window is NOT_GRADED gets totalCount === 0 and passRate === 0. The type is the hole: evalboard/lib/trends.ts:29 declares passRate: number; // 0-1, with no way to say "unmeasured", which is exactly why the code coalesces. The same PR got this right twice elsewhere — evalboard/lib/overview.ts:867 declares passRate: number | null; and evalboard/lib/watchlist.ts:228 added if (outcomes === 0) continue; with a comment citing CE049 ("never coalesce a possibly-unmeasured score to a numeric literal") — so trends.ts is the one site missed. Consequence at the view: evalboard/app/trends/trends-view.tsx:492 renders {fmtPct(t.passRate)} through fmtPct(p: number) (line 43-45, Math.round(p * 100)) → literal "0%", and the table's default sort is useState<SortKey>("passRate") ascending (lines 607-608), whose comparator at 112-116 is a.passRate - b.passRate — so the unmeasured task lands at the very top of the "worst offenders first" list. The PR's own test admits it: evalboard/lib/__tests__/trends.test.ts adds test("a fully ungraded task reports no runs rather than a 0% pass rate") but asserts only totalRuns/successRuns and pointedly never asserts passRate. Fix: widen TaskTrend.passRate to number | null, emit null when totalCount === 0, render it through a null-aware formatter (the file's own fmtScore(s: number | null) at line 47-49 already returns "—"), and sort it with the existing cmpNullable helper used for the other nullable columns.
3. [Axis 3] _assert_grade_honored's evidence check is an unpinned surviving mutant, and its on-disk quarantine has zero coverage (tests/test_detached_grading_boundaries.py:131) — TestDockerGradeBoundary (tests/test_detached_grading_boundaries.py:131-153) is the only test of the guard that stops a stale docker image publishing verdicts under execute, and it pins neither half of the fix the guard's own docstring documents.

(a) The evidence check is dead weight under the current suite. docker_runner.py:910 reads graded_anyway = bool(result.success_criteria_results) or result.weighted_score is not None, whose stated purpose (docker_runner.py:905-909) is that "a stale image return[ed] a fully graded MAX_TURNS_EXHAUSTED row — criteria vector, weighted score and all — unchallenged". Every fixture uses _result(...) (line 60-69), which sets neither success_criteria_results nor weighted_score, so graded_anyway is False in all four tests. I verified this by mutation: replacing line 910 with graded_anyway = False and running the full suite gives 4805 passed, 4 skipped — the fixed defect can be reintroduced with no failing test. Add a case with an execution-fact status AND a criteria vector, e.g. _result(FinalStatus.MAX_TURNS_EXHAUSTED) with success_criteria_results=[CriterionResult(...)] (or weighted_score=1.0), asserting DockerRunError is raised.

(b) The quarantine block (docker_runner.py:915-923, sidecar = task_json.with_suffix(task_json.suffix + ".graded") / os.replace(task_json, sidecar)) is 0% covered — confirmed by --cov-report=term-missing, which lists 918-923 as missing. All four tests call runner_._assert_grade_honored(_result(...)) with task_json left at its None default, so the branch never executes. Per the same docstring, refusing in memory while leaving the graded task.json on disk lets a later execute --resume / aggregate fold in "exactly the row this guard declined to publish". Add a test that writes a real task.json, passes its path, and asserts the file is renamed to task.json.graded before the raise.
4. [Axis 3] evaluate <task.yaml> <run_dir> — one of the command's three documented shapes — has no behavioural test (src/coder_eval/cli/evaluate_command.py:146) — The explicit-task-file-over-a-run-directory form is advertised in evaluate's own help text (evaluate_command.py:313-316: "Iterate on criteria against a run you already paid for, by passing a task file over a run directory") and called "the main reason to keep execute and evaluate separate at all" in evaluate_target.py:92-96. Its CLI branch is never executed by the suite: --cov-report=term-missing lists evaluate_command.py 146-147 as missing, i.e. neither

task, source_yaml = load_task(target.task_file)
console.print(f"[dim]Grading with {target.task_file} (overrides the run's recorded config).[/dim]")

nor anything downstream of them ever runs. tests/test_evaluate_target.py::test_task_file_plus_run_dir_re_grades_with_the_given_task only asserts the pure resolver returns mode is EvaluateMode.RUN_DIR with task_file set — it never invokes the command, so nothing asserts that the supplied criteria actually win over the recorded ones, that the run's trajectory and workspace are still used, or that this form deliberately bypasses check_embedded_commands (the --allow-recorded-commands gate is only reached in the else arm at line 160). Add an end-to-end test alongside the others in tests/test_execute_evaluate_loop.py: execute the agentless task, write an edited task YAML whose criteria differ (e.g. a file the run did not produce), run evaluate <edited.yaml> <run_dir> --run-dir <tmp>, and assert the verdict follows the edited file and graded["iterations"] still matches the executed row.
5. [Axis 5] evaluate <run_dir> --copy writes the grading copy's sandbox_path back into the run's task.json, breaking every later re-grade (src/coder_eval/cli/evaluate_command.py:539) — _seed_from_prior_result deliberately carries the run's artifacts pointer — orchestrator.py:842 self.result.sandbox_path = prior.sandbox_path, documented as "a SECOND grade needs it, since without it the caller falls back to guessing the workspace" — and tests/test_seed_from_prior_result.py:58 lists sandbox_path under CARRIED. The --copy branch then overwrites it twice: orchestrator.py:1508 self.result.sandbox_path = str(self.sandbox.sandbox_dir) (the grading tempdir), and cleanup at orchestrator.py:3172-3174 (preserved_path = await asyncio.to_thread(self.sandbox.preserve_to, artifacts_dir) / self.result.sandbox_path = str(preserved_path)) under the default PreservationMode.MOVE_ON_WRITE chosen at evaluate_command.py:418, or self.result.sandbox_path = None at orchestrator.py:3193 under --no-preserve. That result is then written into the ORIGINAL run directory by evaluate_command.py:539 _write_back(target.target, result). After one evaluate <run_dir> --copy, the run's task.json points at the new run dir's artifacts (or at nothing), so the next evaluate <run_dir> hits orchestration/regrade.py::default_workspace's containment guard — raise RegradeError(f"The recorded sandbox_path ({recorded}) is outside the run directory ({run_dir}). Pass --workspace explicitly to grade it.") — and run --resume fails the same row. Fix: restore the carried sandbox_path before write-back whenever prior_result is set and the sandbox was not adopted (i.e. make the copy path leave the run's artifacts pointer alone, the same way _finalize_result already restores duration_seconds at orchestrator.py:1053), and add a test that evaluate <run_dir> --copy followed by evaluate <run_dir> still resolves the workspace.
6. [Axis 5] The ungraded-denominator guard was applied to RunSummary only — VariantAggregate (and SuiteRollup) still publish a fabricated pass_rate = 0.0 against their own documented contract (src/coder_eval/models/experiment.py:285) — RunSummary got a guard for the exact case its own docstring describes (results.py:1173-1188 _nothing_was_measured — "a 100-task execute night with 5 crashes published pass_rate 0.0 and error_share 1.0 — a measured-looking total failure for a run that was never measured at all"), but the two parallel rate models added in the same PR did not. models/experiment.py:285 is return self.tasks_succeeded / self.tasks_graded if self.tasks_graded else None under a docstring that claims "Mirrors RunSummary.pass_rate", and reports.py:1008 is pass_rate=rows_passed / rows_graded if rows_graded else None,. Verified numerically on this checkout for a 10-task execute run with 1 crash (tasks_run=10, tasks_error=1, tasks_not_graded=9): RunSummary.pass_rate = None, VariantAggregate.pass_rate = 0.0, SuiteRollup.pass_rate = 0.0. reports_experiment.py:622 then renders pass_rate_str = f"{agg.pass_rate * 100:.1f}%" → "Pass Rate: 0.0% (0/1)" for a run that measured nothing, while reports.py:255 prints n/a for the same run. Fix: lift _nothing_was_measured to a shared helper (it is a pure function of succeeded/failed/not_graded) and apply it in all three — VariantAggregate.pass_rate, RunSummary.pass_rate/error_share, and the SuiteRollup construction in reports.py::_build_suite_rollup — with a test asserting the three agree on the same bucket counts.
7. [Axis 6] reference_digest is written during _stage_reference and then wiped by the wholesale environment_info reassignment, so verify_reference_unchanged is still a permanent no-op on every real run (src/coder_eval/orchestrator.py:1614) — _stage_reference records the answer-key hash at orchestrator.py:1418 — self.result.environment_info["reference_digest"] = self._reference_digest — but _setup() runs _stage_reference() at line 1502 and then, on the agent path, REPLACES the whole dict at lines 1613-1616:

# Re-capture environment_info with sandbox path (for CLAUDE.md hash)
self.result.environment_info = get_version_info(
    sandbox_path=Path(self.result.sandbox_path) if self.result.sandbox_path else None,
)

get_version_info builds a fresh dict (utils.py:451 version_info = {}), so the key never reaches task.json. Verified empirically: a NoOpAgent end-to-end run of a task carrying reference: {directory: reference} finishes with 'reference_digest' not in orch.result.environment_info (the assertion fails against the full env dict: {'anthropic': ..., 'api_routing': 'anthropic_direct', 'claude_code_cli': ..., 'cli_version': ...}). Consequence: orchestration/regrade.py:320 recorded = prior.environment_info.get("reference_digest") is always None, so the guard takes its if not isinstance(recorded, str) early return at lines 321-328 and only logs "This run recorded no reference_digest, so the answer key cannot be verified". A reference edited between execute and evaluate <run_dir> (or before run --resume grades the row) goes undetected and the row is scored against a different answer key, reported as an ordinary verdict. This is exactly the defect CE047 and CLAUDE.md claim this PR closed ("the reference_digest anti-cheat guard shipped as a read with no writer anywhere"): CE047 is satisfied because a write now exists in src/, but the write is dead. Fix: merge instead of replace — e.g. self.result.environment_info = {**get_version_info(sandbox_path=...), **{k: v for k, v in self.result.environment_info.items() if k == "reference_digest"}}, or move the digest write to after line 1616. Add a round-trip test that runs a reference-carrying task end-to-end and asserts "reference_digest" in result.environment_info — tests/test_detached_grading_boundaries.py:434 only tests _staged_digest in isolation and never asserts the key survives a run, which is why the wipe shipped.
8. [Axis 7] _owes_a_grade routes docker-death BUILD_FAILED/ERROR rows into to_grade, overwriting the container diagnostic and contradicting the resume contract (src/coder_eval/orchestration/batch.py:354) — _owes_a_grade keys on evidence, not on NOT_GRADED:

return result.weighted_score is None and not result.success_criteria_results

That is broader than the shipped contract. docs/USER_GUIDE.md:117 says | Any other status, **including FAILURE/ERROR** | already complete | already complete |, and partition_for_resume's own docstring at batch.py:385 says "FAILURE and ERROR rows that DO carry a verdict stay complete under both commands" — but the rows written by isolation/docker_runner.py::_write_synthetic_task_json (via build_error_result, docker_runner.py:1647-1660) carry NO verdict: weighted_score defaults to None and success_criteria_results is empty. I confirmed this by construction in the PR worktree: _owes_a_grade(build_error_result-shaped ERROR) → True, and the same for BUILD_FAILED. So every container that died before writing task.json, and every failed image build, is now routed to to_grade on a plain coder-eval run --resume — a path that previously skipped them.

The consequence is diagnostic loss. In src/coder_eval/cli/run_command.py:810 the fold-back does:

                result = prior
                result.error_message = f"Grading failed during --resume: {e}"

The comment two lines above justifies this by asserting "the folded-back result keeps the execute phase's empty error_message" — true for a NOT_GRADED row, false for these. prior.error_message here is "Container exited with code 137 without producing task.json. See <log> for container output." (docker_runner.py:864-866), and it is replaced by "Grading failed during --resume: No workspace to grade: …artifacts does not exist and the recorded sandbox_path (unset) is gone. The run was probably made with --preservation-mode NONE." — a message that names the wrong cause. Because regrade.default_workspace raises before back_up_pre_grade_record is reached, the on-disk task.json keeps the original text while run.json publishes the replacement, so the two records for the same row disagree.

Fix: either (a) narrow the routing test so a row is only owed a grade when it was executed AND is ungraded — e.g. result.final_status is FinalStatus.NOT_GRADED or (result.final_status.is_execution_fact and result.iteration_count > 0) — or, more simply, exclude BUILD_FAILED and any row with iteration_count == 0 (no agent phase ever ran, so there is nothing to grade); and (b) append rather than replace in run_command.py:810 (result.error_message = f"{prior.error_message or ''}\nGrading failed during --resume: {e}".strip()) so a pre-existing cause is never destroyed. Then reconcile docs/USER_GUIDE.md:117 with whichever rule ships — today the table and the code state different contracts.
9. [Axis 8] _nothing_was_measured misses the failed-category execution facts, so an execute night with one TIMEOUT or budget-stopped row publishes a fabricated pass_rate = 0.0 (src/coder_eval/models/results.py:1188) — The new guard is return self.tasks_not_graded > 0 and (self.tasks_succeeded + self.tasks_failed) == 0 (results.py:1188). It treats a non-zero tasks_failed as evidence that something was measured — but under coder-eval execute TIMEOUT, TOKEN_BUDGET_EXCEEDED and COST_BUDGET_EXCEEDED are all category failed (models/enums.py:60-65) and all reachable: _terminal_status returns NOT_GRADED before the max-turns arm, but _check_run_limits(iteration=iteration) is still called on the ungraded branch (orchestrator.py:681) and task_timeout still fires. Verified by execution:

execute night with 1 TIMEOUT -> pass_rate= 0.0 error_share= 0.0 tasks_graded= 1
execute night with 5 ERROR   -> pass_rate= None error_share= None

(RunSummary(tasks_run=100, tasks_succeeded=0, tasks_failed=1, tasks_error=0, tasks_not_graded=99)). This is the exact defect the docstring above it claims to close ("a measured-looking total failure for a run that was never measured at all, and a real 0% point on the evalboard trend") — it just fixed the ERROR spelling and not the failed one. Blast radius reaches the persisted contract and the cross-repo consumers: reports.py::_pass_rate_lines guards on if summary.tasks_not_graded and summary.pass_rate is None, so run.md renders - **Pass Rate**: 0.0% (0/1); run.json serializes pass_rate: 0.0; and evalboard/lib/overview.ts:809 computes successRate from tasksSucceeded/tasksGraded, i.e. a real 0% trend point. Fix: base the test on the evidence the docstring names — no row produced a criteria verdict — e.g. count rows whose weighted_score is not None or whose success_criteria_results are non-empty, or at minimum exclude the execution-fact statuses from the tasks_failed term. Apply the same rule to SuiteRollup.pass_rate (reports.py:1007, pass_rate=rows_passed / rows_graded if rows_graded else None), which has no _nothing_was_measured twin at all and fabricates the same 0.0 for a suite whose only graded row is a timeout.
10. [Axis 8] run --resume grading an executed row truncates that row's task.log, destroying the agent trajectory log the run paid for (src/coder_eval/orchestrator.py:623) — _grade_resumed_tasks calls regrade_in_place(..., run_dir=rt.run_dir, ...) (cli/run_command.py:787) — the row's OWN directory. Orchestrator.run() then does task_log_file = task_log_path(self.run_dir) (orchestrator.py:619) and with task_log_handler(task_log_file, task_id=self._log_task_id) as log_tail: (orchestrator.py:623), and task_log_handler opens handler = logging.FileHandler(task_log_file, mode="w", encoding="utf-8") (logging_config.py:297). mode="w" truncates. Verified empirically:

before: 'AGENT RUN LOG — 5000 lines of trajectory\n'
after : ''

So the documented coder-eval execute -> coder-eval run --resume flow wipes every graded row's task.log and replaces it with the grading pass's handful of lines. This directly contradicts the contract the same change states one function up — _apply_resume: "to_grade is deliberately NOT cleared: its artifacts are the run's output and the very thing being graded" — and task.log is a documented run-layout artifact (.claude/shared/run-layout.md:16: "the human-readable task log"), also folded into run.log. coder-eval evaluate <run_dir> is unaffected because it grades into a fresh prepared_run_dir. Fix: either back the log up the way back_up_pre_grade_record backs up task.json (e.g. task.execute.log), open the handler in append mode when prior_result is not None, or route the re-grade's log to a distinct filename (grade.log) in the row dir.

Non-blocking, but please consider before merge

Condensed for GitHub's comment size limit — full detail in the report files.

1. [Axis 1] evaluate's run-dir path builds a Sandbox/SandboxConfig it discards and restates guards regrade_in_place already performs (src/coder_eval/cli/evaluate_command.py:387)
2. [Axis 1] execute_command.py restates 146 lines / 20 of run_command's Typer options verbatim, guarded only by an option-NAME parity test (src/coder_eval/cli/execute_command.py:38)
3. [Axis 3] The recorded-command trust gate's two widest-capability arms, and --allow-host-grading, have no test at any level (src/coder_eval/orchestration/regrade.py:138)
4. [Axis 3] The heartbeat watchdog's newly added container gate is asserted only on its negative arm — the armed (in-container) branch never executes in the suite (src/coder_eval/cli/run_task_internal_command.py:90)
5. [Axis 3] Every evaluate invocation in tests/test_execute_evaluate_loop.py omits --run-dir, so each one creates a repo-relative runs/<second-resolution-timestamp>/ that xdist workers share (tests/test_execute_evaluate_loop.py:69)
6. [Axis 3] The pass-rate rule now differs between Python and the evalboard, and unlike the status table it has no parity test (evalboard/lib/overview.ts:809)
7. [Axis 4] Grading workspace escapes the run directory: default_workspace roots its containment check at the attacker-supplied artifacts/ dir and does not check the single-child / flat-artifacts returns at all (src/coder_eval/orchestration/regrade.py:290)
8. [Axis 4] The --allow-recorded-commands setup-phase scan (regrade.py:157) names only RepoSource, so evaluate <run_dir> --copy applies recorded template_dir / starter_files sources — reading arbitrary host paths into the graded… (src/coder_eval/orchestration/regrade.py:157)
9. [Axis 5] _EXECUTION_FACT_STATUSES[MAX_TURNS_EXHAUSTED] = True contradicts _terminal_status's own rationale and pins a re-graded max-turns row at exit 1 with score 1.000 (src/coder_eval/models/enums.py:103)
10. [Axis 6] evaluate's execution-fact / crash arm (evaluate_command.py:476) misreports a grading crash as 'Result count mismatch', skips restore_pre_grade_record, and has no coverage (src/coder_eval/cli/evaluate_command.py:476)
11. [Axis 6] back_up_pre_grade_record writes the pre-grade snapshot non-atomically, and restore_pre_grade_record copies it back over task.json without validating it (src/coder_eval/orchestration/regrade.py:392)
12. [Axis 7] evaluate <run_root> (the directory --run-dir creates) is rejected with a recovery hint that grades the run root as a plain work directory (src/coder_eval/cli/evaluate_target.py:86)
13. [Axis 8] Fields carried by _seed_from_prior_result are dead: the grading pass unconditionally overwrites them (error_log_tail, completed_at) (src/coder_eval/orchestrator.py:774)
14. [Axis 8] stamp_host_grading runs after Orchestrator.run() has already persisted task.json/task.html, so on run --resume --allow-host-grading the graded_on_host stamp exists only in memory and reaches no on-disk… (src/coder_eval/orchestration/regrade.py:534)

Nits

1. [Axis 1] Comment cites a source line number that is already 20 lines stale (src/coder_eval/cli/evaluate_command.py:428)
2. [Axis 1] VENV_DIRNAME lives in path_utils.py but is used only by sandbox.py (src/coder_eval/path_utils.py:28)
3. [Axis 1] evaluate_target.py wraps one boolean in a StrEnum, a dataclass, and an inverse-constructor for a single caller (src/coder_eval/cli/evaluate_target.py:27)
4. [Axis 1] The recorded-shell refusal counts non-shell capabilities as "shell command(s)" (src/coder_eval/orchestration/regrade.py:189)
5. [Axis 2] The status→CSS map in _status_badge is an inferred dict[str, str], so a new FinalStatus.category falls through to a silent runtime KeyError instead of a type error (src/coder_eval/reports_html.py:290)
6. [Axis 2] RawTaskResult.weighted_score?: number omits | null, which coder-eval execute now makes the routine on-the-wire value (evalboard/lib/runs.ts:414)
7. [Axis 4] task_config.source_file from an untrusted run directory is used unvalidated as the task directory, so a recorded reference.directory can stage a copy of any host directory (src/coder_eval/cli/evaluate_command.py:168)
8. [Axis 4] The git clone -- hardening comment overstates its protection: a recorded RepoSource URL is still arbitrary command execution via git's ext:: transport, and the gate renders it to the operator as a… (src/coder_eval/sandbox.py:460)
9. [Axis 5] The new shared format_score ungraded helper was adopted by two of the three report renderers (src/coder_eval/reports.py:474)
10. [Axis 7] execute_command's module docstring claims only one run flag is withheld; two are (src/coder_eval/cli/execute_command.py:15)
11. [Axis 7] REPORT_SCHEMA.md's error_share row omits the None-when-nothing-was-measured case the code implements (docs/REPORT_SCHEMA.md:65)
12. [Axis 7] evaluate's workspace knobs are silently inert in the new modes (--workspace ignored on the work-dir fallback; --preserve/--no-preserve inert by default) (src/coder_eval/cli/evaluate_command.py:101)
13. [Axis 8] evaluate <run_dir> write-back updates task.json but leaves the sibling task.html rendering the pre-grade NOT_GRADED row (src/coder_eval/cli/evaluate_command.py:582)
14. [Axis 8] New run-directory sidecars (task.json.graded, task.json.<pid>.<hex>.tmp) are not documented in the two run-layout contract mirrors that document task.json.malformed (.claude/shared/run-layout.md:15)

What's Missing

• 🟡 [parallel-paths] The n/a-not-zero rule stopped at the Markdown/HTML score cells and never reached the Replicate Statistics table. aggregate_results now builds per_replicate_scores through the new _measured_scores (orchestration/experiment.py:956), which drops…
• 🟡 [parallel-paths] The detached-grading write-back refreshes one of four report artifacts, and aggregate cannot rebuild the other three. _write_back (cli/evaluate_command.py:559-590) updates the row's task.json and prints a hint to run coder-eval aggregate for…
• 🔵 [parallel-paths] The --allow-recorded-commands prose and the code's actual scan disagree in both directions, and neither was reconciled. docs/USER_GUIDE.md:196-199 describes the gate as covering "run_command criteria, agent_judge, uipath_eval, and on the…
• 🟠 [tests] No test anywhere runs execute (or renders an ungraded experiment) with --repeats > 1. grep -rn repeats tests/test_execute_command.py tests/test_execute_evaluate_loop.py tests/test_ungraded_reporting.py tests/test_experiment_reports.py returns…
• 🟡 [tests] aggregate's new ungraded counts line — and the documented execute → aggregate round trip it exists for — have no test. cli/aggregate_command.py:82-89 adds the fourth bucket to the console summary with a comment saying "coder-eval aggregate <run>…
• 🟡 [downstream-consumers] per_replicate_scores changed meaning on the wire and both of its contract descriptions still state the old invariant. models/experiment.py:312-318 documents the field as "Raw weighted_score per replicate … list length equals the replicate count" —…
• 🟡 [downstream-consumers] graded_on_host has no reader on any surface, so even where it is persisted the mitigation it stands for is inert. CLAUDE.md, docs/USER_GUIDE.md:45/201/225 and CE051's docstring all justify --allow-host-grading on the grounds that the stamped row…
• 🔵 [display-mapping] The HTML status badge learned a neutral class for the new ungraded category; the score pill beside it did not. _status_badge (reports_html.py:287-290) deliberately maps "ungraded" -> "neutral" so an unmeasured row reads as neither green nor red…
• 🟠 [daily-nightly] The PR widens three cross-repo contracts at once and states its nightly blast radius nowhere. (a) Run-record schema: NOT_GRADED is a new final_status value with a fourth category (ungraded); run.json gains…
• 🟡 [daily-nightly] execute under driver: docker requires an agent image built from this change, and that requirement is documented only inside an exception string. DockerRunner._assert_grade_honored (isolation/docker_runner.py:915-930) hard-fails the task with…

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE054 — an environment_info write must survive to the persisted record. New rule in tests/lint/rules/ce054_env_info_write_reaches_disk.py, wired in tests/lint/runner.py; it extends CE047…
• [ruff] Enable C901 (mccabe) in [tool.ruff.lint] select with [tool.ruff.lint.mccabe] max-complexity ratcheted at today's worst non-exempt value, alongside the existing PLR0915/PLR0912 caps,…
• [ce-lint] CE055 — one formula per published rate. tests/lint/rules/ce055_single_rate_formula.py: lift RunSummary._nothing_was_measured (models/results.py:1173) into a shared module-level helper, then…
• [ce-lint] CE056 — no numeric-literal fallback on a ratio, in Python and in evalboard/lib/**.ts. Extend CE049's rationale across the JS twin (which has no eslint at all, so nothing polices it): forbid…
• [ce-lint] CE057 — a capability-consent scanner must be exhaustive over the union it scans. tests/lint/rules/ce057_consent_scan_exhaustive.py: registry-derived, in the CE024/assertNever family. Every…
• [ce-lint] CE058 — every path resolved out of an untrusted run directory goes through one containment helper. Add resolve_under(root: Path, candidate: Path) -> Path (resolve both sides, raise…
• [ce-lint] CE059 — a git argv list must carry -- before any non-literal value. tests/lint/rules/ce059_git_argv_separator.py: flag a list literal starting with "git" that contains a non-constant…
• [ce-lint] CE060 — a run-record file must be written with write_text_atomic. tests/lint/rules/ce060_run_record_atomic_write.py: forbid .write_text( or open(..., "w") on a path built from…
• [ce-lint] CE061 — no truncating writer on a documented run artifact. tests/lint/rules/ce061_no_truncating_run_artifact.py: a logging.FileHandler(...)/open(...) in "w" mode on a path derived from…
• [ce-lint] CE062 — no in-comment source line-number cross-reference. tests/lint/rules/ce062_no_line_number_comment.py: a regex rule over src/ comments matching (?i)\bline \d+\b when it refers to this…
• [ce-lint] CE063 — a Typer option shared by two commands must be one shared declaration. tests/lint/rules/ce063_shared_typer_option.py: collect every flag string declared by an inline typer.Option(...)…
• [ce-lint] CE064 — a run-directory sidecar filename constructed in src/ must be documented in the run-layout contract. Doc-surface rule in the CE028/CE033 family: any sidecar built from a run-record…
• [ce-lint] CE065 — Python↔TS optional-field parity for the Raw* interfaces. Same trick evalboard/lib/__tests__/status-parity.test.ts already uses (it parses _STATUS_CATEGORIES out of…
• [pyright] Annotate closed-category maps with the Literal union so pyright can see a new member. Make reports_html._status_badge's status→CSS map a module-level constant typed `dict[Literal["succeeded",…

Harness improvements:

• Diff-coverage gate in CI: run diff-cover (or --cov-fail-under scoped to the diff) requiring ≥90% of lines a PR adds to src/ to be executed by the suite, in addition to the repo-wide 80%…
• Diff-scoped mutation testing (mutmut/cosmic-ray restricted to lines the PR changed, behind a make mutate target and an optional CI job) for guard-shaped code — the boolean predicates in…
• Round-trip the _seed_from_prior_result CARRIED/RECOMPUTED partition through run(), not only through the seeding call: seed a prior result, run the Orchestrator with a stub agent on the…
• A run vs execute + evaluate differential golden test: drive one recorded trajectory (agentless fixture) through both routes and assert identical final_status, weighted_score, criteria…
• Row-directory artifact-preservation snapshot around a re-grade: hash every file in a finished row dir, run run --resume and evaluate <run_dir>, then assert no pre-existing artifact shrank or…
• A shared rate fixture asserted on both sides of the language boundary: one JSON fixture of bucket counts → expected pass_rate / error_share / success_rate, read by a pytest case and by a…
• Autouse conftest fixture pointing settings.runs_dir at tmp_path for the whole suite (with an explicit opt-out for the handful of tests that assert on the default), removing the need for every…
• Resume-partition matrix test with doc parity: a property test over FinalStatus × (verdict evidence present / absent) × iteration_count (0 / >0) asserting partition_for_resume's routing,…
• Inert-flag smoke test per CLI command: for every option on run / execute / evaluate, assert in each mode the command supports that the flag either changes an observable output or is…
• Executable-recovery-hint test: every recovery command a CLI error message prints must itself resolve to a working invocation for the shape that triggered the error (assert by re-invoking the…
• Assert single-execution of expensive/consented setup on the delegating path: a caplog + call-count test that evaluate <run_dir> emits the host-grading downgrade warning exactly once and…

Top 5 Priority Actions

1. Make the execution-fact table agree with the chain that reads it — set FinalStatus.MAX_TURNS_EXHAUSTED: False at src/coder_eval/models/enums.py:103 (or amend _terminal_status's docstring at src/coder_eval/orchestrator.py:540-558, but not both as they stand), because today a prior max-turns row re-graded through evaluate is written back as final_status=MAX_TURNS_EXHAUSTED with weighted_score=1.000 and exit 1, a combination run can never produce for the same trajectory.
2. Containment-check every return path of default_workspace against run_dir rather than against the attacker-supplied artifacts/ — src/coder_eval/orchestration/regrade.py:276-290, where the children[0] and flat-artifacts returns have no check at all and the by_task_id check is vacuous once artifacts is itself a symlink — since Sandbox.adopt then makes the escaped tree the grading root (sandbox.py:330) and the resulting verdict, criterion detail text included, is written back into the run's own task.json; add the symlink case beside tests/test_detached_grading_boundaries.py:412, and while in the same function extend the setup-phase scan at regrade.py:157 to name TemplateDirSource/StarterFilesSource so --allow-recorded-commands consent actually covers what --copy will read into the graded tree.
3. Stop the wholesale environment_info rebind at src/coder_eval/orchestrator.py:1613-1616 from discarding the reference_digest written at orchestrator.py:1418 (merge instead of replace, or move the write below the rebind), because the key never reaches task.json, so verify_reference_unchanged (regrade.py:320-328) always takes its early return and a reference edited between execute and evaluate goes undetected — the exact anti-cheat CE047 and CLAUDE.md claim this PR closed — and add a round-trip test that a reference-carrying run ends with the key present, since tests/test_detached_grading_boundaries.py:434 only tests _staged_digest in isolation.
4. Lift _nothing_was_measured to one shared helper and fix its predicate — src/coder_eval/models/results.py:1188 counts a non-zero tasks_failed as evidence of measurement, but TIMEOUT and the budget stops are category failed and reachable under execute, so a 100-task night with one timeout publishes pass_rate: 0.0; then apply the same rule to VariantAggregate.pass_rate (src/coder_eval/models/experiment.py:285, which claims to mirror RunSummary and does not, and is reached unconditionally on the execute path via run_command.py:1037-1044), to SuiteRollup (reports.py:1008), and mirror it in the evalboard, where evalboard/lib/trends.ts:196 now coalesces a fully-ungraded task to a literal 0% that sorts to the top of the default Trends view and evalboard/lib/overview.ts:809 re-derives its own denominator instead of reading the serialized rate.
5. Stop run --resume from destroying what the run already paid for: route the re-grade's log away from the row's own task.log, which task_log_handler's mode="w" truncates at src/coder_eval/orchestrator.py:623, narrow _owes_a_grade at src/coder_eval/orchestration/batch.py:354 so docker-death BUILD_FAILED/ERROR rows (no verdict, iteration_count == 0, no task_config to rebuild) are not routed to grading where run_command.py:810 replaces the container diagnostic with a wrong-cause message, and append rather than replace that error_message; pin the result with the two tests the suite is missing — an execution-fact status carrying a criteria vector against _assert_grade_honored (docker_runner.py:910, whose evidence check survives being replaced by False with a fully green suite) and the armed arm of the heartbeat gate at run_task_internal_command.py:90.

---

Stats: 0 🔴 · 10 🟠 · 14 🟡 · 14 🔵 across 8 axes reviewed.

---

Adversarial verification: 46 findings proposed, 30 medium+ verified, 1 refuted as false-positive, 26 corrected in place, 16 lows passed through unverified. 42 agents, 8 axes.

[uipreliga]

Fix what you agree with and 🚢

[uipreliga]

Architectural assessment

A follow-up to the 8-axis review above, looking at one question that axis 5 does not answer: axis 5 asks "does this change make the architecture worse?" — this asks "is the shape right in the first place?"

Verdict: the boundaries are right. The composition mechanism is wrong.

This PR gets the hard decisions correct — what to separate, where the trust boundary sits, what NOT_GRADED means. But it composes those pieces by adding modes to the existing Orchestrator instead of separating grading from running. That single choice generates most of the defects the review found.

What is correct, and worth keeping

• run and execute share one body (run_pipeline) and differ by one flag. There is no third code path. Right call.
• regrade.py is genuinely one implementation with two callers. I checked the function-local from coder_eval.orchestrator import Orchestrator — it does not mask a cycle; hoisting it to module scope imports cleanly. A style wart, not a layering violation.
• evaluate_target.py is a pure resolver. Shape detection is separated from I/O and testable alone.
• NOT_GRADED is a real fourth category, not an overloaded FAILURE, and weighted_score is None rather than 0.0. That distinction is correct, and CE049 institutionalizes it.
• The trust boundary is named and gated at all. Most harnesses never treat a run directory as untrusted input.
• Sandbox.adopt is a verb, not setup(copy=False). Right instinct.

The one root cause

Grading is modeled as a degenerate mode of running, rather than as an operation over a recorded artifact.

The evidence is consistent:

• Orchestrator.__init__ takes 15 parameters, including two new orthogonal mode flags (grade, prior_result) plus a third implicit one (sandbox.was_adopted). That is 8 nominal combinations for roughly 4 legal ones.
• One illegal combination is rejected at runtime, because the types permit it (orchestrator.py:2267): grade=False with no agent raises ValueError("grade=False is meaningless on the evaluate-only path"). An unrepresentable state would need no such guard.
• _seed_from_prior_result is a ~40-field manual carry list, each field with a paragraph explaining why it must survive.
• tests/test_seed_from_prior_result.py partitions every EvaluationResult field as CARRIED or RECOMPUTED and fails closed on new ones — a hand-maintained allowlist standing in for structure.
• The class grew 2772 → 3212 lines, 49 methods.

Now count the highs from the review: 6 of 10 are the same bug — the run path wrote something the grade path should not have written.

Finding	The run path writes…
sandbox_path clobbered	orchestrator.py:1508, then again in cleanup
task.log truncated	log handler opens mode="w" at :623
reference_digest wiped	environment_info rebound wholesale at :1613
duration_seconds	written at :1046, then manually restored at :1051
error_log_tail / completed_at	carried, then overwritten downstream
MAX_TURNS_EXHAUSTED ordering	the grade × prior_result interaction itself

duration_seconds is the tell. The code writes the wrong value and then puts the right one back two lines later. That is a design fighting its own default.

The improvements, ranked

1. Invert the seeding default. Make grading a pure function over a recorded run:

def grade_trajectory(prior, task, workspace, ...) -> GradeOutcome
# GradeOutcome = (success_criteria_results, weighted_score, final_status, graded_by_*)

Apply it with prior.model_copy(update=outcome.as_update()). "Carried" becomes structural, so the explicit list shrinks from ~40 fields to ~4 — and it lists what grading replaces, which is the short, stable, obviously-correct set. The CARRIED/RECOMPUTED partition test stops being necessary. Four of the six clobbers above become impossible rather than fixed, because the grade path never enters the code that writes them.

2. Make the modes a union, not flags. Three entry shapes instead of one 15-parameter constructor: run-and-grade, run-only, grade-only. The third needs no Orchestrator at all — no agent, no sandbox setup, no log handler. That deletes the runtime ValueError, because the illegal combination stops being expressible.

3. Extract the status algebra as a pure function. terminal_status(prior_status, success, graded, max_turns) -> FinalStatus, with no self. Today that logic is a 4-way interaction inside a method, explained by a 30-line docstring — and the MAX_TURNS_EXHAUSTED bug is precisely a disagreement between the table at enums.py:103 and the chain that reads it. The full product is about 40 cases. An exhaustive table test would have caught that contradiction mechanically, before it reached a review.

4. One Rates value object. Three implementations of pass_rate with three different behaviors for the same inputs is a DRY violation that already produced a wrong number. Construct rates once from bucket counts.

Improvements 3 and 4 are small and self-contained. 1 and 2 are the real refactor, and 1 is the one that pays.

Should this block the merge?

No. The boundaries are right, the modal flags are contained inside one class rather than leaking through the codebase, and each specific bug is individually fixable. The design is recoverable without a rewrite.

But improvement 1 is worth doing before more features land on this seam. Every new EvaluationResult field is now a potential silent-drop bug whose only guard is a hand-maintained test. That cost compounds, and this PR is where the seam is still small enough to invert cheaply.