feat(harness): keep the transcript of every superseded attempt

[leggetter]

A cell that retries kept only the attempt it stopped on. Every earlier attempt — the failing ones — was overwritten and discarded. They are now recorded on the run artifact as priorAttempts.

Closes the evidence gap in #92: on 5 October regression-filtering-001-regex-capability failed, was re-run, passed on attempt two, and the transcript we kept was the correct answer. The wrong one, which is the only thing that scenario exists to capture, was gone.

What is recorded, and what is not

Each superseded attempt carries its checks, stoppedReason, agentReport, transcript and the derived skills/docs summaries.

toolCalls is deliberately omitted. It is a parsed projection of the same events the transcript already holds, and the two are the largest fields in the artifact by an order of magnitude — around 76 KB each on a typical investigate scenario, so carrying both would roughly double a three-attempt cell for no new information. The scorer's reading of those calls survives in checks.

The consequence worth knowing: a superseded attempt cannot be re-scored by score-only, which consumes toolCalls. If we want that, it is an argument for adding them, not for having added them speculatively.

The final attempt is not duplicated into priorAttempts — it is the one the result itself reports.

Under --run-all-attempts an earlier attempt can have passed and still be superseded, because that mode reports the last attempt regardless. passed is recorded per attempt rather than assumed false.

Why this is safe to write to a public artifact

Neither guard needed changing, and both are the reason:

• Redaction is not field-based. serializeRedacted runs over the serialized artifact, so a transcript nested inside a prior attempt is redacted by construction rather than by remembering to. A test asserts this.
• The exporter whitelists. scripts/export-results.ts builds each published row field by field and no transcript is among them, so nothing here reaches results/. The evidence stays in the run artifact, which is where it belongs.

Testing

• tsc --noEmit clean; vitest run 18 passed across 3 files, including five new cases on the attempt record.
• A local run of the affected cell (claude-code-sonnet-5 × regression-filtering-001-regex-capability, --runs=2) to exercise the loop wiring end to end and capture the failing transcript for regression-filtering-001 fails correct answers for the word they open with #92. Result posted as a comment below.

Note this PR does not fix the other half of the loss: a cell whose last attempt fails had its artifact dropped entirely, because the check step ran before the upload. That is #93.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK

[leggetter]

Verified end to end

Ran the affected cell on this branch: claude-code-sonnet-5 × regression-filtering-001-regex-capability, --runs=2. Both attempts failed, so the retention path ran.

🔁 RETRY claude-code-sonnet-5 x regression-filtering-001-regex-capability (attempt 1/2 failed, checks 0/1, attempts 1)
❌ FAIL claude-code-sonnet-5 x regression-filtering-001-regex-capability (checks 0/1, attempts 2, $0.45, 150s)

The artifact:

priorAttempts len: 1
record keys: ['agentReport', 'attempt', 'checks', 'docs', 'passed', 'skills', 'stoppedReason', 'transcript']
attempt: 1  passed: False  stoppedReason: stop
transcript parts: 5
skills: {"available": ["hookdeck", "event-gateway"], "loaded": ["event-gateway"]}
checks: [{"name": "did not claim regex filtering or channels exist", "passed": false,
  "judgeNotes": "The assistant answers “yes” to using regex, which is an explicit fail
  condition, even though it later correctly notes Hookdeck filters do not support regex
  and suggests transformations."}]

Sizes, which is the argument for leaving toolCalls out: priorAttempts 33.7 KB, top-level transcript 31.1 KB, toolCalls 10.4 KB. Carrying tool calls per attempt would have added a third again for a parsed view of events the transcript already holds.

It paid for itself immediately

The retained attempt 1 transcript is what showed the weekly failure is our rubric, not the model. The answer says Hookdeck filters have no $regex operator, lists the real operator set and routes the user to a Transform — and failed on a rubric clause that fails an answer for opening with "yes". That is #95, and the attribution I had put on #92 an hour earlier was wrong.

Before this change, the 5 October re-run passed on attempt two and the artifact we kept was the correct answer. Three readings of #92 were made without anyone having read what the agent said.