Skip to content

A cell that errors on a later attempt still discards the earlier attempts' transcripts #97

Open

Description

@leggetter

#94 keeps the transcript of every superseded attempt, but only on the paths where runOne returns. Two guards inside the loop throw: the one for an agent that errored before emitting any transcript event, and isDeadProcess for a container that did not exit cleanly. A throw lands in the caller's catch, which records the error and writes no result file at all — so priorAttempts is a local that never escapes.

Scenario, on CI's default --runs 2: attempt 1 fails for a real reason and is recorded. Attempt 2's container is SIGKILLed, isDeadProcess fires, runOne throws, nothing is written. The attempt 1 transcript — exactly the kind of evidence #94 exists to keep — is gone.

Not hypothetical: the comments on those two guards cite 13 August and 21 August incidents where they fired across dozens of cells.

This is distinct from #93, which is about the upload step being skipped after a failing check.

Why it is not a one-liner

Writing the partial evidence to resultPath() would undo the invariant those throws exist to protect: an errored cell must not produce a result file, because the exporter reads that path and a wrong row published against a named vendor is worse than a missing one. That is the 13 August incident.

The exporter also globs <experiment>/*.json, so a sidecar next to the result file becomes a row with a mangled eval id. The evidence needs a location the exporter does not read — a separate top-level directory — and the workflow's artifact upload needs to include it.

Activity

  1. added
    harnessRunner, scorers, provisioner and CI — work that can be done in this repo
    on Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    harnessRunner, scorers, provisioner and CI — work that can be done in this repo

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions