Skip to content

feat: resume runs from workspace checkpoints - #1142

Draft
mkceichelbeck wants to merge 18 commits into
mainfrom
codex/workspace-checkpoint-resume
Draft

mkceichelbeck wants to merge 18 commits into
mainfrom
codex/workspace-checkpoint-resume

Conversation

@mkceichelbeck

@mkceichelbeck mkceichelbeck commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Failed local and direct-target runs can resume at a saved node or concurrent group, restoring its input workspace and completed prerequisite outputs. Plain zeroshot resume RUN_ID restarts the original graph on the latest retained workspace.

Snapshots now live in one encrypted, deduplicated Restic repository per recovery lineage. Full filesystem copies exist only in private temporary staging directories and are removed after each operation. Successful lineages delete their checkpoint repository; failed or lost runs retain it for recovery.

Changes Made

  • Add the opt-in OECP openengine.workspace-checkpoints/v1 extension, typed checkpoint listing and resume selection, client/server support, and generated protocol/CLI documentation.
  • Let the reducer define checkpoint boundaries. An outer parallel or mapped group remains one unit across nested work and dispatch batches; capture waits for all owned work to settle.
  • Back local and direct-target checkpoints with supervised Restic 0.19.1 processes using bounded output, timeouts, no persistent cache, private random repository credentials, and a minimal environment.
  • Keep the existing Git-aware filesystem snapshotter only for temporary normalization and safe restore. Interrupted staging copies are removed during controller reconciliation and retained-workspace cleanup.
  • Share one Restic repository across all attempts in a recovery lineage. Successful and force-stopped runs delete checkpoint bytes; failed and runtime-lost runs retain them. Reconstructed successors resolve the original lineage before cleanup.
  • Bundle the pinned Restic sidecar in native archives, npm installs, Python wheels, self-update, and the target image for all five release platforms.
  • Reap remaining worker descendants immediately after the main process exits, before draining inherited output streams.
  • Allow hosting applications to inject workspace storage after checkout and before dispatch, and add optional hosted resume/checkpoints/discard routes for Cloud.

Testing

  • Real Anthropic-backed local and Docker runs covered initial failure, restart from latest, selected parallel-group resume, and selected mapped-group resume. Assertions checked exact execution counts, workspace counters, restored files, skipped prerequisites, and successful-lineage deletion.
  • Real Restic tests verify incremental deduplication, latest-byte restore, failed backup/restore behavior, invalid snapshot rejection, private staging cleanup, and lineage cleanup after restart.
  • Built and inspected pinned Restic sidecars for Linux x64/arm64, macOS x64/arm64, and Windows x64. Built the target image and smoke-tested both bundled executables.
  • Hosted Linux and Windows CI passed the complete native workspace suite, real Restic checkpoint storage, strict documentation, target image packaging, and canonical executable smoke tests. The Windows lane also passed Unicode and spaced-path coverage.
  • Full Rust workspace, optional UI, Clippy, rustdoc, protocol generation, npm/release tooling, and Python SDK suites pass. Opcore Zero Verify is clean; Sense has no findings.

Checklist

  • Relevant validation checks pass
  • Documentation updated
  • Follows commit guidelines

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant