Skip to content

engine: remove the recovery sidecar machinery (RFC 0067 step 5) - #741

Closed
ragnorc wants to merge 1 commit into
claude/rfc-0067-system-column-upgrade-detachedfrom
claude/rfc-0067-remove-recovery-machinery
Closed

ragnorc wants to merge 1 commit into
claude/rfc-0067-system-column-upgrade-detachedfrom
claude/rfc-0067-remove-recovery-machinery

Conversation

@ragnorc

@ragnorc ragnorc commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Stacked on #740 (system-column upgrade detached). RFC 0067 rollout step 5, second half: with no writer left that arms a recovery sidecar, the recovery machinery is deleted.

What is removed

  • db/manifest/recovery.rs (about 11,000 lines: sidecar grammar, shape validators, the effect classifier, roll-forward and rollback executors, Restore compensation, both recovery modes, the in-source truth tables) and db/recovery_audit.rs (the recovery audit dataset has no appender left).
  • The write-entry heal and barrier: heal_pending_recovery_sidecars and its variants, the under-gate rechecks in mutation, load, merge, index maintenance, schema apply, the system-column upgrade and branch control, the staging owner check, Optimize's and cleanup's clean-recovery-state refusals, repair's sidecar probe, and the sidecar arm of the baseline check. Every write loses a __recovery/ LIST.
  • Seams recovery.*, optimize.post_recovery_check_pre_main_gate, system_column_upgrade.after_lock_reclaim; branch_control.post_recovery_barrier is renamed branch_control.pre_gates and cleanup.post_recovery_check_pre_gates is renamed cleanup.pre_gates (the one window where a failure aborts a whole cleanup run). The DST crash-window catalog lists 61 windows.
  • Dead code the removal exposed: publish_stamp_advance, the unmarked-staging writer and its exact-staging helpers, CommitGraph::latest_commit_matching, TableStore::append_batch.
  • The GQ logic test known_failure marker, which existed only to pin known recovery-sidecar defects and was carried by no case (RFC 0045 records the removal).

What remains

  • settle_pending_schema_install: the write-entry pass is a flag-gated install of this handle's own published schema contract; no storage probe on the common path.
  • A read-write open refuses a graph whose __recovery/ directory still holds a sidecar (db/manifest/legacy_sidecars.rs). Only a build that predates detached commits can have written one and this build cannot interpret it; the message names the operation ids and the remedy. The storage upgrade keeps the same refusal. It then settles a staged schema contract by its publishing commit and reclaims a stale schema-apply sentinel.
  • A read-only open never lists __recovery/ (reads are pinned to published versions) and keeps only the published-contract coherence check, now in schema_state.rs. refresh installs a published staged contract and nothing else.
  • repair's drift adoption stays, against the RFC's step-5 list. It is not sidecar machinery, and it is the operator's one remedy for a foreign linear commit that arrived before any pending pin; how a blocked table is unblocked is still the RFC's open question. The RFC text records this.

Invariants

docs/dev/invariants.md restates invariants 2 and 5 for the detached protocol (pre-publication effects stay unreachable; what is published carries the identity that finishes it; there is no side record to classify) and replaces the Optimize/full-recovery one-mutation-process boundary with the one that remains: the open-time discard of an unpublished staged contract and the sentinel reclaim. AGENTS.md rule 4 follows.

Tests

  • tests/recovery.rs shrinks to the contracts that remain: a clean open creates nothing, a legacy sidecar refuses a read-write open and not a read-only one, index maintenance leaves no sidecar, a read-only open never touches schema staging.
  • Six failpoint cells and one maintenance cell that injected synthetic sidecars retire; the recovery test helpers shrink to the two with users; the storage-upgrade refusal uses a plain file; forbidden_apis.rs drops the recovery executor rows and the branch-control guard now pins only the gates-before-capture order.
  • DST: the double-fault lever dies on the open path (open.before_schema_contract_read), which every reopen crosses; the recovery seam it named had been unreachable since mutations stopped arming sidecars. The harness no longer reads a recovery audit. The staleness pin moves to seed 278 (removing the per-write __recovery/ listing shifted the storage-action schedule; the seed search lists the green seeds). The sidecar-weather pin retires: lost and misdirected writes only ever acted on sidecar writes. The cost golden loses every per-operation adapter listing (for example AddFriend a.list 8 to 0, Optimize a.list 6 to 0).

Size

69 files, about 1,000 lines added and 17,200 removed.

Follow-up

The DST harness's keep-serving machinery (KeepServingWatch, reconcile_watch_resolution) can no longer be entered and is removed in a separate harness-only PR, so this one keeps its DST delta small.

Validation

Local: the workspace test graph with failpoints, the full engine suite, GQT crate tests and corpus, DST from its crate directory with the regenerated golden, lance_surface_guards, fmt, both clippy graphs and GQT clippy, docs/typos/agents/pins checks, OpenAPI drift.

Every writer now stages detached table commits and publishes pins, so
nothing arms a recovery sidecar and nothing is left for a classifier to
decide. This removes the machinery:

- db/manifest/recovery.rs (the sidecar grammar, shape validators, effect
  classifier, roll-forward and rollback executors, Restore compensation,
  both recovery modes and their in-source truth tables) and
  db/recovery_audit.rs (the recovery audit dataset, which has no appender
  left) are deleted.
- The write-entry heal and barrier are gone: heal_pending_recovery_sidecars
  and its for-write and for-branch-delete variants, the under-gate rechecks,
  the staging owner check, Optimize's and cleanup's clean-recovery-state
  refusals, repair's sidecar probe, and the sidecar arm of the baseline
  check. What remains of the write-entry pass is
  settle_pending_schema_install: a flag-gated install of this handle's own
  published schema contract, with no storage probe on the common path.
- A read-write open no longer sweeps: it refuses a graph whose __recovery/
  directory still holds a sidecar (db/manifest/legacy_sidecars.rs; only a
  build that predates detached commits can have written one, and this build
  cannot interpret it), settles a staged schema contract by its publishing
  commit, and reclaims a stale schema-apply sentinel. The storage upgrade
  keeps the same refusal. A read-only open never lists __recovery/ and keeps
  only the published-contract coherence check, now in schema_state.rs.
  refresh installs a published staged contract and nothing else.
- Seams recovery.*, optimize.post_recovery_check_pre_main_gate and
  system_column_upgrade.after_lock_reclaim are gone;
  branch_control.post_recovery_barrier is renamed branch_control.pre_gates
  and cleanup.post_recovery_check_pre_gates is renamed cleanup.pre_gates
  (the one window where a failure aborts a whole cleanup run; per-table GC
  failures are isolated). The DST crash-window catalog lists 61 windows.
- Dead code the removal exposed goes with it: publish_stamp_advance, the
  unmarked-staging writer and its exact-staging helpers,
  CommitGraph::latest_commit_matching, TableStore::append_batch, the fork
  helper's unused operation-id parameter, the two Lance restore fences the
  recovery sweep depended on, and the retired audit table's name in the
  internal-table list; the coordinator's open and open_branch and the
  publisher's publish shorthand are test-only. Merge's limit and ownership
  messages say "detached transaction chain" and "planned bound" instead of
  "recovery transactions" and "armed bound".

repair's drift adoption stays. It is not sidecar machinery, and it is the
operator's one remedy for a foreign linear commit that arrived before any
pending pin; how a blocked table is unblocked remains the RFC's open
question.

The GQ logic test format loses its known_failure marker, which existed only
to pin known recovery-sidecar defects and was carried by no case.

Tests: tests/recovery.rs shrinks to the contracts that remain (a clean open
creates nothing, a legacy sidecar refuses a read-write open and not a
read-only one, index maintenance leaves no sidecar, a read-only open never
touches schema staging); the failpoint and maintenance cells that injected
synthetic sidecars retire; the recovery test helpers shrink to the two with
users; the storage-upgrade refusal uses a plain file; the registry drops
the recovery executor rows. DST: the double-fault lever dies on the open
path instead of an unreachable recovery seam; the harness no longer reads a
recovery audit; the staleness pin moves to seed 278 because removing the
per-write __recovery/ listing shifted the storage-action schedule; the
sidecar-weather pin retires because lost and misdirected writes only ever
acted on sidecar writes; the cost golden loses every per-operation adapter
listing.

Docs: recovery.md is rewritten around pins, promotion, staged contracts and
the legacy-sidecar refusal; writes.md, invariants.md (invariants 2 and 5 and
the one-mutation-process boundary), AGENTS.md rule 4, versioning.md,
testing.md, merge.md, lance.md, the user troubleshooting and maintenance
guides, the release notes, RFC 0067's step 5, a removal note in RFC 0045,
and the engine's own comments, which no longer describe sidecars. The
OpenAPI description of the resource-limit output drops its sidecar wording.
@ragnorc

ragnorc commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Consolidated into #744, which carries this stack's complete history merged with current main and revalidated as one tree.

@ragnorc ragnorc closed this Sep 17, 2026
@azimafroozeh
azimafroozeh deleted the claude/rfc-0067-remove-recovery-machinery branch September 29, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant