Repository navigation
Conversation
Every writer now stages detached table commits and publishes pins, so nothing arms a recovery sidecar and nothing is left for a classifier to decide. This removes the machinery: - db/manifest/recovery.rs (the sidecar grammar, shape validators, effect classifier, roll-forward and rollback executors, Restore compensation, both recovery modes and their in-source truth tables) and db/recovery_audit.rs (the recovery audit dataset, which has no appender left) are deleted. - The write-entry heal and barrier are gone: heal_pending_recovery_sidecars and its for-write and for-branch-delete variants, the under-gate rechecks, the staging owner check, Optimize's and cleanup's clean-recovery-state refusals, repair's sidecar probe, and the sidecar arm of the baseline check. What remains of the write-entry pass is settle_pending_schema_install: a flag-gated install of this handle's own published schema contract, with no storage probe on the common path. - A read-write open no longer sweeps: it refuses a graph whose __recovery/ directory still holds a sidecar (db/manifest/legacy_sidecars.rs; only a build that predates detached commits can have written one, and this build cannot interpret it), settles a staged schema contract by its publishing commit, and reclaims a stale schema-apply sentinel. The storage upgrade keeps the same refusal. A read-only open never lists __recovery/ and keeps only the published-contract coherence check, now in schema_state.rs. refresh installs a published staged contract and nothing else. - Seams recovery.*, optimize.post_recovery_check_pre_main_gate and system_column_upgrade.after_lock_reclaim are gone; branch_control.post_recovery_barrier is renamed branch_control.pre_gates and cleanup.post_recovery_check_pre_gates is renamed cleanup.pre_gates (the one window where a failure aborts a whole cleanup run; per-table GC failures are isolated). The DST crash-window catalog lists 61 windows. - Dead code the removal exposed goes with it: publish_stamp_advance, the unmarked-staging writer and its exact-staging helpers, CommitGraph::latest_commit_matching, TableStore::append_batch, the fork helper's unused operation-id parameter, the two Lance restore fences the recovery sweep depended on, and the retired audit table's name in the internal-table list; the coordinator's open and open_branch and the publisher's publish shorthand are test-only. Merge's limit and ownership messages say "detached transaction chain" and "planned bound" instead of "recovery transactions" and "armed bound". repair's drift adoption stays. It is not sidecar machinery, and it is the operator's one remedy for a foreign linear commit that arrived before any pending pin; how a blocked table is unblocked remains the RFC's open question. The GQ logic test format loses its known_failure marker, which existed only to pin known recovery-sidecar defects and was carried by no case. Tests: tests/recovery.rs shrinks to the contracts that remain (a clean open creates nothing, a legacy sidecar refuses a read-write open and not a read-only one, index maintenance leaves no sidecar, a read-only open never touches schema staging); the failpoint and maintenance cells that injected synthetic sidecars retire; the recovery test helpers shrink to the two with users; the storage-upgrade refusal uses a plain file; the registry drops the recovery executor rows. DST: the double-fault lever dies on the open path instead of an unreachable recovery seam; the harness no longer reads a recovery audit; the staleness pin moves to seed 278 because removing the per-write __recovery/ listing shifted the storage-action schedule; the sidecar-weather pin retires because lost and misdirected writes only ever acted on sidecar writes; the cost golden loses every per-operation adapter listing. Docs: recovery.md is rewritten around pins, promotion, staged contracts and the legacy-sidecar refusal; writes.md, invariants.md (invariants 2 and 5 and the one-mutation-process boundary), AGENTS.md rule 4, versioning.md, testing.md, merge.md, lance.md, the user troubleshooting and maintenance guides, the release notes, RFC 0067's step 5, a removal note in RFC 0045, and the engine's own comments, which no longer describe sidecars. The OpenAPI description of the resource-limit output drops its sidecar wording.
This was referenced Sep 17, 2026
Contributor
Author
|
Consolidated into #744, which carries this stack's complete history merged with current main and revalidated as one tree. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #740 (system-column upgrade detached). RFC 0067 rollout step 5, second half: with no writer left that arms a recovery sidecar, the recovery machinery is deleted.
What is removed
db/manifest/recovery.rs(about 11,000 lines: sidecar grammar, shape validators, the effect classifier, roll-forward and rollback executors,Restorecompensation, both recovery modes, the in-source truth tables) anddb/recovery_audit.rs(the recovery audit dataset has no appender left).heal_pending_recovery_sidecarsand its variants, the under-gate rechecks in mutation, load, merge, index maintenance, schema apply, the system-column upgrade and branch control, the staging owner check, Optimize's and cleanup's clean-recovery-state refusals, repair's sidecar probe, and the sidecar arm of the baseline check. Every write loses a__recovery/LIST.recovery.*,optimize.post_recovery_check_pre_main_gate,system_column_upgrade.after_lock_reclaim;branch_control.post_recovery_barrieris renamedbranch_control.pre_gatesandcleanup.post_recovery_check_pre_gatesis renamedcleanup.pre_gates(the one window where a failure aborts a whole cleanup run). The DST crash-window catalog lists 61 windows.publish_stamp_advance, the unmarked-staging writer and its exact-staging helpers,CommitGraph::latest_commit_matching,TableStore::append_batch.known_failuremarker, which existed only to pin known recovery-sidecar defects and was carried by no case (RFC 0045 records the removal).What remains
settle_pending_schema_install: the write-entry pass is a flag-gated install of this handle's own published schema contract; no storage probe on the common path.__recovery/directory still holds a sidecar (db/manifest/legacy_sidecars.rs). Only a build that predates detached commits can have written one and this build cannot interpret it; the message names the operation ids and the remedy. The storage upgrade keeps the same refusal. It then settles a staged schema contract by its publishing commit and reclaims a stale schema-apply sentinel.__recovery/(reads are pinned to published versions) and keeps only the published-contract coherence check, now inschema_state.rs.refreshinstalls a published staged contract and nothing else.repair's drift adoption stays, against the RFC's step-5 list. It is not sidecar machinery, and it is the operator's one remedy for a foreign linear commit that arrived before any pending pin; how a blocked table is unblocked is still the RFC's open question. The RFC text records this.Invariants
docs/dev/invariants.mdrestates invariants 2 and 5 for the detached protocol (pre-publication effects stay unreachable; what is published carries the identity that finishes it; there is no side record to classify) and replaces the Optimize/full-recovery one-mutation-process boundary with the one that remains: the open-time discard of an unpublished staged contract and the sentinel reclaim.AGENTS.mdrule 4 follows.Tests
tests/recovery.rsshrinks to the contracts that remain: a clean open creates nothing, a legacy sidecar refuses a read-write open and not a read-only one, index maintenance leaves no sidecar, a read-only open never touches schema staging.forbidden_apis.rsdrops the recovery executor rows and the branch-control guard now pins only the gates-before-capture order.open.before_schema_contract_read), which every reopen crosses; the recovery seam it named had been unreachable since mutations stopped arming sidecars. The harness no longer reads a recovery audit. The staleness pin moves to seed 278 (removing the per-write__recovery/listing shifted the storage-action schedule; the seed search lists the green seeds). The sidecar-weather pin retires: lost and misdirected writes only ever acted on sidecar writes. The cost golden loses every per-operation adapter listing (for exampleAddFriend a.list8 to 0,Optimize a.list6 to 0).Size
69 files, about 1,000 lines added and 17,200 removed.
Follow-up
The DST harness's keep-serving machinery (
KeepServingWatch,reconcile_watch_resolution) can no longer be entered and is removed in a separate harness-only PR, so this one keeps its DST delta small.Validation
Local: the workspace test graph with failpoints, the full engine suite, GQT crate tests and corpus, DST from its crate directory with the regenerated golden,
lance_surface_guards, fmt, both clippy graphs and GQT clippy, docs/typos/agents/pins checks, OpenAPI drift.