Skip to content

engine: ensure_indices and the full-text rebuild stage detached commits (RFC 0067 step 4) - #733

Closed
ragnorc wants to merge 3 commits into
claude/rfc-0067-mutation-load-detachedfrom
claude/rfc-0067-ensure-indices-detached
Closed

ragnorc wants to merge 3 commits into
claude/rfc-0067-mutation-load-detachedfrom
claude/rfc-0067-ensure-indices-detached

Conversation

@ragnorc

@ragnorc ragnorc commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

RFC 0067 rollout step 4, second writer: ensure_indices and the explicit full-text rebuild stage detached commits. Stacked on #731 (mutation and load), which is stacked on #728, #727 and #725.

What changes

  • Every index batch is a detached commit of the pinned base. maintain_indices_for_branch opens each productive table at its pin (open_pinned_for_write, promoting a pending predecessor first), stages the complete BTREE/FTS/vector batch before the gates as before, and under the gates commits each batch with commit_staged_detached, publishes the pins (expected + 1, staged, uuid) in the one manifest commit, and promotes them from the handles it holds. No EnsureIndices sidecar is armed, confirmed or deleted; the pre-minted transaction identities, the exact-commit outcome check and the RecoveryRequired wrapping are gone with it. A first-touch fork on a branch (explicit rebuild only) is created under the gates with no intent record, exactly like a mutation's first touch; its batch stages on the fork.
  • The full-text certificate is unchanged: stage_create_indices writes it before the commit, the detached batch carries the same index uuid as its promoted twin, and search is served through a pending pin.
  • Seams. ensure_indices.post_sidecar_pre_fork and ensure_indices.post_effects_pre_confirm are gone; ensure_indices.post_fork_pre_commit and ensure_indices.post_publish_pre_promotion are new; post_stage_pre_commit_btree, post_table_effect and post_phase_b_pre_manifest_commit keep their names over the detached effects.
  • The classifier keeps the EnsureIndices sidecar kind for sidecars written before this change, as engine: mutation and load stage detached commits and publish pins (RFC 0067 step 4a) #731 did for Mutation/Load; step 5 removes them together. new_ensure_indices_sidecar_v9 is cfg(test), confirm_ensure_indices_sidecar_v9 is deleted.

Tests

  • detached_commit_matrix.rs gains Writer::EnsureIndices (a declared BTREE the schema apply leaves unbuilt; the same windows, faults and recovery actors as an insert; the oracle additionally runs a converging index pass after recovery).
  • failpoints.rs: the nine sidecar-era index tests (armed-effect rollback, partial arming, first-touch crash and rollback, post-effect winners, confirmed full-text recovery) retire with the sidecar; new ensure_indices_interrupted_after_publish_leaves_a_pending_pin_the_next_writer_promotes, ensure_indices_failure_before_publish_leaves_no_residue, full_text_rebuild_pending_pin_serves_search_and_promotes and full_text_rebuild_first_touch_fork_failure_leaves_no_residue. The recovery-machinery tests (discovery races, the cleanup barrier, the storage-fault contracts, in-process healing) now take their pending sidecar from branch merge (graph_with_merge_work + leave_confirmed_merge_sidecar); the schema-apply heal test produces its own schema-apply sidecar because schema apply needs a main-only graph. The blocked-pin test uses Optimize as the writer that still refuses.
  • write_cost.rs: an index pass with work writes and deletes no control object.
  • changes_cost.rs: the delta-bounded page cell seeds nine commits instead of eight so the padding dataset has rows at both depth points; the stale-index arm's index build then covers the same table set at both points. With the detached index build, a two-table index commit leaves the following page two __manifest fragment reads cheaper than a one-table one, which the old fixture (Company empty at the deep point only) read as growth with history; the page results and the deep point's cost are unchanged.
  • The S3 sidecar-lifecycle cell from engine: mutation and load stage detached commits and publish pins (RFC 0067 step 4a) #731 takes its sidecar from branch merge, like the other machinery tests.
  • crates/omnigraph-dst: the crash-window catalog follows the renamed index seams (the maintenance-obligation cell crashes the index writer at post_table_effect, the window that still leaves work), and the cost golden is regenerated for the index writer's detached batches.
  • Docs: writes.md (writer table row and the index writer's protocol), recovery.md, testing.md, release note.

Notes

  • Step 4 continues with branch merge, then schema apply, then Optimize.

@ragnorc
ragnorc force-pushed the claude/rfc-0067-ensure-indices-detached branch 3 times, most recently from dd349da to de3afe6 Compare September 16, 2026 18:20
@ragnorc
ragnorc force-pushed the claude/rfc-0067-mutation-load-detached branch from 61e462b to ed911c5 Compare September 17, 2026 07:01
@ragnorc
ragnorc force-pushed the claude/rfc-0067-ensure-indices-detached branch from de3afe6 to 74318a3 Compare September 17, 2026 07:01
…ts (RFC 0067 step 4)

The index writer follows mutation and load: it opens each productive
table at its pin, stages the complete index batch before the gates as
before, commits every batch as a detached version of the pin, publishes
the pins in the one manifest commit and promotes them from the handles it
holds. No EnsureIndices sidecar is armed, confirmed or deleted; a
first-touch fork on a branch is created under the gates with no intent
record, and the full-text certificate written before the commit serves
search through a pending pin.

Seams ensure_indices.post_sidecar_pre_fork and
ensure_indices.post_effects_pre_confirm are gone;
ensure_indices.post_fork_pre_commit and
ensure_indices.post_publish_pre_promotion name the new windows. The
failure matrix gains the index writer, the sidecar-era index tests
retire, and the recovery-machinery tests take their pending sidecar from
branch merge. The classifier keeps the EnsureIndices kind until step 5.
… seams

ensure_indices.post_sidecar_pre_fork and post_effects_pre_confirm are
gone; their slots name post_fork_pre_commit and post_publish_pre_promotion,
the harness recipes follow, and the maintenance-obligation cell crashes the
index writer at post_table_effect, the window that still leaves work.
The cost golden follows the index writer's pinned opens and promotions,
and the bounded-staleness pin takes a seed that stays green on the
detached mutation and index schedules.
@ragnorc
ragnorc force-pushed the claude/rfc-0067-mutation-load-detached branch from ed911c5 to 05e5f32 Compare September 17, 2026 07:52
@ragnorc
ragnorc force-pushed the claude/rfc-0067-ensure-indices-detached branch from 74318a3 to 7bba37d Compare September 17, 2026 07:53
@ragnorc

ragnorc commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Consolidated into #744, which carries this stack's complete history merged with current main and revalidated as one tree.

@ragnorc ragnorc closed this Sep 17, 2026
@azimafroozeh
azimafroozeh deleted the claude/rfc-0067-ensure-indices-detached branch September 29, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant