[ragnorc]

Stacked on #735 (schema apply). RFC 0067 rollout step 4, fifth and last writer: Optimize.

What changes

• Compaction stages detached. Lance's compaction plan is executed against the pinned base and staged as one Rewrite whose new fragments take ids above the base's high-water mark, so no ReserveFragments commit is needed (its replay would not conflict with its twin). Stable-row-id rewrites carry scalar index coverage over to the new fragments when Lance applies them.
• Index folds stage detached as whole rebuilds. Lance 11 folds only through a linear optimize_indices and keeps merge_indices crate-private, so each foldable index whose coverage lags the rewritten layout is rebuilt whole under its name with the public builder, chained on the rewrite; a vector index keeps its partition count. A vector column that cannot train an index is reported under pending_indexes, as before. Full-text indexes stay excluded (RFC 0043).
• Deferred index builds chain on the fold. Declared-but-unbuilt indexes are the third link, staged with the index writer's own stage_create_indices.
• One exact publication. The batch publishes through the shared publisher with an exact CAS on every pin it planned from and the captured graph head, then promotes the held chains. A pin a concurrent writer moved fails the run with a read-set conflict and the next run re-plans; the writer's row is never hidden. A failure before the manifest commit leaves only reclaimable detached versions; one after it leaves pending pins the next writer of each table or cleanup promotes. No Optimize sidecar, no RecoveryRequired, no one-mutation-process boundary.
• No config strip on data tables. Every engine commit and promotion replay skips Lance's auto-cleanup, so a stale lance.auto_cleanup.* key is inert; stripping it would add a commit whose replay does not conflict with its twin. The internal __manifest compaction keeps its linear compact_files and strip.
• A strict mutation queued behind Optimize reports the typed conflict. Before, the Optimize sidecar made a concurrent update or delete wait at the recovery barrier and replan silently. Now Optimize is an ordinary publication: an update or delete prepared before it publishes gets ReadSetChanged on graph_head:main under the branch gate, exactly as after any other writer, and the caller retries. Inserts and loads reprepare on their bounded loop as before.

Lance fence

plan_compaction bins neighbouring candidate fragments only when the same indexes cover them. After a run whose fold changed coverage, the next run's plan can be non-empty on a table the previous run already compacted; the maintenance guide records it and the matrix asserts promotion rather than a single-run fixpoint.

Tests

• failpoints.rs: pre-publish failure leaves no residue (manifest and every linear HEAD unmoved, retry re-plans); lost publish acknowledgement and skipped promotion leave pending pins the next writer and cleanup promote; a pending-only vector table publishes no pin; a partial batch leaves no residue and the retry compacts both tables; the two branch-gate cells park on a rendezvous instead of polling for a sidecar; the queued strict delete reports the typed read-set conflict and its retry lands. Retired: ten recovery-machinery cells whose only producer was the Optimize sidecar (graph_with_optimize_work, leave_confirmed_optimize_sidecar) and the S3 sidecar-lifecycle cell, replaced by s3_optimize_pending_pin_is_promoted_by_the_next_write.
• detached_commit_matrix.rs: Writer::Optimize (four windows, three faults; the race cells show the exact-CAS loser, and a run after any recovery leaves Person promoted). Full run for the Optimize writer, all five recovery actors: 52 cells green.
• write_cost.rs: optimize_writes_no_control_object.
• maintenance.rs: the stale-config cell now asserts versions are preserved with the key left in place; the fold and materialize cells pass unchanged.
• search.rs: the issue-567 fixture builds its four-partition IVF index explicitly (the engine builds one partition and a fold keeps the count) and publishes it through repair.
• forbidden_apis.rs: stage_compaction, stage_index_fold registered; Optimize rows moved to the exact adapter.
• DST: catalog at 73 windows; the cost golden records the new Optimize counts (adapter deletes 2 to 0, puts 3 to 1, Lance gets 608 to 310). Two pins retired as vacuous by construction, each after its own search instrument found no schedule entering the shape: dst_ack_loss_client_retry (the recovery sidecar was the standard workload's only adapter-realm write whose lost acknowledgement failed an op; the failpoint suite owns the manifest-CAS lost-acknowledgement contract, and a Lance-realm ack-loss verb is the DST follow-up) and dst_keep_serving_widened_arbitration_no_false_reds (a deferral needs a live handle refused on a pending recovery operation, which no workload writer arms). The issue-554 availability panel stays.

Validation

Local: the canonical workspace test graph with failpoints, the engine failpoints suite (113), the full-mode Optimize matrix (52 cells), GQT corpus, DST suites with the regenerated golden, lance_surface_guards, fmt, both clippy graphs, docs/typos/agents/pins checks.

Follow-ups (step 5)

The adapter-realm write-fault verbs of the DST harness (ack loss, corrupt, lost and misdirected writes) now target only Optimize's best-effort adjacency artifact and the cleanup reaper, since the sidecars they were written for are gone; re-targeting them at the Lance realm, and removing the keep-serving machinery with the classifier, belong to the removal step.

[ragnorc]

Consolidated into #744, which carries this stack's complete history merged with current main and revalidated as one tree.