Skip to content

feat: build shared cell-group multi-Raft runtime - #160

Open
forhappy wants to merge 248 commits into
mainfrom
codex/shared-cell-groups
Open

forhappy wants to merge 248 commits into
mainfrom
codex/shared-cell-groups

Conversation

@forhappy

@forhappy forhappy commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Cellar’s production host still maps each cell to a Raft group. This PR builds its replacement: many independent SQLite cells share a physical Raft group, while groups share node scheduling, storage, transport and resource budgets. The public SDK/fleet cutover remains unfinished.

The executable package in docs/design/multi-raft/cell-groups/ defines 16 tasks, 163 required cases, 14 invariants and nine qualification profiles. CG-01 through CG-05 passed their scoped native gates; CG-06 remains in progress.

Delivered boundaries:

  • Separate logical cell histories, physical group identities and paged range routing. Complete manifests retain enrollment, versions, receipts, effects and SQLite/LTX image roots.
  • Real raft-rs owners share node workers, persistence batches, authenticated TLS pools and bounded admission. SQL execution preserves command/query authority, receipt reconciliation and physical ownership through cancellation.
  • Complete group checkpoints, authenticated learner installation, joint membership changes and retained recovery pass the native gates. Complete checkpoint/tail archives verify every dependency before S3 coverage; retained ownership protects remote collection and local log reclamation.
  • Exact recovery intents bind the held archive, replacement identities and topology mutation. Archive verification precedes fencing; topology/lifecycle CAS boundaries close old bootstrap and competing publication. The external fencing-authority receipt is independently verified, including on resume.
  • The latest change prepares that fenced cut into complete roots under fresh cell histories. SQLite bytes stay exact; materialized image identities and extent history metadata are rebound. Imported roots remain pending across reopen/snapshot paths, and application authorities refuse admission. Import limits and physical pins survive cloned roots and cancellation during disk sync. Deterministic activation preparation binds the operation and seed commitment; leader admission and actual replacement-quorum activation remain to be connected.

Local qualification at runtime source 7052d218d60181d7b0bb5a4c12686e80e294a61c:

  • Seven native shared-group suites: 246 passing cases. Recovery-related unit tests: 36 passing cases, including all four new lifecycle/admission cases. Eleven existing disaster-recovery API cases pass.
  • The exact import case passes against real RustFS in both materialized and extent modes, with seven signed S3 probes. It verifies all-cell bytes, new histories/witnesses, old-token rejection, admission limits and stable retries. The fence authority is a contract fixture; Docker voter fencing and serving after total disk loss remain pending.
  • Locked all-target Clippy for Cellar/logic; format/layout over 1971 Rust files; native dependency, documentation, pinned-source and plan checks; 37 plan tests and five backend-controller tests pass.
  • Independent audit verifies 2588 source inputs against Git blobs/symlinks, copied executables, exact selection and credential exclusion. The owned RustFS backend/profile are stopped with data retained; the default Docker context is preserved.

Evidence and retained predecessors are recorded in docs/design/multi-raft/cell-groups/progress.md. The scoped qualification SHA256 is 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. Recovery metadata uses the internal marker-1 hard cutover with no reader for older intermediate manifest/archive shapes. Earlier socket/deadline/host-pressure and slow-reader timing limitations remain documented; no causal fix is claimed for them.

MVP implementation priorities:

  1. Finish CG-06: durable fresh genesis and physical seed identity binding, one publication under the recovery lock, permit-bound activation committed/applied through a fresh quorum, and the Docker-fenced RustFS drill after erasing every owned voter disk. Prove registered receipt retries, stored minima/client floors, old-token rejection and zero duplicate callbacks.
  2. CG-08 background effects and safe idle SQLite caches, then CG-07 public SDK/fleet cutover to shared groups.
  3. Follow-on work retains CG-09–12 transfers, split/merge, placement and hibernation; CG-14/13/15 workload infrastructure, managed recovery and crash/load qualification; and CG-16 final legacy removal and the complete release matrix.

PreparedCellGroupRecovery currently supplies private data preparation and grants no engine, bootstrap, voting or serving permission. The original fenced_group_seed_new_incarnations case and full CG-06 gate remain open. Sustained-load, independent-domain and production hard-cutover qualification remain required.

The branch contains fetched origin/main at 8a93ffd72e914f8f50fed3e087efd49d379e4c6e. The requested MR-13/MR-14 progress JSON basenames remain absent from the PR diff.

@forhappy forhappy changed the title docs: select shared cell groups as the Raft replacement docs: add executable cell-group multi-Raft delivery plan Oct 29, 2026
@forhappy forhappy changed the title docs: add executable cell-group multi-Raft delivery plan feat: add cell-group identities and version contracts Oct 29, 2026
@forhappy forhappy changed the title feat: add cell-group identities and version contracts feat: define cell-group multi-Raft delivery and wire contracts Oct 29, 2026
@forhappy forhappy changed the title feat: define cell-group multi-Raft delivery and wire contracts feat: build cell-group contracts and manifest storage Oct 29, 2026
@forhappy forhappy changed the title feat: build cell-group contracts and manifest storage feat: persist cell-group manifests through Raft Engine Oct 29, 2026
@forhappy forhappy changed the title feat: persist cell-group manifests through Raft Engine feat: persist and incrementally prepare cell-group manifests Oct 29, 2026
@forhappy forhappy changed the title feat: persist and incrementally prepare cell-group manifests feat: persist cell-group manifests with retention and reclamation Oct 29, 2026
@forhappy forhappy changed the title feat: persist cell-group manifests with retention and reclamation feat: build cell-group durability and enrollment Oct 29, 2026
@forhappy forhappy changed the title feat: build cell-group durability and enrollment feat: build cell-group durability, enrollment and range directory Oct 29, 2026
Leader readiness cannot become true on followers. Require the selected cut, retired log debt and released source quota on every voter.
The composite restart case overflows the default test stack. Use the existing non-inlined fixture constructor, preserving all assertions and resource limits.
Expose the fixture to its enclosing module and require fresh TLS writes after receipt recovery. Preserve the original physical reclamation assertions and limits.
Bind recovery to a verified complete archive and serialize its permanent fence with topology and fleet import. Resume only the same intent and independently verify the winning receipt.

This barrier grants data and fencing proofs only. Fresh-genesis seeding, receipt rebinding, activation and replacement-quorum serving remain required before disaster recovery is complete.
A by-value recovery lock enlarged common directory futures and made the retained learner-replacement test overflow its default stack. The previous copied executable passes and the candidate reproduces the failure; boxing the optional lock restores the exact test without changing its stack limit or serialized record.
Record the source-bound native and RustFS checks, retained failures and
independent audit. Keep CG-06 in progress until fresh-genesis restoration,
activation and replacement-quorum serving pass their original gate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant