Skip to content

feat(relational): add authoritative packed base-row storage - #502

Merged
ajroetker merged 155 commits into
mainfrom
feature/relational-core
Sep 12, 2026
Merged

ajroetker merged 155 commits into
mainfrom
feature/relational-core

Conversation

@ajroetker

@ajroetker ajroetker commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

What changed

  • carries relational storage mode and the typed column catalog through the durable runtime schema
  • keeps the hardened schema validation, exact document-to-cell projection, and versioned packed-row codec from the current mega-branch tree
  • adds a dedicated relational base-row keyspace and narrow relational_store facade
  • makes packed rows authoritative for writes, point reads, scans, filtering, identity, resolution, enrichment, and existing derived-index/rebuild consumers
  • supports normal transaction commit, one-shot/orphaned-intent recovery, replay, TTL deletion, split/merge cleanup, and portable backup/restore without creating generic JSON primary rows
  • rejects storage-mode changes and first-schema reinterpretation of existing physical rows
  • validates every final post-transform relational write at the DB boundary against one generation-scoped compiled public validator, covering direct/embedded batches, transactions, replicated apply, and recovery without reparsing schema JSON or rebuilding physical-field constraints on the write hot path
  • commits runtime schema, version history, and public validator JSON in one durable transaction; all fallible validator and recovery-schema preparation happens before publication
  • owns and synchronizes transaction-recovery column generations so the worker cannot race schema replacement or retain freed column slices
  • replicates the complete runtime/public schema pair in HA metadata v2, verifies the pair before standby publication, and remains compatible with v1 records
  • preflights portable archives with a bounded-memory two-pass validator that binds format version, row mode, runtime schema, and public schema before destination mutation; structurally valid but schema-invalid packed rows are rejected
  • stages embedded and Lite imports outside the live DB, rebinds identity in staging, rejects non-empty targets, and publishes every portable block in one atomic store transaction under a short apply-lock window
  • reloads durable schema ownership and every schema-dependent runtime context before an already-open embedded, Lite, or server restore becomes visible or starts rebuilding indexes

Scope

This folds the former R2 authoritative-base-row slice into #502 so the PR lands one minimal, usable relational storage path instead of an admitted schema and unused codec. The code was ported surgically from the current final tree of combine-pr-141-143-144, not reconstructed by cherry-picking its historical commits.

Still excluded:

  • relational primary/unique/covering/expression index lifecycle
  • checks, defaults, generated values, collations, arrays, temporal constraints, and distributed foreign keys
  • typed row HTTP APIs
  • relational SQL binding and lowering
  • catalog/namespace routing

Validation

  • complete post-merge bounded storage aggregate: 3,128 passed, 8 skipped, 0 failed, 0 leaked (16/16 build steps)
  • complete HA suite: 371 passed, 0 failed, 0 leaked, including v1 metadata compatibility and promoted-standby constraint enforcement
  • API HTTP/runtime suite: 39 passed, 0 failed, 0 leaked
  • C ABI suite: 11 passed, 0 failed, 0 leaked
  • embedded and standalone embedded package checks: 4 passed, 0 failed, 0 leaked
  • focused direct-batch constraint, reopen, portable schema-binding, atomic import, and immediate post-restore read/write regressions passed with no leaks
  • unit-storage-test-audit, storage compile, formatting, E2E collection, and git diff --check passed after the final main merge
  • exact-head Zig Tests run 32546016284 passed: x86_64 hermetic/TLA+, ARM64 codec, Antfly E2E, inference E2E, and low-FD regressions
  • current origin/main (46b702a8e) is merged cleanly

@ajroetker
ajroetker force-pushed the feature/relational-core branch from 430ad08 to 2bc3374 Compare August 19, 2026 16:51
ajroetker and others added 27 commits August 19, 2026 10:21
# Conflicts:
#	zig/pkg/antfly/src/storage/db/algebraic/schema_capability.zig
Integrate native generation storage with persistent run directories, logical publication identity, snapshot-bound backfill, and streamed manifest recovery. Serialize split ownership behind the durability lane, use the borrowed runtime clock, and run VOPR maintenance on its scheduler. Cover publication intervals, bulk admission, replay snapshots, and declared backup integrity.
Normalize empty contracts to the table-creation schema and atomically persist public/runtime forms. Update restore fixtures to respect LSM writer ownership and native vector readiness; retain cached status fixture authority.
…hots safely

Keep bulk selections and delta certificates across admission denial, prepare scheduler metadata in bounded off-lock work, and defer GC wakes without delaying durability. Revalidate identities after publication and use representation-independent async memtable access. Add allocation, lifecycle, concurrency and scaling regressions.
Keep prepared domain and GC work across admission denial with bounded preparation, validation, retry deadlines, and reclamation. Consume pinned immutable metadata directly during persisted execution, index scheduler conflicts, and preserve foreground time/job budgets. Add regressions and reproducible handoff/admission benchmarks.
# Conflicts:
#	zig/build.zig
#	zig/pkg/antfly/src/storage/lsm_backend/compaction.zig
@ajroetker
ajroetker marked this pull request as ready for review September 12, 2026 02:05
@ajroetker
ajroetker merged commit aefe3ba into main Sep 12, 2026
30 of 34 checks passed
@ajroetker
ajroetker deleted the feature/relational-core branch September 12, 2026 02:05
ajroetker added a commit that referenced this pull request Sep 12, 2026
The README still described Antfly as built on etcd's raft library and
accelerated via go-highway, neither of which is true of the Zig runtime.
Rewrite the opening around the zero-dependency engine, and bring the
feature list up to date:

- add late interaction and fusion to hybrid search
- replace the go-highway bullet with CPU, Metal, and CUDA backends
- add Lite, hot standby, and serverless deployment shapes
- add the C API, Go binding, in-browser WASM build, Wasmtime
  extensions, and fine-tuning
- add an "In progress" section linking the relational, catalog, and
  SQL/pgwire PRs (#502, #691, #145)
- add the Rust SDK, memoryaf, and antflylite to the tables
- fix the three broken rs/pgaf links (crate lives at rs/crates/pgaf)
- describe owned dependencies and the VOPR harness under Architecture,
  and list the LSM lifecycle and trace-validated etcdraft TLA+ specs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant