Skip to content

Add lake-native Parquet and Iceberg serving foundations - #521

Merged
ajroetker merged 25 commits into
mainfrom
feature/row-source-contract
Aug 23, 2026
Merged

ajroetker merged 25 commits into
mainfrom
feature/row-source-contract

Conversation

@ajroetker

@ajroetker ajroetker commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • add the storage-independent row-source contract and local/external adapters
  • port the complete lake-native design from the current mega-branch state in zig/LAKES.md
  • add snapshot-pinned Parquet and Iceberg inventory, metadata, footer, page, row-group, predicate, delete-file, and row decoding
  • add range coalescing, bounded cache behavior, object checksum validation, configured filesystem/S3/GCS credentials, and fail-closed snapshot validation
  • persist external base sources and artifact references in manifests and resolve/pin external publication plans through the catalog/runtime stack
  • add row-fragment publication plus text, sparse, vector, graph, and algebraic sidecar build/selection/rebuild/promotion/GC paths
  • add explain/compatibility surfaces and a focused lake-test target covering the complete standalone lake stack

Porting approach

This is a surgical state port, not a cherry-pick series. Every lake-owned implementation and design file comes from the current origin/combine-pr-141-143-144 tree. Shared files were merged onto current main so the PR retains the object-store hardening already merged through #504 and does not regress newer serverless/runtime work.

The lake-owned files match the mega branch state except for compatibility with #504's expanded object-checksum contract and the review hardening listed below. Public typed relational HTTP/SQL routing remains outside this standalone PR because that adapter depends on the broader relational query and catalog surfaces intentionally excluded from the minimal #502 boundary. The complete standalone lake engine, publication stack, and row-source bridge are present here; zig/LAKES.md now marks mega-only public routing claims explicitly.

Review hardening

  • publish the initial external manifest atomically even when a read-only external table has no WAL records
  • publish changed .current snapshots without document-segment fallbacks, while avoiding duplicate versions for unchanged inventories
  • fail closed when a .current binding has no configured discovery resolver
  • validate GCS CRC32C and supported S3 full-object checksums without heap allocation, and safely ignore composite or not-yet-portable optional checksums
  • keep empty external manifests honest by publishing no document lineage or search-source claims before sidecars exist

Validation

  • zig build lake-test -fincremental --summary all — 275 passed
  • zig build serverless-test -fincremental --summary all — 60 passed, 2 opt-in cloud tests skipped
  • make zig-generated-check
  • zig fmt on all changed Zig sources
  • git diff --check

Depends on the object-store semantics merged in #504. The original row-source layer remains the L1 foundation in the first commit of this PR.

@ajroetker ajroetker changed the title Add row-source contract for lake scans Add lake-native Parquet and Iceberg serving stack Aug 21, 2026
@ajroetker ajroetker changed the title Add lake-native Parquet and Iceberg serving stack Add lake-native Parquet and Iceberg serving foundations Aug 21, 2026
@ajroetker

Copy link
Copy Markdown
Contributor Author

Scope update: the unrelated index repair-lifecycle change previously in commit 34249bf was removed from this branch. Its production-hardened replacement, including bounded cleanup, corrupt-metadata handling, restart coverage, and the stateful delete/recreate regression, is now isolated in #532.

@ajroetker
ajroetker marked this pull request as ready for review August 23, 2026 00:52
@ajroetker
ajroetker merged commit 1ed35c7 into main Aug 23, 2026
28 of 30 checks passed
@ajroetker
ajroetker deleted the feature/row-source-contract branch August 23, 2026 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant