[WIP][iceberg] Support the v3 variant type with row lineage and REST catalog#8795
Draft
plusplusjiajia wants to merge 8 commits into
Draft
[WIP][iceberg] Support the v3 variant type with row lineage and REST catalog#8795plusplusjiajia wants to merge 8 commits into
plusplusjiajia wants to merge 8 commits into
Conversation
…lished row ids" This reverts commit 5b5668d.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add Iceberg v3
varianttype support to Paimon's Iceberg compatibility layer, with the row lineage and REST catalog upgrade it requires. The three are interdependent: v3variantmetadata without row lineage is spec-invalid (Iceberg 1.10+ won't parse metadata lackingnext-row-id/first-row-id), and row lineage without the REST upgrade still leavesrest-catalogstorage creating v2 tables that reject v3 features.IcebergDataFieldmaps PaimonVARIANT↔ Iceberg"variant"both ways. Before a snapshot is published,IcebergCommitCallbackrejects types that cannot produce readable metadata:variantwhenformat-version < 3(and as a partition key, unconditionally), and nanosecond timestamps on every version (Paimon writes them as ParquetINT96, which Iceberg cannot read astimestamp_ns). Validation runs before publish and before a tag is written.IcebergManifestFileMeta/IcebergDataFileMetagainfirst_row_id; ranges are assigned when writing the manifest list, mirroring Iceberg'sManifestListWriter. Tables upgraded from v2 get their surviving manifests assigned on the first v3 commit; regeneration continues past the highest published id so ids are never reused; rewritten/merged manifests keep unchanged rows'_row_id.variant). Creation passes the configuredformat-version; a populated table whose version differs is dropped and recreated. Fallout: Hadoop 3.3.6 forFileSystem.openFile, jetty/commons-lang3 test-classpath alignment,RemoveSnapshot→RemoveSnapshots.