[lake/paimon] Support custom Paimon lake table path

[wzx140]

Summary

• Support mapping a Fluss table to a Paimon lake table with a custom database and/or table name through table.datalake.database-name and table.datalake.table-name.
• Fall back to the corresponding Fluss database or table name when either mapping option is omitted.
• Use the resolved lake table path for lake table creation and alteration, Paimon tiering writes and commits, Flink $lake and metadata-table reads, and Flink union reads.
• Validate custom lake database and table names, and restrict custom lake table path mapping to Paimon.
• Allow path-changing mapping alterations only when datalake is disabled and no tiering progress exists. Reject such alterations after datalake is enabled or tiering progress has been recorded.
• Keep Spark log-only fallback working before a lake snapshot exists, but reject actual Spark lake reads for tables with custom lake path mapping in this phase.
• Document the new mapping options, defaults, and alteration restrictions.

Test Coverage

• FlinkCatalogTest#testGetLakeTableWithCustomLakePath: verifies Flink catalog resolves $lake and metadata-table reads to the mapped Paimon database and table.
• FlinkUnionReadPrimaryKeyTableITCase#testUnionReadWithCustomLakeTablePath: verifies Flink reads from the mapped Paimon table and merges lake data with the Fluss tail.
• PaimonTieringITCase#testTieringWithCustomLakeTablePath: verifies Paimon tiering writes to the mapped table and records the expected snapshot offsets.
• LakeEnabledTableCreateITCase#testAlterLakePathOptionsValidation: verifies mapping options can be changed while datalake is disabled and no tiering progress exists, are used when enabling datalake, and cannot change the resolved lake path after datalake is enabled or tiering progress exists.
• LakeEnabledTableCreateITCase#testAlterLakePathOptionsOnlyForPaimon: verifies custom lake table path mapping is rejected for non-Paimon lake formats.
• TableDescriptorValidationTest: verifies custom lake database and table names are validated and valid mapped names are accepted.
• SparkLakePaimonLogTableReadTest: verifies Spark can fall back to log-only reads before a lake snapshot exists and rejects actual lake reads when custom lake path mapping is configured.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for mapping Fluss lake-enabled tables to a custom Paimon lake table database/table name, and routes lake reads/writes/alter operations through the resolved lake table path while validating/guarding unsupported connectors.

Changes:

• Introduce table.datalake.database-name / table.datalake.table-name and LakeTableUtil to resolve the effective lake TablePath.
• Route Flink $lake reads, coordinator lake-catalog operations, and Paimon tiering writer/committer to the mapped lake table path.
• Clear persisted tiering progress when lake table path mapping changes; add Hive-catalog IT coverage and Spark connector restriction for custom mapping.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pom.xml	Adds Hive/Derby/Datanucleus version properties used by new Hive-catalog IT.
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/SparkTable.scala	Rejects custom lake path usage in Spark connector (currently).
fluss-server/src/test/java/org/apache/fluss/server/zk/data/lake/LakeTableHelperTest.java	Adds test for clearing lake progress snapshot + offset files.
fluss-server/src/test/java/org/apache/fluss/server/coordinator/CoordinatorEventProcessorTest.java	Adds test to ensure altering lake path clears committed progress when datalake disabled.
fluss-server/src/main/java/org/apache/fluss/server/zk/data/lake/LakeTableHelper.java	Adds clearLakeTableProgress API to delete ZK entry + discard snapshot metadata.
fluss-server/src/main/java/org/apache/fluss/server/zk/ZooKeeperClient.java	Adds deleteLakeTable helper to remove lake progress node.
fluss-server/src/main/java/org/apache/fluss/server/utils/TableDescriptorValidation.java	Validates custom lake path support + alter-property restrictions (including paimon.path rules).
fluss-server/src/main/java/org/apache/fluss/server/coordinator/MetadataManager.java	Routes lake catalog create/alter/schema ops via resolved lake table path.
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorService.java	Routes lake table creation via resolved lake table path.
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorEventProcessor.java	Clears lake table progress when resolved lake path changes.
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/tiering/PaimonTieringITCase.java	Adds tiering IT verifying custom mapped lake table path.
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/flink/FlinkUnionReadPrimaryKeyTableITCase.java	Adds union-read IT validating $lake reads with custom mapping.
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/LakeEnabledTableCreateWithHiveCatalogITCase.java	Adds Hive metastore IT verifying paimon.path alteration rules with mapping options.
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/LakeEnabledTableCreateITCase.java	Adds IT validating alter behavior of lake path options when enabled/disabled.
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/utils/PaimonConversions.java	Allows path option to be set (removes CoreOptions.PATH from unsettable).
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/PaimonLakeWriter.java	Uses resolved lake table path when opening the Paimon table for writing.
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/PaimonLakeCommitter.java	Uses resolved lake table path for table access and stats computation.
fluss-lake/fluss-lake-paimon/pom.xml	Adds Hive/Derby/Datanucleus test dependencies for Hive-catalog IT.
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/catalog/FlinkCatalogTest.java	Adds Flink catalog test for $lake resolution with custom mapping.
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/utils/LakeSourceUtils.java	Creates lake sources using resolved lake table path.
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/LakeTableFactory.java	Resolves Flink lake ObjectIdentifier using mapped database/table + metadata suffix.
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/catalog/FlinkCatalog.java	Routes lake table lookup to mapped lake path and correct metadata-table naming.
fluss-common/src/main/java/org/apache/fluss/metadata/TableInfo.java	Exposes getLakeTablePath() / hasCustomLakePath() helpers.
fluss-common/src/main/java/org/apache/fluss/metadata/LakeTableUtil.java	New utility to resolve lake TablePath and metadata table names with mapping.
fluss-common/src/main/java/org/apache/fluss/config/FlussConfigUtils.java	Allows lake mapping options to be altered (when permitted).
fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java	Adds new config options for lake database/table mapping.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wzx140]

@luoyuxia Could you please help review this one?

[luoyuxia]

@wzx140 Thanks for the work!

#1953 only asks for custom database mapping. This PR goes quite a bit beyond that scope — on top of database mapping it also adds table-name mapping, custom paimon.path, Spark-read rejection, tiering-progress clearing, and a new Hive-metastore ITCase (pulling in hive-exec / datanucleus / derby test deps), ending up at ~900 lines across 26 files.

Could we start with just the database-name mapping that #1953 needs and confirm whether that already satisfies the use case? If so, the rest (table-name mapping, custom paimon.path, …) can come as separate follow-ups once we agree they're actually needed. That keeps this PR small, scoped to the issue, and free of the heavy Hive test dependencies.

[wzx140]

@luoyuxia I was looking at both #1953 and #1952 together when working on this.

The database mapping part is already in progress, but internally the per-table mapping is actually the higher-priority requirement for us.

To keep the scope manageable, I’m happy to remove the custom paimon.path, Hive ITCase / test dependency changes, and other unrelated pieces from this PR. Then we can focus the review on the mapping functionality first and iterate on the rest separately if needed.

Does that sound reasonable to you?

[luoyuxia]

@luoyuxia I was looking at both #1953 and #1952 together when working on this.

The database mapping part is already in progress, but internally the per-table mapping is actually the higher-priority requirement for us.

To keep the scope manageable, I’m happy to remove the custom paimon.path, Hive ITCase / test dependency changes, and other unrelated pieces from this PR. Then we can focus the review on the mapping functionality first and iterate on the rest separately if needed.

Does that sound reasonable to you?

Thanks. It sounds reasonable to me

[wzx140]

@luoyuxia I’ve removed the unrelated changes. The PR is now around 600 lines across 22 files, with roughly half of the diff consisting of new unit tests, so the implementation itself is significantly smaller than before.

If you have time, I’d appreciate another review.

[luoyuxia]

Would it meet your requirements if these options could only be set when creating the table and could not be modified afterward?

Supporting path changes would introduce additional complexity around existing lake tables, tiering progress, and data migration

[wzx140]

@luoyuxia Yes, that would meet my requirements.

The current expectation is that once the Paimon table is created, these name mapping options become immutable. Allowing them to be modified afterward would require handling much more complex scenarios, including existing lake tables, tiering progress, and potential data migration, which we’d prefer to avoid.

[luoyuxia]

also add the option to document

[wzx140]

done

[luoyuxia]

@wuchong Could you please take a look at the public API design here?

This PR introduces two table options, table.datalake.database-name and table.datalake.table-name, to map a Fluss table to a Paimon table with a different database or table name.