diff --git a/docs/en/engines/table-engines/integrations/iceberg.md b/docs/en/engines/table-engines/integrations/iceberg.md index 7a93d3f18c35..4cd0667f68cc 100644 --- a/docs/en/engines/table-engines/integrations/iceberg.md +++ b/docs/en/engines/table-engines/integrations/iceberg.md @@ -131,9 +131,16 @@ ClickHouse supports reading Iceberg tables that use the following deletion metho - [Position deletes](https://iceberg.apache.org/spec/#position-delete-files) - [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) (supported from version 25.8+) +- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) stored in Puffin files (Iceberg v3, read-only) -The following deletion method is **not supported**: -- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (introduced in v3) +The following limitations apply to deletion vectors: + +- Only `deletion-vector-v1` Puffin blobs are supported +- Data files must be in Parquet format +- Column-scoped deletion vectors (`fields` in the Puffin blob) are not supported +- Writing deletion vectors is not supported + +Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the puffin file has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the puffin file. The cache can be cleared with `SYSTEM DROP PUFFIN_FILES_CACHE`. ### Basic usage {#basic-usage} ```sql diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 38e16b525be6..c8be12560afc 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -110,6 +110,8 @@ The supported formats are: | [AvroConfluent](./formats/Avro/AvroConfluent.md) | ✔ | ✔ | | [Parquet](./formats/Parquet/Parquet.md) | ✔ | ✔ | | [ParquetMetadata](./formats/Parquet/ParquetMetadata.md) | ✔ | ✗ | +| [Puffin](./formats/Puffin/Puffin.md) | ✔ | ✗ | +| [PuffinMetadata](./formats/Puffin/PuffinMetadata.md) | ✔ | ✗ | | [Arrow](./formats/Arrow/Arrow.md) | ✔ | ✔ | | [ArrowStream](./formats/Arrow/ArrowStream.md) | ✔ | ✔ | | [ORC](./formats/ORC.md) | ✔ | ✔ | diff --git a/docs/en/interfaces/formats/Puffin/Puffin.md b/docs/en/interfaces/formats/Puffin/Puffin.md new file mode 100644 index 000000000000..7f9bf4b78ad9 --- /dev/null +++ b/docs/en/interfaces/formats/Puffin/Puffin.md @@ -0,0 +1,50 @@ +--- +description: 'Documentation for the Puffin format' +input_format: true +output_format: false +keywords: ['Puffin'] +sidebar_label: 'Puffin' +sidebar_position: 1 +slug: /interfaces/formats/Puffin +title: 'Puffin' +doc_type: 'reference' +--- + +## Description {#description} + +Input format for reading [Apache Iceberg Puffin](https://iceberg.apache.org/puffin-spec/) files. + +The format exposes deleted row positions from `deletion-vector-v1` blobs. Other blob types (for example `apache-datasketches-theta-v1`) are skipped. +If a puffin file contains multiple `deletion-vector-v1` blobs, the format outputs one row per such blob. + +Fixed output columns: +- `referenced_data_file` (`String`) - location of the data file the deletion vector applies to (`referenced-data-file` blob property) +- `deleted_rows` (`Array(UInt64)`) - 64-bit row positions deleted according to the deletion vector roaring bitmap + +Deletion vectors whose declared `cardinality` exceeds an absolute materialization ceiling are rejected when `deleted_rows` is requested. Footer `deletion-vector-v1` properties (including that `cardinality` parses as an unsigned integer) are always validated. Selecting only `referenced_data_file` skips on-disk payload I/O and therefore also skips envelope, CRC, roaring deserialize, and the materialization ceiling — intentionally, so a path-only projection does not read up to the blob-size cap. + +On-disk `deletion-vector-v1` blob length is bounded by an absolute ceiling (aligned with Iceberg's 2 GiB content-size check). When `deleted_rows` is requested, the reader peeks the envelope header (combined length and magic) before allocating the full payload; CRC is verified after the bounded read. + +LZ4-compressed and uncompressed puffin footers are supported. Footer payload size (and declared LZ4 content size) is bounded by a compression ratio where applicable and an absolute ceiling; oversized footers are rejected before allocation. + +Only a subset of output columns can be requested. A user-provided structure with unexpected column names or types is rejected when the format is created. + +## Example usage {#example-usage} + +Read deleted row positions with the referenced data file: + +```sql +SELECT referenced_data_file, deleted_rows +FROM file(deletes.puffin, Puffin); +``` + +Expand deleted positions into individual rows: + +```sql +SELECT referenced_data_file, row_number +FROM file(deletes.puffin, Puffin) +ARRAY JOIN deleted_rows AS row_number +ORDER BY referenced_data_file, row_number; +``` + +Use `PuffinMetadata` to inspect footer blob descriptors before reading deletion vectors. diff --git a/docs/en/interfaces/formats/Puffin/PuffinMetadata.md b/docs/en/interfaces/formats/Puffin/PuffinMetadata.md new file mode 100644 index 000000000000..240d217869b6 --- /dev/null +++ b/docs/en/interfaces/formats/Puffin/PuffinMetadata.md @@ -0,0 +1,42 @@ +--- +description: 'Documentation for the PuffinMetadata format' +input_format: true +output_format: false +keywords: ['PuffinMetadata'] +sidebar_label: 'PuffinMetadata' +sidebar_position: 2 +slug: /interfaces/formats/PuffinMetadata +title: 'PuffinMetadata' +doc_type: 'reference' +--- + +## Description {#description} + +Special input format for reading [Apache Iceberg Puffin](https://iceberg.apache.org/puffin-spec/) file footer metadata. +It outputs one row per blob entry from the footer `BlobMetadata` list. + +Fixed output columns: +- `blob_type` (`String`) - blob type, for example `deletion-vector-v1` +- `snapshot_id` (`Int64`) - snapshot id of the blob +- `sequence_number` (`Int64`) - sequence number of the blob +- `fields` (`Array(Int32)`) - list of field ids the blob applies to +- `offset` (`Int64`) - offset of the blob payload in the file +- `length` (`Int64`) - length of the blob payload in bytes +- `compression_codec` (`String`) - compression codec of the blob payload, if present +- `properties` (`Map(String, String)`) - blob-specific properties + +Optional top-level `FileMetadata.properties` in the footer (for example `created-by`) are type-checked when present but are not returned as columns. If the key is present it must be a JSON object with string values (null is rejected). + +LZ4-compressed and uncompressed puffin footers are supported. Footer payload size (and declared LZ4 content size) is bounded by a compression ratio where applicable and an absolute ceiling; oversized footers are rejected before allocation. + +## Example usage {#example-usage} + +Inspect footer blobs: + +```sql +SELECT blob_type, snapshot_id, sequence_number, offset, length, compression_codec, + mapKeys(properties), mapValues(properties) +FROM file(deletes.puffin, PuffinMetadata); +``` + +Pair with the `Puffin` format to read `deletion-vector-v1` blob payloads. diff --git a/docs/en/sql-reference/statements/system.md b/docs/en/sql-reference/statements/system.md index 69b2b59cb0ae..9707c96b0d15 100644 --- a/docs/en/sql-reference/statements/system.md +++ b/docs/en/sql-reference/statements/system.md @@ -109,6 +109,10 @@ Clears the per-URL Confluent Schema Registry caches used by the `AvroConfluent` Clears the parquet metadata cache. +## SYSTEM DROP PUFFIN_FILES_CACHE {#drop-puffin-files-cache} + +Clears the Puffin files cache used for parsed Iceberg puffin file content such as deletion vectors. + ## SYSTEM CLEAR|DROP TEXT INDEX CACHES {#drop-text-index-caches} Clears the text index's header, dictionary and postings caches. diff --git a/docs/en/sql-reference/table-functions/iceberg.md b/docs/en/sql-reference/table-functions/iceberg.md index 22f567d590cd..9ab0b5b8071c 100644 --- a/docs/en/sql-reference/table-functions/iceberg.md +++ b/docs/en/sql-reference/table-functions/iceberg.md @@ -120,11 +120,20 @@ ClickHouse supports time travel for Iceberg tables, allowing you to query histor ## Processing of tables with deleted rows {#deleted-rows} -Currently, only Iceberg tables with [position deletes](https://iceberg.apache.org/spec/#position-delete-files) are supported. +ClickHouse supports reading Iceberg tables that use the following deletion methods: -The following deletion methods are **not supported**: -- [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) -- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (introduced in v3) +- [Position deletes](https://iceberg.apache.org/spec/#position-delete-files) +- [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) (supported from version 25.8+) +- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) stored in Puffin files (Iceberg v3, read-only) + +The following limitations apply to deletion vectors: + +- Only `deletion-vector-v1` Puffin blobs are supported +- Data files must be in Parquet format +- Column-scoped deletion vectors (`fields` in the Puffin blob) are not supported +- Writing deletion vectors is not supported + +Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the puffin file has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the puffin file. The cache can be cleared with `SYSTEM DROP PUFFIN_FILES_CACHE`. ### Basic usage {#basic-usage} diff --git a/programs/local/LocalServer.cpp b/programs/local/LocalServer.cpp index ded35800452f..e2a08bfe6c7b 100644 --- a/programs/local/LocalServer.cpp +++ b/programs/local/LocalServer.cpp @@ -172,6 +172,10 @@ namespace ServerSetting extern const ServerSettingsUInt64 parquet_metadata_cache_size; extern const ServerSettingsUInt64 parquet_metadata_cache_max_entries; extern const ServerSettingsDouble parquet_metadata_cache_size_ratio; + extern const ServerSettingsString puffin_files_cache_policy; + extern const ServerSettingsUInt64 puffin_files_cache_size; + extern const ServerSettingsUInt64 puffin_files_cache_max_entries; + extern const ServerSettingsDouble puffin_files_cache_size_ratio; extern const ServerSettingsUInt64 max_active_parts_loading_thread_pool_size; extern const ServerSettingsUInt64 max_io_thread_pool_free_size; extern const ServerSettingsUInt64 max_io_thread_pool_size; @@ -1540,6 +1544,17 @@ void LocalServer::processConfig() global_context->setParquetMetadataCache(parquet_metadata_cache_policy, parquet_metadata_cache_size, parquet_metadata_cache_max_entries, parquet_metadata_cache_size_ratio); #endif + String puffin_files_cache_policy = server_settings[ServerSetting::puffin_files_cache_policy]; + size_t puffin_files_cache_size = server_settings[ServerSetting::puffin_files_cache_size]; + size_t puffin_files_cache_max_entries = server_settings[ServerSetting::puffin_files_cache_max_entries]; + double puffin_files_cache_size_ratio = server_settings[ServerSetting::puffin_files_cache_size_ratio]; + if (puffin_files_cache_size > max_cache_size) + { + puffin_files_cache_size = max_cache_size; + LOG_INFO(log, "Lowered Puffin files cache size to {} because the system has limited RAM", formatReadableSizeWithBinarySuffix(puffin_files_cache_size)); + } + global_context->setPuffinFilesCache(puffin_files_cache_policy, puffin_files_cache_size, puffin_files_cache_max_entries, puffin_files_cache_size_ratio); + Names allowed_disks_table_engines; splitInto<','>(allowed_disks_table_engines, server_settings[ServerSetting::allowed_disks_for_table_engines].value); global_context->setAllowedDisksForTableEngines(std::unordered_set(allowed_disks_table_engines.begin(), allowed_disks_table_engines.end())); diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index f449c2cc6858..8231e30d087c 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -288,6 +288,10 @@ namespace ServerSetting extern const ServerSettingsUInt64 parquet_metadata_cache_size; extern const ServerSettingsUInt64 parquet_metadata_cache_max_entries; extern const ServerSettingsDouble parquet_metadata_cache_size_ratio; + extern const ServerSettingsString puffin_files_cache_policy; + extern const ServerSettingsUInt64 puffin_files_cache_size; + extern const ServerSettingsUInt64 puffin_files_cache_max_entries; + extern const ServerSettingsDouble puffin_files_cache_size_ratio; extern const ServerSettingsUInt64 io_thread_pool_queue_size; extern const ServerSettingsBool jemalloc_enable_global_profiler; extern const ServerSettingsBool jemalloc_collect_global_profile_samples_in_trace_log; @@ -2276,6 +2280,16 @@ try } global_context->setParquetMetadataCache(parquet_metadata_cache_policy, parquet_metadata_cache_size, parquet_metadata_cache_max_entries, parquet_metadata_cache_size_ratio); #endif + String puffin_files_cache_policy = server_settings[ServerSetting::puffin_files_cache_policy]; + size_t puffin_files_cache_size = server_settings[ServerSetting::puffin_files_cache_size]; + size_t puffin_files_cache_max_entries = server_settings[ServerSetting::puffin_files_cache_max_entries]; + double puffin_files_cache_size_ratio = server_settings[ServerSetting::puffin_files_cache_size_ratio]; + if (puffin_files_cache_size > max_cache_size) + { + puffin_files_cache_size = max_cache_size; + LOG_INFO(log, "Lowered Puffin files cache size to {} because the system has limited RAM", formatReadableSizeWithBinarySuffix(puffin_files_cache_size)); + } + global_context->setPuffinFilesCache(puffin_files_cache_policy, puffin_files_cache_size, puffin_files_cache_max_entries, puffin_files_cache_size_ratio); Names allowed_disks_table_engines; splitInto<','>(allowed_disks_table_engines, server_settings[ServerSetting::allowed_disks_for_table_engines].value); @@ -2705,6 +2719,7 @@ try #if USE_PARQUET global_context->updateParquetMetadataCacheConfiguration(config(), max_cache_size_in_bytes); #endif + global_context->updatePuffinFilesCacheConfiguration(config(), max_cache_size_in_bytes); } #if USE_SSL diff --git a/src/Access/Common/AccessType.h b/src/Access/Common/AccessType.h index f8c472987345..fd1dc6b4dbae 100644 --- a/src/Access/Common/AccessType.h +++ b/src/Access/Common/AccessType.h @@ -318,6 +318,7 @@ enum class AccessType : uint8_t M(SYSTEM_DROP_ICEBERG_METADATA_CACHE, "SYSTEM CLEAR ICEBERG_METADATA_CACHE, SYSTEM DROP ICEBERG_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_AVRO_SCHEMA_CACHE, "SYSTEM CLEAR AVRO SCHEMA CACHE, SYSTEM DROP AVRO SCHEMA CACHE, DROP AVRO SCHEMA CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_PARQUET_METADATA_CACHE, "SYSTEM DROP PARQUET_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ + M(SYSTEM_DROP_PUFFIN_FILES_CACHE, "SYSTEM DROP PUFFIN_FILES_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_PREWARM_PRIMARY_INDEX_CACHE, "SYSTEM PREWARM PRIMARY INDEX, PREWARM PRIMARY INDEX CACHE, PREWARM PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_PRIMARY_INDEX_CACHE, "SYSTEM CLEAR PRIMARY INDEX CACHE, SYSTEM DROP PRIMARY INDEX, DROP PRIMARY INDEX CACHE, DROP PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_UNCOMPRESSED_CACHE, "SYSTEM CLEAR UNCOMPRESSED CACHE, SYSTEM DROP UNCOMPRESSED, DROP UNCOMPRESSED CACHE, DROP UNCOMPRESSED", GLOBAL, SYSTEM_DROP_CACHE) \ diff --git a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h index fde3743850ba..a8cc053d21fc 100644 --- a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h +++ b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h @@ -93,6 +93,13 @@ class RoaringBitmapWithSmallSet : private boost::noncopyable return roaring_bitmap->cardinality(); } + UInt64 getAllocatedBytes() const + { + if (isSmall()) + return sizeof(small); + return roaring_bitmap->getSizeInBytes(); + } + void merge(const RoaringBitmapWithSmallSet & r1) { if (r1.isLarge()) diff --git a/src/Client/BuzzHouse/Generator/SessionSettings.cpp b/src/Client/BuzzHouse/Generator/SessionSettings.cpp index d7ea0279dd63..0057657fc26a 100644 --- a/src/Client/BuzzHouse/Generator/SessionSettings.cpp +++ b/src/Client/BuzzHouse/Generator/SessionSettings.cpp @@ -1644,6 +1644,7 @@ static std::unordered_map serverSettings2 = { {"use_page_cache_for_local_disks", trueOrFalseSetting}, {"use_page_cache_for_object_storage", trueOrFalseSetting}, {"use_parquet_metadata_cache", trueOrFalseSetting}, + {"use_puffin_files_cache", trueOrFalseSetting}, {"use_query_cache", trueOrFalseSetting}, {"use_roaring_bitmap_iceberg_positional_deletes", trueOrFalseSetting}, {"use_skip_indexes_if_final_exact_mode", CHSetting(trueOrFalse, {"0", "1"}, true)}, diff --git a/src/Common/CurrentMetrics.cpp b/src/Common/CurrentMetrics.cpp index 9b5a18748f1c..b40437684886 100644 --- a/src/Common/CurrentMetrics.cpp +++ b/src/Common/CurrentMetrics.cpp @@ -337,6 +337,8 @@ M(IcebergMetadataFilesCacheFiles, "Number of cached files in the Iceberg metadata cache") \ M(ParquetMetadataCacheBytes, "Size of the Parquet metadata cache in bytes") \ M(ParquetMetadataCacheFiles, "Number of cached files in the Parquet metadata cache") \ + M(PuffinFilesCacheBytes, "Size of the Puffin files cache in bytes") \ + M(PuffinFilesCacheFiles, "Number of cached entries in the Puffin files cache") \ M(AvroSchemaCacheBytes, "Size of the Avro schema cache in bytes") \ M(AvroSchemaCacheCells, "Number of cached Avro schemas, including both registered and fetched schemas.") \ M(AvroSchemaRegistryCacheBytes, "Size of the Avro schema registry cache in bytes") \ diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index 0b0874fe3a5e..c86b108f2d9d 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -110,6 +110,11 @@ M(IcebergMetadataReturnedObjectInfos, "Total number of returned object infos from iceberg iterator.", ValueType::Number) \ M(IcebergMinMaxNonPrunedDeleteFiles, "Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.", ValueType::Number) \ M(IcebergMinMaxPrunedDeleteFiles, "Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.", ValueType::Number) \ + M(PuffinFilesRead, "Number of Puffin files read (footer or deletion vector blob).", ValueType::Number) \ + M(PuffinFileReadMicroseconds, "Total time spent reading Puffin files.", ValueType::Microseconds) \ + M(PuffinFilesCacheHits, "Number of times parsed Puffin file content has been found in the cache.", ValueType::Number) \ + M(PuffinFilesCacheMisses, "Number of times parsed Puffin file content has not been found in the cache and had to be read from disk.", ValueType::Number) \ + M(PuffinFilesCacheWeightLost, "Approximate number of bytes evicted from the Puffin files cache.", ValueType::Number) \ M(VectorSimilarityIndexCacheHits, "Number of times an index granule has been found in the vector index cache.", ValueType::Number) \ M(VectorSimilarityIndexCacheMisses, "Number of times an index granule has not been found in the vector index cache and had to be read from disk.", ValueType::Number) \ M(VectorSimilarityIndexCacheWeightLost, "Approximate number of bytes evicted from the vector index cache.", ValueType::Number) \ diff --git a/src/Core/Defines.h b/src/Core/Defines.h index c99199f76446..b351507566e8 100644 --- a/src/Core/Defines.h +++ b/src/Core/Defines.h @@ -124,6 +124,10 @@ static constexpr auto DEFAULT_PARQUET_METADATA_CACHE_POLICY = "SLRU"; static constexpr auto DEFAULT_PARQUET_METADATA_CACHE_MAX_SIZE = 512_MiB; static constexpr auto DEFAULT_PARQUET_METADATA_CACHE_SIZE_RATIO = 0.5; static constexpr auto DEFAULT_PARQUET_METADATA_CACHE_MAX_ENTRIES = 5000; +static constexpr auto DEFAULT_PUFFIN_FILES_CACHE_POLICY = "SLRU"; +static constexpr auto DEFAULT_PUFFIN_FILES_CACHE_MAX_SIZE = 512_MiB; +static constexpr auto DEFAULT_PUFFIN_FILES_CACHE_SIZE_RATIO = 0.5; +static constexpr auto DEFAULT_PUFFIN_FILES_CACHE_MAX_ENTRIES = 5000; static constexpr auto DEFAULT_QUERY_CONDITION_CACHE_POLICY = "SLRU"; static constexpr auto DEFAULT_QUERY_CONDITION_CACHE_MAX_SIZE = 100_MiB; static constexpr auto DEFAULT_QUERY_CONDITION_CACHE_SIZE_RATIO = 0.5l; diff --git a/src/Core/ServerSettings.cpp b/src/Core/ServerSettings.cpp index 1f6eb7e54653..41105dd5cc98 100644 --- a/src/Core/ServerSettings.cpp +++ b/src/Core/ServerSettings.cpp @@ -24,6 +24,7 @@ #if USE_PARQUET # include #endif +#include #include #include #include @@ -560,6 +561,10 @@ namespace DECLARE(UInt64, parquet_metadata_cache_size, DEFAULT_PARQUET_METADATA_CACHE_MAX_SIZE, "Maximum size of parquet metadata cache in bytes. Zero means disabled.", 0) \ DECLARE(UInt64, parquet_metadata_cache_max_entries, DEFAULT_PARQUET_METADATA_CACHE_MAX_ENTRIES, "Maximum size of parquet metadata files cache in entries. Zero means disabled.", 0) \ DECLARE(Double, parquet_metadata_cache_size_ratio, DEFAULT_PARQUET_METADATA_CACHE_SIZE_RATIO, "The size of the protected queue (in case of SLRU policy) in the parquet metadata cache relative to the cache's total size.", 0) \ + DECLARE(String, puffin_files_cache_policy, DEFAULT_PUFFIN_FILES_CACHE_POLICY, "Puffin files cache policy name.", 0) \ + DECLARE(UInt64, puffin_files_cache_size, DEFAULT_PUFFIN_FILES_CACHE_MAX_SIZE, "Maximum size of Puffin files cache in bytes. Zero means disabled.", 0) \ + DECLARE(UInt64, puffin_files_cache_max_entries, DEFAULT_PUFFIN_FILES_CACHE_MAX_ENTRIES, "Maximum size of Puffin files cache in entries. Zero means disabled.", 0) \ + DECLARE(Double, puffin_files_cache_size_ratio, DEFAULT_PUFFIN_FILES_CACHE_SIZE_RATIO, "The size of the protected queue (in case of SLRU policy) in the Puffin files cache relative to the cache's total size.", 0) \ DECLARE(String, allowed_disks_for_table_engines, "", "List of disks allowed for use with Iceberg", 0) \ DECLARE(String, vector_similarity_index_cache_policy, DEFAULT_VECTOR_SIMILARITY_INDEX_CACHE_POLICY, "Vector similarity index cache policy name.", 0) \ DECLARE(UInt64, vector_similarity_index_cache_size, DEFAULT_VECTOR_SIMILARITY_INDEX_CACHE_MAX_SIZE, R"(Size of cache for vector similarity indexes. Zero means disabled. @@ -2010,6 +2015,10 @@ ChangeableSettingsMap collectChangeableServerSettings(ContextPtr context) {"parquet_metadata_cache_size", {std::to_string(context->getParquetMetadataCache()->maxSizeInBytes()), ChangeableWithoutRestart::Yes}}); #endif + if (context->getPuffinFilesCache()) + changeable_settings.insert( + {"puffin_files_cache_size", + {std::to_string(context->getPuffinFilesCache()->maxSizeInBytes()), ChangeableWithoutRestart::Yes}}); /// `keeper_hosts` is not a regular config setting; it is derived from the `` config and follows /// it on config reload, so the live value diverges from the empty default stored in `ServerSettings`. diff --git a/src/Core/Settings.cpp b/src/Core/Settings.cpp index 199590245db4..3fd53c612d65 100644 --- a/src/Core/Settings.cpp +++ b/src/Core/Settings.cpp @@ -5558,6 +5558,14 @@ If turned on, parquet format may utilize the parquet metadata cache. Possible values: +- 0 - Disabled +- 1 - Enabled +)", 0) \ + DECLARE(Bool, use_puffin_files_cache, true, R"( +If turned on, Iceberg reads may utilize the Puffin files cache for parsed puffin file content such as deletion vectors. + +Possible values: + - 0 - Disabled - 1 - Enabled )", 0) \ diff --git a/src/Core/SettingsChangesHistory.cpp b/src/Core/SettingsChangesHistory.cpp index ef7d5710e59f..f2819a95bf77 100644 --- a/src/Core/SettingsChangesHistory.cpp +++ b/src/Core/SettingsChangesHistory.cpp @@ -39,6 +39,11 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() /// controls new feature and it's 'true' by default, use 'false' as previous_value). /// It's used to implement `compatibility` setting (see https://github.com/ClickHouse/ClickHouse/issues/35972) /// Note: please check if the key already exists to prevent duplicate entries. + addSettingsChanges(settings_changes_history, "26.6.1.20001.altinityantalya", + { + {"use_puffin_files_cache", false, true, "Enables cache of parsed Puffin file content such as deletion vectors."}, + }); + addSettingsChanges(settings_changes_history, "26.6", { {"output_format_image_width", 1024, 1024, "New setting controlling the width of the output image for image output formats such as PNG."}, diff --git a/src/Formats/registerFormats.cpp b/src/Formats/registerFormats.cpp index 5c23eac98d80..b8538d0bdff2 100644 --- a/src/Formats/registerFormats.cpp +++ b/src/Formats/registerFormats.cpp @@ -107,6 +107,7 @@ void registerInputFormatJSONAsObject(FormatFactory & factory); void registerInputFormatLineAsString(FormatFactory & factory); void registerInputFormatMySQLDump(FormatFactory & factory); void registerInputFormatParquetMetadata(FormatFactory & factory); +void registerInputFormatPuffin(FormatFactory & factory); void registerInputFormatDWARF(FormatFactory & factory); void registerInputFormatOne(FormatFactory & factory); @@ -150,6 +151,7 @@ void registerTemplateSchemaReader(FormatFactory & factory); void registerMySQLSchemaReader(FormatFactory & factory); void registerBSONEachRowSchemaReader(FormatFactory & factory); void registerParquetMetadataSchemaReader(FormatFactory & factory); +void registerPuffinSchemaReaders(FormatFactory & factory); void registerDWARFSchemaReader(FormatFactory & factory); void registerOneSchemaReader(FormatFactory & factory); void registerNpySchemaReader(FormatFactory & factory); @@ -266,6 +268,7 @@ void registerFormats() registerInputFormatMySQLDump(factory); registerInputFormatParquetMetadata(factory); + registerInputFormatPuffin(factory); registerInputFormatDWARF(factory); registerInputFormatOne(factory); @@ -305,6 +308,7 @@ void registerFormats() registerMySQLSchemaReader(factory); registerBSONEachRowSchemaReader(factory); registerParquetMetadataSchemaReader(factory); + registerPuffinSchemaReaders(factory); registerDWARFSchemaReader(factory); registerOneSchemaReader(factory); registerNpySchemaReader(factory); diff --git a/src/Interpreters/ClusterFunctionReadTask.cpp b/src/Interpreters/ClusterFunctionReadTask.cpp index f3cfad1fba31..57f15dd0baa4 100644 --- a/src/Interpreters/ClusterFunctionReadTask.cpp +++ b/src/Interpreters/ClusterFunctionReadTask.cpp @@ -82,6 +82,46 @@ void ClusterFunctionReadTaskResponse::serialize(WriteBuffer & out, size_t worker { auto protocol_version = std::min(static_cast(worker_protocol_version), static_cast(DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION)); + + /// Fail closed: downgrading would omit deletion / selection vectors and return deleted rows. + if (protocol_version < DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_EXCLUDED_ROWS + && hasNonEmptyExcludedRows(data_lake_metadata)) + { + throw Exception( + ErrorCodes::UNKNOWN_PROTOCOL, + "Worker protocol version {} cannot carry `excluded_rows`, which is required for distributed " + "reads with deletion vectors / selection vectors (minimum protocol version: {})", + protocol_version, + DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_EXCLUDED_ROWS); + } + + /// Fail closed: protocol < 3 omits `iceberg_info`, so workers rebuild a plain `ObjectInfo` and + /// skip equality / parquet position delete transforms — returning deleted rows. + if (protocol_version < DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA + && iceberg_info.has_value() + && (!iceberg_info->position_deletes_objects.empty() || !iceberg_info->equality_deletes_objects.empty())) + { + throw Exception( + ErrorCodes::UNKNOWN_PROTOCOL, + "Worker protocol version {} cannot carry `iceberg_info` with equality / position deletes " + "(minimum protocol version: {})", + protocol_version, + DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA); + } + + /// Fail closed: protocol < 4 omits `file_bucket_info`, so each bucket task becomes a full-file + /// read and bucket-split cluster queries return duplicated rows. + if (protocol_version < DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO + && file_bucket_info) + { + throw Exception( + ErrorCodes::UNKNOWN_PROTOCOL, + "Worker protocol version {} cannot carry `file_bucket_info`, which is required for " + "distributed bucket-split reads (minimum protocol version: {})", + protocol_version, + DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO); + } + writeVarUInt(protocol_version, out); writeStringBinary(path, out); diff --git a/src/Interpreters/Context.cpp b/src/Interpreters/Context.cpp index fc2e58c60368..a12c044aedf8 100644 --- a/src/Interpreters/Context.cpp +++ b/src/Interpreters/Context.cpp @@ -55,6 +55,7 @@ #include #include #include +#include #include #include #include @@ -586,6 +587,7 @@ struct ContextSharedPart : boost::noncopyable #if USE_PARQUET mutable ParquetMetadataCachePtr parquet_metadata_cache TSA_GUARDED_BY(mutex); /// Cache of deserialized parquet metadata files. #endif + mutable PuffinFilesCachePtr puffin_files_cache TSA_GUARDED_BY(mutex); /// Cache of parsed puffin file content. AsynchronousMetrics * asynchronous_metrics TSA_GUARDED_BY(mutex) = nullptr; /// Points to asynchronous metrics mutable PageCachePtr page_cache TSA_GUARDED_BY(mutex); /// Userspace page cache. ProcessList process_list; /// Executing queries at the moment. @@ -4755,6 +4757,52 @@ void Context::clearParquetMetadataCache() const } #endif +void Context::setPuffinFilesCache(const String & cache_policy, size_t max_size_in_bytes, size_t max_entries, double size_ratio) +{ + std::lock_guard lock(shared->mutex); + + if (shared->puffin_files_cache) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Puffin files cache has been already created."); + + shared->puffin_files_cache = std::make_shared(cache_policy, max_size_in_bytes, max_entries, size_ratio); +} + +void Context::updatePuffinFilesCacheConfiguration(const Poco::Util::AbstractConfiguration & config, size_t max_cache_size) +{ + std::lock_guard lock(shared->mutex); + + if (!shared->puffin_files_cache) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Puffin files cache was not created yet."); + + size_t size = config.getUInt64("puffin_files_cache_size", DEFAULT_PUFFIN_FILES_CACHE_MAX_SIZE); + size_t max_entries = config.getUInt64("puffin_files_cache_max_entries", DEFAULT_PUFFIN_FILES_CACHE_MAX_ENTRIES); + if (size > max_cache_size) + { + size = max_cache_size; + LOG_DEBUG(shared->log, "Lowered Puffin files cache size to {} because the system has limited RAM", formatReadableSizeWithBinarySuffix(size)); + } + shared->puffin_files_cache->setMaxSizeInBytes(size); + shared->puffin_files_cache->setMaxCount(max_entries); +} + +std::shared_ptr Context::getPuffinFilesCache() const +{ + SharedLockGuard lock(shared->mutex); + + if (!shared->puffin_files_cache) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Puffin files cache was not created yet."); + return shared->puffin_files_cache; +} + +void Context::clearPuffinFilesCache() const +{ + auto cache = getPuffinFilesCache(); + + /// Clear the cache without holding context mutex to avoid blocking context for a long time + if (cache) + cache->clear(); +} + void Context::setQueryConditionCache(const String & cache_policy, size_t max_size_in_bytes, double size_ratio) { std::lock_guard lock(shared->mutex); diff --git a/src/Interpreters/Context.h b/src/Interpreters/Context.h index 4d8d171b9daf..d8d45c63e09c 100644 --- a/src/Interpreters/Context.h +++ b/src/Interpreters/Context.h @@ -109,6 +109,7 @@ class MMappedFileCache; class UncompressedCache; class IcebergMetadataFilesCache; class ParquetMetadataCache; +class PuffinFilesCache; class VectorSimilarityIndexCache; class TextIndexTokensCache; class TextIndexHeaderCache; @@ -1508,6 +1509,11 @@ class Context: public ContextData, public std::enable_shared_from_this void clearParquetMetadataCache() const; #endif + void setPuffinFilesCache(const String & cache_policy, size_t max_size_in_bytes, size_t max_entries, double size_ratio); + void updatePuffinFilesCacheConfiguration(const Poco::Util::AbstractConfiguration & config, size_t max_cache_size); + std::shared_ptr getPuffinFilesCache() const; + void clearPuffinFilesCache() const; + void setAllowedDisksForTableEngines(std::unordered_set && allowed_disks_) { allowed_disks = std::move(allowed_disks_); } const std::unordered_set & getAllowedDisksForTableEngines() const { return allowed_disks; } diff --git a/src/Interpreters/InterpreterSystemQuery.cpp b/src/Interpreters/InterpreterSystemQuery.cpp index 9b515844c721..422b8e4fa048 100644 --- a/src/Interpreters/InterpreterSystemQuery.cpp +++ b/src/Interpreters/InterpreterSystemQuery.cpp @@ -481,6 +481,10 @@ BlockIO InterpreterSystemQuery::execute() #else throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "The server was compiled without the support for Parquet"); #endif + case Type::CLEAR_PUFFIN_FILES_CACHE: + getContext()->checkAccess(AccessType::SYSTEM_DROP_PUFFIN_FILES_CACHE); + system_context->clearPuffinFilesCache(); + break; case Type::CLEAR_PRIMARY_INDEX_CACHE: getContext()->checkAccess(AccessType::SYSTEM_DROP_PRIMARY_INDEX_CACHE); system_context->clearPrimaryIndexCache(); @@ -2474,6 +2478,7 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() case Type::CLEAR_ICEBERG_METADATA_CACHE: case Type::CLEAR_AVRO_SCHEMA_CACHE: case Type::CLEAR_PARQUET_METADATA_CACHE: + case Type::CLEAR_PUFFIN_FILES_CACHE: case Type::CLEAR_PRIMARY_INDEX_CACHE: case Type::CLEAR_MMAP_CACHE: case Type::CLEAR_QUERY_CONDITION_CACHE: diff --git a/src/Parsers/ASTSystemQuery.cpp b/src/Parsers/ASTSystemQuery.cpp index c4ee97348899..b6dbd9b298a8 100644 --- a/src/Parsers/ASTSystemQuery.cpp +++ b/src/Parsers/ASTSystemQuery.cpp @@ -603,6 +603,7 @@ void ASTSystemQuery::formatImpl(WriteBuffer & ostr, const FormatSettings & setti case Type::CLEAR_S3_CLIENT_CACHE: case Type::CLEAR_ICEBERG_METADATA_CACHE: case Type::CLEAR_PARQUET_METADATA_CACHE: + case Type::CLEAR_PUFFIN_FILES_CACHE: case Type::CLEAR_AVRO_SCHEMA_CACHE: case Type::RESET_COVERAGE: case Type::RESTART_REPLICAS: diff --git a/src/Parsers/ASTSystemQuery.h b/src/Parsers/ASTSystemQuery.h index adf76d898d42..814e76989893 100644 --- a/src/Parsers/ASTSystemQuery.h +++ b/src/Parsers/ASTSystemQuery.h @@ -44,6 +44,7 @@ class ASTSystemQuery : public IAST, public ASTQueryWithOnCluster CLEAR_COMPILED_EXPRESSION_CACHE, CLEAR_ICEBERG_METADATA_CACHE, CLEAR_PARQUET_METADATA_CACHE, + CLEAR_PUFFIN_FILES_CACHE, CLEAR_FILESYSTEM_CACHE, CLEAR_DISTRIBUTED_CACHE, CLEAR_DISK_METADATA_CACHE, diff --git a/src/Parsers/ParserSystemQuery.cpp b/src/Parsers/ParserSystemQuery.cpp index 07b79465ea04..d44527847f22 100644 --- a/src/Parsers/ParserSystemQuery.cpp +++ b/src/Parsers/ParserSystemQuery.cpp @@ -285,6 +285,7 @@ bool ParserSystemQuery::parseImpl(IParser::Pos & pos, ASTPtr & node, Expected & {"DROP COMPILED EXPRESSION CACHE", Type::CLEAR_COMPILED_EXPRESSION_CACHE}, {"DROP ICEBERG METADATA CACHE", Type::CLEAR_ICEBERG_METADATA_CACHE}, {"DROP PARQUET METADATA CACHE", Type::CLEAR_PARQUET_METADATA_CACHE}, + {"DROP PUFFIN_FILES_CACHE", Type::CLEAR_PUFFIN_FILES_CACHE}, {"DROP FILESYSTEM CACHE", Type::CLEAR_FILESYSTEM_CACHE}, {"DROP DISTRIBUTED CACHE", Type::CLEAR_DISTRIBUTED_CACHE}, {"DROP DISK METADATA CACHE", Type::CLEAR_DISK_METADATA_CACHE}, diff --git a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp index edbf421ccbeb..567a018a00b6 100644 --- a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp +++ b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp @@ -25,6 +25,7 @@ namespace DB namespace ErrorCodes { extern const int LOGICAL_ERROR; + extern const int INCORRECT_DATA; } static Parquet::ReadOptions convertReadOptions(const FormatSettings & format_settings) @@ -120,23 +121,68 @@ parquet::format::FileMetaData ParquetV3BlockInputFormat::getFileMetadata(Parquet } } -Chunk ParquetV3BlockInputFormat::read() +void ParquetV3BlockInputFormat::prepareNeedOnlyCountRowGroups(const parquet::format::FileMetaData & file_metadata) { - if (need_only_count) + const size_t num_row_groups = file_metadata.row_groups.size(); + std::vector global_offsets(num_row_groups + 1, 0); + for (size_t i = 0; i < num_row_groups; ++i) { - if (reported_count) - return {}; + if (file_metadata.row_groups[i].num_rows < 0) + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Parquet row group {} has negative row count: {}", + i, + file_metadata.row_groups[i].num_rows); + global_offsets[i + 1] = global_offsets[i] + static_cast(file_metadata.row_groups[i].num_rows); + } - /// Don't init Reader and ReadManager if we only need file metadata. - Parquet::Prefetcher temp_prefetcher; - temp_prefetcher.init(in, read_options, parser_shared_resources); - parquet::format::FileMetaData file_metadata = getFileMetadata(temp_prefetcher); + need_only_count_row_groups.clear(); + need_only_count_next = 0; + if (!buckets_to_read) + { + if (global_offsets.back() > 0) + need_only_count_row_groups.push_back({.row_num_offset = 0, .num_rows = global_offsets.back()}); + return; + } - auto chunk = getChunkForCount(size_t(file_metadata.num_rows)); - chunk.getChunkInfos().add(std::make_shared(0)); + for (size_t row_group_id : buckets_to_read->row_group_ids) + { + if (row_group_id >= num_row_groups) + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Parquet bucket row group {} is out of range (file has {} row groups)", + row_group_id, + num_row_groups); + + const size_t num_rows = static_cast(file_metadata.row_groups[row_group_id].num_rows); + if (num_rows == 0) + continue; + + need_only_count_row_groups.push_back( + {.row_num_offset = global_offsets[row_group_id], .num_rows = num_rows}); + } +} + +Chunk ParquetV3BlockInputFormat::read() +{ + if (need_only_count) + { + if (!need_only_count_prepared) + { + /// Don't init Reader and ReadManager if we only need file metadata. + Parquet::Prefetcher temp_prefetcher; + temp_prefetcher.init(in, read_options, parser_shared_resources); + prepareNeedOnlyCountRowGroups(getFileMetadata(temp_prefetcher)); + need_only_count_prepared = true; + } + + if (need_only_count_next >= need_only_count_row_groups.size()) + return {}; - reported_count = true; + const auto & row_group = need_only_count_row_groups[need_only_count_next++]; + auto chunk = getChunkForCount(row_group.num_rows); + chunk.getChunkInfos().add(std::make_shared(row_group.row_num_offset)); return chunk; } @@ -199,6 +245,9 @@ void ParquetV3BlockInputFormat::resetParser() reader.reset(); } previous_block_missing_values.clear(); + need_only_count_prepared = false; + need_only_count_next = 0; + need_only_count_row_groups.clear(); IInputFormat::resetParser(); } diff --git a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.h b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.h index 87a5192ae641..2c5baf00ea92 100644 --- a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.h +++ b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.h @@ -81,12 +81,21 @@ class ParquetV3BlockInputFormat final : public IInputFormat std::mutex reader_mutex; std::optional reader; - bool reported_count = false; // if need_only_count + bool need_only_count_prepared = false; + size_t need_only_count_next = 0; + + struct NeedOnlyCountRowGroup + { + size_t row_num_offset = 0; + size_t num_rows = 0; + }; + std::vector need_only_count_row_groups; BlockMissingValues previous_block_missing_values; size_t previous_approx_bytes_read_for_chunk = 0; void initializeIfNeeded(); + void prepareNeedOnlyCountRowGroups(const parquet::format::FileMetaData & file_metadata); std::shared_ptr buckets_to_read; parquet::format::FileMetaData getFileMetadata(Parquet::Prefetcher & prefetcher) const; diff --git a/src/Processors/Formats/Impl/PuffinBlockInputFormat.cpp b/src/Processors/Formats/Impl/PuffinBlockInputFormat.cpp new file mode 100644 index 000000000000..caea204cacab --- /dev/null +++ b/src/Processors/Formats/Impl/PuffinBlockInputFormat.cpp @@ -0,0 +1,1179 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ProfileEvents +{ +extern const Event PuffinFilesRead; +extern const Event PuffinFileReadMicroseconds; +} + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int BAD_ARGUMENTS; + extern const int LZ4_DECODER_FAILED; +} + +namespace +{ + +struct ScopedPuffinFileReadProfileEvent +{ + ProfileEventTimeIncrement watch; + + ScopedPuffinFileReadProfileEvent() + : watch(ProfileEvents::PuffinFileReadMicroseconds) + { + ProfileEvents::increment(ProfileEvents::PuffinFilesRead); + } +}; + +constexpr UInt8 PUFFIN_MAGIC[4] = {0x50, 0x46, 0x41, 0x31}; +constexpr UInt8 PUFFIN_FOOTER_COMPRESSED_FLAG = 0x01; +constexpr size_t PUFFIN_FOOTER_TRAILER_SIZE = 12; +constexpr size_t PUFFIN_FOOTER_LZ4_MAX_RATIO = 255; +/// Absolute cap on footer payload size (uncompressed JSON bytes, or declared LZ4 contentSize). +/// Prevents crafted FooterPayloadSize / contentSize values from forcing huge allocations. +/// Intentionally a compile-time safety ceiling (same class as other format hard caps), not a +/// FormatSettings / input_format_puffin_* knob: oversized footers are never legitimate input. +constexpr size_t PUFFIN_FOOTER_MAX_PAYLOAD_SIZE = 16 * 1024 * 1024; +/// Absolute cap on materialized deleted positions (~800 MiB of UInt64s at this limit). +/// Applied only when `deleted_rows` is requested; subset reads that skip materialization do not +/// enforce it. Intentionally not a FormatSettings knob: this is a fail-closed amplification +/// guard for Array(UInt64) expansion of roaring bitmaps, not a tunable acceptance policy. +/// Global `max_memory_usage` still applies later; this rejects before allocating the array. +constexpr UInt64 PUFFIN_DV_MAX_MATERIALIZED_POSITIONS = 100'000'000; +/// Absolute cap on on-disk deletion-vector-v1 blob length before reading bytes into memory. +/// Aligns with Iceberg DeleteLoader's 2 GiB content-size check. Fail closed before allocate: +/// footer `length` alone can point at the whole blob region while `cardinality` stays small. +/// Combined with an 8-byte envelope peek (magic + combined_length vs footer length) so a mismatched +/// header cannot force a full read up to this cap. A consistent huge envelope still reads up to the +/// cap (CRC needs the payload); streaming roaring without holding the blob is intentionally out of +/// scope. Intentionally a compile-time safety ceiling, not a FormatSettings knob. +constexpr size_t PUFFIN_DV_MAX_BLOB_SIZE = 2ULL * 1024 * 1024 * 1024; +constexpr UInt8 DELETION_VECTOR_MAGIC[4] = {0xD1, 0xD3, 0x39, 0x64}; +/// Matches Iceberg `RoaringPositionBitmap.MAX_POSITION` / iceberg-cpp `kMaxPosition`. +/// Key INT32_MAX is unsupported: the reference bitmap array length is a signed Int32. +constexpr Int64 DELETION_VECTOR_MAX_POSITION = 0x7FFFFFFE80000000LL; +constexpr Int32 DELETION_VECTOR_MAX_KEY = std::numeric_limits::max() - 1; + +UInt32 readBigEndianUInt32(const UInt8 * data) +{ + return (static_cast(data[0]) << 24) + | (static_cast(data[1]) << 16) + | (static_cast(data[2]) << 8) + | static_cast(data[3]); +} + +UInt64 positionFromKeyAndSubPosition(UInt32 key, UInt32 sub_position) +{ + return (static_cast(key) << 32) | static_cast(sub_position); +} + +void checkMagic(const UInt8 * p, const char * context) +{ + if (std::memcmp(p, PUFFIN_MAGIC, 4) != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid Puffin magic ({})", context); +} + +/// `blob_region_end` is the end of the `Blob₁ ... Blobₙ` region, i.e. the offset of the footer's +/// leading magic. Blob offset/length must stay within [PUFFIN_MAGIC, blob_region_end), so a malformed +/// footer cannot point a blob into the footer payload or trailer. +void validatePuffinBlobBounds(Int64 offset, Int64 length, size_t blob_region_end, size_t blob_index) +{ + if (offset < static_cast(sizeof(PUFFIN_MAGIC)) || length < 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: offset/length out of bounds", blob_index); + + Int64 end = 0; + if (common::addOverflow(offset, length, end)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: offset/length out of bounds", blob_index); + + if (static_cast(end) > blob_region_end) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: offset/length out of bounds", blob_index); +} + +void validatePuffinFooterFlags(const UInt8 flags[4]) +{ + if (flags[1] != 0 || flags[2] != 0 || flags[3] != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown Puffin footer flags"); + if ((flags[0] & ~PUFFIN_FOOTER_COMPRESSED_FLAG) != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown Puffin footer flags"); +} + +String decompressPuffinFooterPayload(const char * data, size_t size) +{ + LZ4F_dctx * dctx = nullptr; + size_t ret = LZ4F_createDecompressionContext(&dctx, LZ4F_VERSION); + if (LZ4F_isError(ret)) + throw Exception(ErrorCodes::LZ4_DECODER_FAILED, "Failed to create LZ4 decompression context: {}", LZ4F_getErrorName(ret)); + + struct DecompressionContextGuard + { + LZ4F_dctx * dctx; + explicit DecompressionContextGuard(LZ4F_dctx * dctx_) : dctx(dctx_) { } + ~DecompressionContextGuard() { LZ4F_freeDecompressionContext(dctx); } + }; + DecompressionContextGuard guard(dctx); + + LZ4F_frameInfo_t frame_info{}; + size_t header_size = size; + ret = LZ4F_getFrameInfo(dctx, &frame_info, data, &header_size); + if (LZ4F_isError(ret)) + throw Exception(ErrorCodes::LZ4_DECODER_FAILED, "Failed to read LZ4 frame info for Puffin footer: {}", LZ4F_getErrorName(ret)); + + const char * src = data + header_size; + size_t src_remaining = size - header_size; + + size_t max_by_ratio = 0; + if (common::mulOverflow(size, PUFFIN_FOOTER_LZ4_MAX_RATIO, max_by_ratio)) + throw Exception(ErrorCodes::LZ4_DECODER_FAILED, "Puffin footer compressed payload is too large"); + + if (frame_info.contentSize == 0) + throw Exception(ErrorCodes::LZ4_DECODER_FAILED, "Puffin footer LZ4 frame must declare content size"); + + const size_t max_decompressed_size = std::min(max_by_ratio, PUFFIN_FOOTER_MAX_PAYLOAD_SIZE); + if (frame_info.contentSize > max_decompressed_size) + { + if (max_decompressed_size == PUFFIN_FOOTER_MAX_PAYLOAD_SIZE) + throw Exception( + ErrorCodes::LZ4_DECODER_FAILED, + "Puffin footer LZ4 content size {} exceeds absolute decompression limit {}", + frame_info.contentSize, + PUFFIN_FOOTER_MAX_PAYLOAD_SIZE); + + throw Exception( + ErrorCodes::LZ4_DECODER_FAILED, + "Puffin footer LZ4 content size {} exceeds decompression limit {}", + frame_info.contentSize, + max_decompressed_size); + } + + String result; + result.resize(static_cast(frame_info.contentSize)); + + size_t dst_offset = 0; + while (true) + { + size_t src_read = src_remaining; + size_t dst_write = result.size() - dst_offset; + ret = LZ4F_decompress(dctx, result.data() + dst_offset, &dst_write, src, &src_read, nullptr); + if (LZ4F_isError(ret)) + throw Exception(ErrorCodes::LZ4_DECODER_FAILED, "Failed to decompress Puffin footer: {}", LZ4F_getErrorName(ret)); + + src += src_read; + src_remaining -= src_read; + dst_offset += dst_write; + + if (ret == 0) + break; + + /// No forward progress while the frame is still open means the input is truncated: + /// LZ4F_decompress wants more input but we have given it everything. Stop to avoid spinning. + if (dst_write == 0 && src_read == 0) + throw Exception(ErrorCodes::LZ4_DECODER_FAILED, "Puffin footer LZ4 frame is incomplete"); + + if (dst_offset == result.size()) + throw Exception( + ErrorCodes::LZ4_DECODER_FAILED, + "Puffin footer decompressed size exceeds declared content size {}", + frame_info.contentSize); + } + + if (dst_offset != frame_info.contentSize) + throw Exception( + ErrorCodes::LZ4_DECODER_FAILED, + "Puffin footer LZ4 decompressed size {} does not match content size {}", + dst_offset, + frame_info.contentSize); + + if (src_remaining != 0) + throw Exception( + ErrorCodes::LZ4_DECODER_FAILED, + "Puffin footer LZ4 frame has {} trailing bytes", + src_remaining); + + result.resize(dst_offset); + return result; +} + +void requireBlobMetadataField(const Poco::JSON::Object::Ptr & blob_obj, const char * field_name, size_t blob_index) +{ + if (!blob_obj->has(field_name) || blob_obj->isNull(field_name)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: missing required field '{}'", blob_index, field_name); +} + +/// Poco reports JSON booleans as integers (`std::numeric_limits::is_integer`). +bool isJSONInteger(const Poco::Dynamic::Var & value) +{ + return value.isInteger() && !value.isBoolean(); +} + +/// Poco stores JSON integers that do not fit signed Int64 as unsigned; convert() would throw RangeException. +std::optional tryJSONIntegerAsInt64(const Poco::Dynamic::Var & value) +{ + if (value.isInteger() && !value.isSigned()) + { + const auto as_uint64 = value.convert(); + if (as_uint64 > static_cast(std::numeric_limits::max())) + return std::nullopt; + return static_cast(as_uint64); + } + + return value.convert(); +} + +Int64 requireBlobMetadataInt64(const Poco::JSON::Object::Ptr & blob_obj, const char * field_name, size_t blob_index) +{ + requireBlobMetadataField(blob_obj, field_name, blob_index); + const Poco::Dynamic::Var & value = blob_obj->get(field_name); + if (!isJSONInteger(value)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: field '{}' must be an integer", blob_index, field_name); + + auto as_int64 = tryJSONIntegerAsInt64(value); + if (!as_int64) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: field '{}' is out of Int64 range", blob_index, field_name); + return *as_int64; +} + +Int32 requireBlobMetadataFieldsElementInt32(const Poco::Dynamic::Var & value, size_t blob_index, size_t field_index) +{ + if (!isJSONInteger(value)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: fields[{}] must be an integer", blob_index, field_index); + + auto as_int64 = tryJSONIntegerAsInt64(value); + if (!as_int64) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: fields[{}] is out of Int64 range", blob_index, field_index); + + if (*as_int64 < std::numeric_limits::min() || *as_int64 > std::numeric_limits::max()) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: fields[{}] is out of Int32 range", blob_index, field_index); + + return static_cast(*as_int64); +} + +String requireJSONStringValue(const Poco::Dynamic::Var & value, size_t blob_index, const char * field_name) +{ + if (!value.isString()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: field '{}' must be a string", blob_index, field_name); + return value.extract(); +} + +String requireBlobMetadataString(const Poco::JSON::Object::Ptr & blob_obj, const char * field_name, size_t blob_index) +{ + requireBlobMetadataField(blob_obj, field_name, blob_index); + return requireJSONStringValue(blob_obj->get(field_name), blob_index, field_name); +} + +String optBlobMetadataString(const Poco::JSON::Object::Ptr & blob_obj, const char * field_name, size_t blob_index) +{ + if (!blob_obj->has(field_name)) + return {}; + if (blob_obj->isNull(field_name)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: field '{}' must be a string", blob_index, field_name); + return requireJSONStringValue(blob_obj->get(field_name), blob_index, field_name); +} + +void requireDeletionVectorV1Properties(const PuffinBlob & blob, size_t blob_index) +{ + static constexpr const char * required_properties[] = {"referenced-data-file", "cardinality"}; + for (const char * key : required_properties) + { + auto it = blob.properties.find(key); + if (it == blob.properties.end() || it->second.empty()) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 missing required property '{}'", + blob_index, + key); + } + + UInt64 cardinality = 0; + if (!tryParse(cardinality, blob.properties.at("cardinality"))) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 property 'cardinality' must be an unsigned integer", + blob_index); +} + +void parseStringValuedProperties( + const Poco::JSON::Object::Ptr & props_obj, + std::map * out, + bool for_blob, + size_t blob_index) +{ + for (const auto & [key, val] : *props_obj) + { + if (!val.isString()) + { + if (for_blob) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: property '{}' must be a string", blob_index, key); + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin footer property '{}' must be a string", key); + } + if (out) + out->emplace(key, val.extract()); + } +} + +void parseBlobProperties(const Poco::JSON::Object::Ptr & blob_obj, PuffinBlob & blob, size_t blob_index, bool required) +{ + if (!blob_obj->has("properties")) + { + if (required) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: missing required field 'properties'", blob_index); + return; + } + + if (blob_obj->isNull("properties")) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: field 'properties' must be an object", blob_index); + + auto props_obj = blob_obj->getObject("properties"); + if (!props_obj) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: field 'properties' must be an object", blob_index); + + parseStringValuedProperties(props_obj, &blob.properties, /*for_blob=*/true, blob_index); +} + +/// Optional FileMetadata.properties: JSON object with string values (Iceberg Puffin spec). +/// Absent is allowed; present null / non-object is rejected. +void validateFileMetadataProperties(const Poco::JSON::Object::Ptr & footer_obj) +{ + if (!footer_obj->has("properties")) + return; + + if (footer_obj->isNull("properties")) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin footer field 'properties' must be an object"); + + auto props_obj = footer_obj->getObject("properties"); + if (!props_obj) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin footer field 'properties' must be an object"); + + parseStringValuedProperties(props_obj, /*out=*/nullptr, /*for_blob=*/false, /*blob_index=*/0); +} + +std::vector parseFooterJSON(const String & footer_json, size_t blob_region_end) +{ + Poco::JSON::Parser parser; + Poco::Dynamic::Var root; + try + { + root = parser.parse(footer_json); + } + catch (const Exception &) + { + throw; + } + catch (const Poco::Exception & e) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot parse Puffin footer JSON: {}", e.displayText()); + } + + if (root.type() != typeid(Poco::JSON::Object::Ptr)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin footer JSON must be an object"); + + auto obj = root.extract(); + + validateFileMetadataProperties(obj); + + if (!obj->has("blobs") || obj->isNull("blobs")) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin footer is missing required field 'blobs'"); + + auto blobs_arr = obj->getArray("blobs"); + if (!blobs_arr) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin footer field 'blobs' must be an array"); + + std::vector blobs; + for (size_t i = 0; i < blobs_arr->size(); ++i) + { + auto blob_obj = blobs_arr->getObject(static_cast(i)); + if (!blob_obj) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: must be an object", i); + + PuffinBlob blob; + + blob.type = requireBlobMetadataString(blob_obj, "type", i); + blob.snapshot_id = requireBlobMetadataInt64(blob_obj, "snapshot-id", i); + blob.sequence_number = requireBlobMetadataInt64(blob_obj, "sequence-number", i); + blob.offset = requireBlobMetadataInt64(blob_obj, "offset", i); + blob.length = requireBlobMetadataInt64(blob_obj, "length", i); + + if (blob.type == "deletion-vector-v1") + { + if (blob.snapshot_id != -1 || blob.sequence_number != -1) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 snapshot-id and sequence-number must be -1", + i); + + /// Spec requires the key to be omitted; present null is not omission. + if (blob_obj->has("compression-codec")) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 must omit 'compression-codec'", + i); + + parseBlobProperties(blob_obj, blob, i, /*required=*/true); + requireDeletionVectorV1Properties(blob, i); + } + else + { + blob.compression_codec = optBlobMetadataString(blob_obj, "compression-codec", i); + parseBlobProperties(blob_obj, blob, i, /*required=*/false); + } + + requireBlobMetadataField(blob_obj, "fields", i); + auto fields_arr = blob_obj->getArray("fields"); + if (!fields_arr) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob {}: field 'fields' must be an array", i); + + for (size_t j = 0; j < fields_arr->size(); ++j) + blob.fields.push_back(requireBlobMetadataFieldsElementInt32(fields_arr->get(static_cast(j)), i, j)); + + validatePuffinBlobBounds(blob.offset, blob.length, blob_region_end, i); + + blobs.push_back(std::move(blob)); + } + return blobs; +} + +std::vector readPuffinFooterFromSeekable(SeekableReadBuffer & seekable, size_t file_size) +{ + if (file_size < 16) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin file too small"); + + seekable.seek(0, SEEK_SET); + char magic_buf[4]; + seekable.readStrict(magic_buf, 4); + checkMagic(reinterpret_cast(magic_buf), "header"); + + seekable.seek(static_cast(file_size - PUFFIN_FOOTER_TRAILER_SIZE), SEEK_SET); + Int32 footer_length_signed = 0; + readBinaryLittleEndian(footer_length_signed, seekable); + + UInt8 flags[4] = {}; + seekable.readStrict(reinterpret_cast(flags), sizeof(flags)); + + char trailing_buf[4]; + seekable.readStrict(trailing_buf, 4); + checkMagic(reinterpret_cast(trailing_buf), "trailing"); + validatePuffinFooterFlags(flags); + + if (footer_length_signed <= 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid Puffin footer length: {}", footer_length_signed); + + UInt64 footer_trailer_size = 0; + if (common::addOverflow( + static_cast(footer_length_signed), + static_cast(PUFFIN_FOOTER_TRAILER_SIZE), + footer_trailer_size) + || footer_trailer_size > file_size) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid Puffin footer length: {}", footer_length_signed); + + const size_t footer_length = static_cast(footer_length_signed); + const size_t payload_start = file_size - PUFFIN_FOOTER_TRAILER_SIZE - footer_length; + /// Footer layout is Magic | FooterPayload | …; require a distinct footer-open Magic after the + /// file header Magic (payload_start == 4 would reuse the header Magic when blobs is empty). + if (payload_start < 2 * sizeof(PUFFIN_MAGIC)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid Puffin footer length: {}", footer_length_signed); + + if (footer_length > PUFFIN_FOOTER_MAX_PAYLOAD_SIZE) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin footer payload size {} exceeds absolute limit {}", + footer_length, + PUFFIN_FOOTER_MAX_PAYLOAD_SIZE); + + seekable.seek(static_cast(payload_start - sizeof(PUFFIN_MAGIC)), SEEK_SET); + char leading_magic_buf[4]; + seekable.readStrict(leading_magic_buf, 4); + checkMagic(reinterpret_cast(leading_magic_buf), "footer"); + + String footer_payload(footer_length, '\0'); + seekable.seek(static_cast(payload_start), SEEK_SET); + seekable.readStrict(footer_payload.data(), footer_length); + + String footer_json; + if ((flags[0] & PUFFIN_FOOTER_COMPRESSED_FLAG) != 0) + footer_json = decompressPuffinFooterPayload(footer_payload.data(), footer_payload.size()); + else + footer_json = std::move(footer_payload); + + const size_t blob_region_end = payload_start - sizeof(PUFFIN_MAGIC); + return parseFooterJSON(footer_json, blob_region_end); +} + +PuffinFooter readPuffinFooter(ReadBuffer & buf, bool seekable_read) +{ + ScopedPuffinFileReadProfileEvent profile_event; + + PuffinFooter result; + + auto * seekable = dynamic_cast(&buf); + auto file_size_opt = tryGetFileSizeFromReadBuffer(buf); + + /// Pipes/FIFOs are SeekableReadBuffer subclasses but fstat reports size 0; require a real + /// regular file before trusting seek+size (same pattern as ORC/Arrow). Also honor + /// `input_format_allow_seeks` via FormatSettings.seekable_read. + if (seekable_read && seekable && seekable->checkIfActuallySeekable() && file_size_opt) + { + result.blobs = readPuffinFooterFromSeekable(*seekable, *file_size_opt); + } + else + { + /// Fail fast on format mismatch before buffering the rest of a (possibly huge) stream — + /// same pattern as `asArrowFileLoadIntoMemory`. + result.data.resize(sizeof(PUFFIN_MAGIC)); + const size_t magic_read = buf.read(reinterpret_cast(result.data.data()), sizeof(PUFFIN_MAGIC)); + if (magic_read < sizeof(PUFFIN_MAGIC)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin file too small"); + checkMagic(result.data.data(), "header"); + + std::vector tmp(DEFAULT_BLOCK_SIZE); + while (!buf.eof()) + { + size_t n = buf.read(reinterpret_cast(tmp.data()), tmp.size()); + result.data.insert(result.data.end(), tmp.data(), tmp.data() + n); + } + + ReadBufferFromMemory mem_buf(result.data.data(), result.data.size()); + result.blobs = readPuffinFooterFromSeekable(mem_buf, result.data.size()); + } + + return result; +} + +String readPuffinBlobBytes( + const PuffinBlob & blob, ReadBuffer & buf, const std::vector & data, bool seekable_read) +{ + const size_t length = static_cast(blob.length); + + /// When the footer path buffered the whole file, copy from that buffer — the original input + /// may still look Seekable (e.g. a consumed pipe) but must not be seeked. + if (!data.empty()) + { + if (static_cast(blob.offset) + static_cast(blob.length) > data.size()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob offset/length out of bounds of buffered data"); + + return String(reinterpret_cast(data.data() + blob.offset), length); + } + + auto * seekable = dynamic_cast(&buf); + if (!seekable_read || !seekable || !seekable->checkIfActuallySeekable()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot read Puffin blob: input is not seekable and was not buffered"); + + seekable->seek(blob.offset, SEEK_SET); + String result(length, '\0'); + seekable->readStrict(result.data(), length); + return result; +} + +void readDeletionVectorEnvelopePrefix( + const PuffinBlob & blob, ReadBuffer & buf, const std::vector & data, bool seekable_read, UInt8 header[8]) +{ + if (!data.empty()) + { + if (static_cast(blob.offset) + 8 > data.size()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin blob offset/length out of bounds of buffered data"); + std::memcpy(header, data.data() + static_cast(blob.offset), 8); + return; + } + + auto * seekable = dynamic_cast(&buf); + if (!seekable_read || !seekable || !seekable->checkIfActuallySeekable()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot read Puffin blob: input is not seekable and was not buffered"); + + seekable->seek(blob.offset, SEEK_SET); + seekable->readStrict(reinterpret_cast(header), 8); +} + +/// Validate absolute size and the first 8 envelope bytes (combined_length + magic) before allocating +/// `blob.length`. CRC still requires the full payload and is checked later in +/// `extractDeletionVectorPayload` after a bounded read — we intentionally do not stream roaring +/// deserialize here (larger redesign; CRC coverage is over magic+vector). +String readDeletionVectorBlobBytes( + const PuffinBlob & blob, ReadBuffer & buf, const std::vector & data, bool seekable_read) +{ + if (blob.length < 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob length is negative"); + + if (static_cast(blob.length) > PUFFIN_DV_MAX_BLOB_SIZE) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector blob length {} exceeds absolute limit {}", + blob.length, + PUFFIN_DV_MAX_BLOB_SIZE); + + if (static_cast(blob.length) < 12) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob is too small"); + + UInt8 header[8]; + readDeletionVectorEnvelopePrefix(blob, buf, data, seekable_read, header); + + const UInt32 combined_length = readBigEndianUInt32(header); + if (std::memcmp(header + sizeof(UInt32), DELETION_VECTOR_MAGIC, sizeof(DELETION_VECTOR_MAGIC)) != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector magic"); + + if (combined_length < sizeof(DELETION_VECTOR_MAGIC)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + UInt64 expected_blob_size = 0; + if (common::addOverflow(static_cast(combined_length), UInt64{8}, expected_blob_size)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + if (static_cast(blob.length) != expected_blob_size) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector blob size {} does not match combined length {}", + blob.length, + combined_length); + + return readPuffinBlobBytes(blob, buf, data, seekable_read); +} + +roaring::Roaring readRoaringPortableSafe(const char * data, size_t size, Int32 key) +{ + try + { + return roaring::Roaring::readSafe(data, size); + } + catch (const std::exception & e) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Failed to deserialize deletion vector roaring bitmap at key {}: {}", key, e.what()); + } +} + +void deserializeRoaringPositionBitmap(std::string_view bytes, UInt64 expected_cardinality, ColumnUInt64 & positions) +{ + if (bytes.size() < sizeof(Int64)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap is too small"); + + const char * ptr = bytes.data(); + size_t remaining = bytes.size(); + + /// Iceberg deletion-vector roaring layout stores count and keys as little-endian. + const Int64 bitmap_count = unalignedLoadLittleEndian(ptr); + ptr += sizeof(Int64); + remaining -= sizeof(Int64); + + if (bitmap_count < 0 || bitmap_count > std::numeric_limits::max()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector bitmap count: {}", bitmap_count); + + Int32 last_key = -1; + Int32 remaining_count = static_cast(bitmap_count); + UInt64 running_cardinality = 0; + + while (remaining_count > 0) + { + if (remaining < sizeof(Int32)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap is truncated while reading key"); + + const Int32 key = unalignedLoadLittleEndian(ptr); + ptr += sizeof(Int32); + remaining -= sizeof(Int32); + + if (key < 0 || key > DELETION_VECTOR_MAX_KEY) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector bitmap key: {}", key); + if (key <= last_key) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap keys must be sorted in ascending order"); + + auto bitmap = readRoaringPortableSafe(ptr, remaining, key); + + const size_t bitmap_size = bitmap.getSizeInBytes(/*portable=*/true); + if (bitmap_size > remaining) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector roaring bitmap at key {} exceeds blob size", key); + + const UInt64 bitmap_cardinality = bitmap.cardinality(); + UInt64 new_running_cardinality = 0; + if (common::addOverflow(running_cardinality, bitmap_cardinality, new_running_cardinality)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality exceeds declared cardinality {}", + expected_cardinality); + + if (new_running_cardinality > expected_cardinality) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds declared cardinality {}", + new_running_cardinality, + expected_cardinality); + + running_cardinality = new_running_cardinality; + + for (UInt32 sub_position : bitmap) + { + const UInt64 position = positionFromKeyAndSubPosition(static_cast(key), sub_position); + if (position > static_cast(DELETION_VECTOR_MAX_POSITION)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector position {} is out of supported range", position); + positions.insertValue(position); + } + + ptr += bitmap_size; + remaining -= bitmap_size; + last_key = key; + --remaining_count; + } + + if (remaining != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap has {} trailing bytes", remaining); + + if (running_cardinality != expected_cardinality) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} does not match deserialized row count {}", + expected_cardinality, + running_cardinality); +} + +std::string_view extractDeletionVectorPayload(std::string_view blob) +{ + // Envelope size/magic are also checked in readDeletionVectorBlobBytes before allocate; CRC must + // run on the full payload and stays here after the bounded read. + if (blob.size() < 12) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob is too small"); + + const auto * blob_bytes = reinterpret_cast(blob.data()); + const UInt32 combined_length = readBigEndianUInt32(blob_bytes); + if (combined_length < sizeof(DELETION_VECTOR_MAGIC)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + const size_t vector_size = combined_length - sizeof(DELETION_VECTOR_MAGIC); + const size_t expected_blob_size = sizeof(UInt32) + combined_length + sizeof(UInt32); + if (blob.size() != expected_blob_size) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob size {} does not match combined length {}", blob.size(), combined_length); + + if (std::memcmp(blob_bytes + sizeof(UInt32), DELETION_VECTOR_MAGIC, sizeof(DELETION_VECTOR_MAGIC)) != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector magic"); + + const UInt8 * crc_input = blob_bytes + sizeof(UInt32); + const size_t crc_input_size = combined_length; + const UInt32 expected_crc = readBigEndianUInt32(blob_bytes + sizeof(UInt32) + combined_length); + const UInt32 actual_crc = static_cast(crc32_z(0L, reinterpret_cast(crc_input), crc_input_size)); + if (expected_crc != actual_crc) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector CRC mismatch"); + + return std::string_view(blob.data() + 2 * sizeof(UInt32), vector_size); +} + +void deserializeDeletionVectorV1(std::string_view blob, UInt64 expected_cardinality, ColumnUInt64 & positions) +{ + // See PUFFIN_DV_MAX_MATERIALIZED_POSITIONS: hard ceiling before payload parse / array reserve. + if (expected_cardinality > PUFFIN_DV_MAX_MATERIALIZED_POSITIONS) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds materialization limit {}", + expected_cardinality, + PUFFIN_DV_MAX_MATERIALIZED_POSITIONS); + + deserializeRoaringPositionBitmap(extractDeletionVectorPayload(blob), expected_cardinality, positions); +} + +NamesAndTypesList getPuffinMetadataSchema() +{ + return { + {"blob_type", std::make_shared()}, + {"snapshot_id", std::make_shared()}, + {"sequence_number", std::make_shared()}, + {"fields", std::make_shared(std::make_shared())}, + {"offset", std::make_shared()}, + {"length", std::make_shared()}, + {"compression_codec", std::make_shared()}, + {"properties", std::make_shared(std::make_shared(), std::make_shared())}, + }; +} + +NamesAndTypesList getPuffinSchema() +{ + return { + {"referenced_data_file", std::make_shared()}, + {"deleted_rows", std::make_shared(std::make_shared())}, + }; +} + +void checkPuffinFormatHeader(const Block & header, const NamesAndTypesList & expected_schema, const char * format_name) +{ + std::unordered_map name_to_type; + for (const auto & [name, type] : expected_schema) + name_to_type[name] = type; + + String allowed_columns; + for (const auto & [name, type] : expected_schema) + { + if (!allowed_columns.empty()) + allowed_columns += ", "; + allowed_columns += name; + } + + for (const auto & [name, type] : header.getNamesAndTypes()) + { + auto it = name_to_type.find(name); + if (it == name_to_type.end()) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Unexpected column: {}. {} format allows only the next columns: {}", + name, + format_name, + allowed_columns); + + if (!it->second->equals(*type)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Unexpected type {} for column {}. Expected type: {}", + type->getName(), + name, + it->second->getName()); + } +} + +void checkPuffinMetadataHeader(const Block & header) +{ + checkPuffinFormatHeader(header, getPuffinMetadataSchema(), "PuffinMetadata"); +} + +void checkPuffinHeader(const Block & header) +{ + checkPuffinFormatHeader(header, getPuffinSchema(), "Puffin"); +} + +} + +PuffinMetadataInputFormat::PuffinMetadataInputFormat(ReadBuffer & buf, SharedHeader header_, const FormatSettings & format_settings_) + : IInputFormat(std::move(header_), &buf) + , seekable_read(format_settings_.seekable_read) +{ + checkPuffinMetadataHeader(getPort().getHeader()); +} + +Chunk PuffinMetadataInputFormat::read() +{ + if (!initialized) + { + blob_index = 0; + footer = readPuffinFooter(*in, seekable_read); + /// Metadata never reads blob payloads; drop the full-file buffer from the non-seekable path. + footer.data.clear(); + footer.data.shrink_to_fit(); + initialized = true; + } + if (footer.blobs.size() <= blob_index) + return {}; + + const PuffinBlob & blob = footer.blobs[blob_index++]; + + auto col_type = ColumnString::create(); + auto col_snap = ColumnInt64::create(); + auto col_seq = ColumnInt64::create(); + auto col_fields_data = ColumnInt32::create(); + auto col_fields_offsets = ColumnArray::ColumnOffsets::create(); + auto col_offset = ColumnInt64::create(); + auto col_length = ColumnInt64::create(); + auto col_codec = ColumnString::create(); + auto col_props_keys = ColumnString::create(); + auto col_props_vals = ColumnString::create(); + auto col_props_offsets = ColumnArray::ColumnOffsets::create(); + + col_type->insertData(blob.type.data(), blob.type.size()); + col_snap->insertValue(blob.snapshot_id); + col_seq->insertValue(blob.sequence_number); + for (Int32 f : blob.fields) + col_fields_data->insertValue(f); + col_fields_offsets->insertValue(blob.fields.size()); + col_offset->insertValue(blob.offset); + col_length->insertValue(blob.length); + col_codec->insertData(blob.compression_codec.data(), blob.compression_codec.size()); + for (const auto & [k, v] : blob.properties) + { + col_props_keys->insertData(k.data(), k.size()); + col_props_vals->insertData(v.data(), v.size()); + } + col_props_offsets->insertValue(blob.properties.size()); + + auto col_fields = ColumnArray::create(std::move(col_fields_data), std::move(col_fields_offsets)); + MutableColumns prop_cols; + prop_cols.push_back(std::move(col_props_keys)); + prop_cols.push_back(std::move(col_props_vals)); + MutableColumnPtr col_props_tuple = ColumnTuple::create(std::move(prop_cols)); + MutableColumnPtr col_props_arr = ColumnArray::create(std::move(col_props_tuple), std::move(col_props_offsets)); + MutableColumnPtr col_props = ColumnMap::create(std::move(col_props_arr)); + + std::unordered_map built; + built.emplace("blob_type", std::move(col_type)); + built.emplace("snapshot_id", std::move(col_snap)); + built.emplace("sequence_number", std::move(col_seq)); + built.emplace("fields", std::move(col_fields)); + built.emplace("offset", std::move(col_offset)); + built.emplace("length", std::move(col_length)); + built.emplace("compression_codec", std::move(col_codec)); + built.emplace("properties", std::move(col_props)); + + const Block & out_header = getPort().getHeader(); + MutableColumns result; + result.reserve(out_header.columns()); + for (const auto & col_with_name : out_header) + result.push_back(std::move(built.at(col_with_name.name))); + return Chunk(std::move(result), 1); +} + +PuffinInputFormat::PuffinInputFormat(ReadBuffer & buf, SharedHeader header_, const FormatSettings & format_settings_) + : IInputFormat(std::move(header_), &buf) + , seekable_read(format_settings_.seekable_read) +{ + checkPuffinHeader(getPort().getHeader()); + need_deleted_rows = getPort().getHeader().has("deleted_rows"); +} + +Chunk PuffinInputFormat::read() +{ + if (!initialized) + { + blob_index = 0; + footer = readPuffinFooter(*in, seekable_read); + /// No deletion-vector payload will be read; drop the full-file buffer from the non-seekable path. + if (!need_deleted_rows) + { + footer.data.clear(); + footer.data.shrink_to_fit(); + } + initialized = true; + } + + while (blob_index < footer.blobs.size()) + { + const size_t current_blob_index = blob_index; + const auto & blob = footer.blobs[blob_index++]; + + if (blob.type != "deletion-vector-v1") + continue; + + const auto & referenced_data_file = blob.properties.at("referenced-data-file"); + + UInt64 expected_cardinality = 0; + if (!tryParse(expected_cardinality, blob.properties.at("cardinality"))) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 property 'cardinality' must be an unsigned integer", + current_blob_index); + + auto col_file = ColumnString::create(); + col_file->insertData(referenced_data_file.data(), referenced_data_file.size()); + + MutableColumnPtr col_rows; + // Payload I/O and validation (envelope / CRC / roaring) run only when `deleted_rows` is + // projected. Footer property checks above stay projection-independent. + // Intentional: subset reads of `referenced_data_file` must not force reading up to + // PUFFIN_DV_MAX_BLOB_SIZE for blobs that will never be materialized. + // Materialization ceiling uses footer `cardinality` and must run before blob allocate. + if (need_deleted_rows) + { + if (expected_cardinality > PUFFIN_DV_MAX_MATERIALIZED_POSITIONS) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds materialization limit {}", + expected_cardinality, + PUFFIN_DV_MAX_MATERIALIZED_POSITIONS); + + const String blob_data = readDeletionVectorBlobBytes(blob, *in, footer.data, seekable_read); + auto col_rows_data = ColumnUInt64::create(); + deserializeDeletionVectorV1(blob_data, expected_cardinality, *col_rows_data); + + auto col_rows_offsets = ColumnArray::ColumnOffsets::create(); + col_rows_offsets->insertValue(col_rows_data->size()); + col_rows = ColumnArray::create(std::move(col_rows_data), std::move(col_rows_offsets)); + } + + const Block & out_header = getPort().getHeader(); + std::unordered_map built; + built.emplace("referenced_data_file", std::move(col_file)); + if (need_deleted_rows) + built.emplace("deleted_rows", std::move(col_rows)); + + MutableColumns result; + result.reserve(out_header.columns()); + for (const auto & col_with_name : out_header) + result.push_back(std::move(built.at(col_with_name.name))); + return Chunk(std::move(result), 1); + } + + return {}; +} + +void PuffinMetadataInputFormat::resetParser() +{ + IInputFormat::resetParser(); + initialized = false; + blob_index = 0; + footer = {}; +} + +void PuffinInputFormat::resetParser() +{ + IInputFormat::resetParser(); + initialized = false; + blob_index = 0; + footer = {}; +} + +PuffinMetadataSchemaReader::PuffinMetadataSchemaReader(ReadBuffer & in_) + : ISchemaReader(in_) +{ +} + +NamesAndTypesList PuffinMetadataSchemaReader::readSchema() +{ + return getPuffinMetadataSchema(); +} + +PuffinSchemaReader::PuffinSchemaReader(ReadBuffer & in_) + : ISchemaReader(in_) +{ +} + +NamesAndTypesList PuffinSchemaReader::readSchema() +{ + return getPuffinSchema(); +} + +void registerInputFormatPuffin(FormatFactory & factory) +{ + factory.registerInputFormat( + "PuffinMetadata", + [](ReadBuffer & buf, const Block & sample, const RowInputFormatParams &, const FormatSettings & settings) + { return std::make_shared(buf, std::make_shared(sample), settings); }); + factory.markFormatSupportsSubsetOfColumns("PuffinMetadata"); + + factory.setDocumentation("PuffinMetadata", Documentation{ + .description = R"DOCS_MD( +## Description {#description} + +Special input format for reading [Apache Iceberg Puffin](https://iceberg.apache.org/puffin-spec/) file footer metadata. +It outputs one row per blob entry from the footer `BlobMetadata` list. + +Fixed output columns: +- `blob_type` (`String`) - blob type, for example `deletion-vector-v1` +- `snapshot_id` (`Int64`) - snapshot id of the blob +- `sequence_number` (`Int64`) - sequence number of the blob +- `fields` (`Array(Int32)`) - list of field ids the blob applies to +- `offset` (`Int64`) - offset of the blob payload in the file +- `length` (`Int64`) - length of the blob payload in bytes +- `compression_codec` (`String`) - compression codec of the blob payload, if present +- `properties` (`Map(String, String)`) - blob-specific properties + +Optional top-level `FileMetadata.properties` in the footer (for example `created-by`) are type-checked when present but are not returned as columns. If the key is present it must be a JSON object with string values (null is rejected). + +LZ4-compressed and uncompressed puffin footers are supported. Footer payload size (and declared LZ4 content size) is bounded by a compression ratio where applicable and an absolute ceiling; oversized footers are rejected before allocation. + +## Example usage {#example-usage} + +Inspect footer blobs: + +```sql +SELECT blob_type, snapshot_id, sequence_number, offset, length, compression_codec, + mapKeys(properties), mapValues(properties) +FROM file(deletes.puffin, PuffinMetadata); +``` + +Pair with the `Puffin` format to read `deletion-vector-v1` blob payloads. +)DOCS_MD", + .related = {"Puffin"}, + }); + + factory.registerInputFormat( + "Puffin", + [](ReadBuffer & buf, const Block & sample, const RowInputFormatParams &, const FormatSettings & settings) + { return std::make_shared(buf, std::make_shared(sample), settings); }); + factory.markFormatSupportsSubsetOfColumns("Puffin"); + + factory.setDocumentation("Puffin", Documentation{ + .description = R"DOCS_MD( +## Description {#description} + +Input format for reading [Apache Iceberg Puffin](https://iceberg.apache.org/puffin-spec/) files. + +The format exposes deleted row positions from `deletion-vector-v1` blobs. Other blob types (for example `apache-datasketches-theta-v1`) are skipped. +If a puffin file contains multiple `deletion-vector-v1` blobs, the format outputs one row per such blob. + +Fixed output columns: +- `referenced_data_file` (`String`) - location of the data file the deletion vector applies to (`referenced-data-file` blob property) +- `deleted_rows` (`Array(UInt64)`) - 64-bit row positions deleted according to the deletion vector roaring bitmap + +Deletion vectors whose declared `cardinality` exceeds an absolute materialization ceiling are rejected when `deleted_rows` is requested, before the blob is allocated. Footer `deletion-vector-v1` properties (including that `cardinality` parses as an unsigned integer) are always validated. Selecting only `referenced_data_file` skips on-disk payload I/O and therefore also skips envelope, CRC, roaring deserialize, and the materialization ceiling — intentionally, so a path-only projection does not read up to the blob-size cap. + +On-disk `deletion-vector-v1` blob length is bounded by an absolute ceiling (aligned with Iceberg's 2 GiB content-size check). When `deleted_rows` is requested, the reader peeks the envelope header (combined length and magic) before allocating the full payload; CRC is verified after the bounded read. + +LZ4-compressed and uncompressed puffin footers are supported. Footer payload size (and declared LZ4 content size) is bounded by a compression ratio where applicable and an absolute ceiling; oversized footers are rejected before allocation. + +Only a subset of output columns can be requested. A user-provided structure with unexpected column names or types is rejected when the format is created. + +## Example usage {#example-usage} + +Read deleted row positions with the referenced data file: + +```sql +SELECT referenced_data_file, deleted_rows +FROM file(deletes.puffin, Puffin); +``` + +Expand deleted positions into individual rows: + +```sql +SELECT referenced_data_file, row_number +FROM file(deletes.puffin, Puffin) +ARRAY JOIN deleted_rows AS row_number +ORDER BY referenced_data_file, row_number; +``` + +Use `PuffinMetadata` to inspect footer blob descriptors before reading deletion vectors. +)DOCS_MD", + .related = {"PuffinMetadata"}, + }); +} + +void registerPuffinSchemaReaders(FormatFactory & factory) +{ + factory.registerSchemaReader( + "PuffinMetadata", + [](ReadBuffer & buf, const FormatSettings &) + { return std::make_shared(buf); }); + + factory.registerSchemaReader( + "Puffin", + [](ReadBuffer & buf, const FormatSettings &) + { return std::make_shared(buf); }); +} + +} diff --git a/src/Processors/Formats/Impl/PuffinBlockInputFormat.h b/src/Processors/Formats/Impl/PuffinBlockInputFormat.h new file mode 100644 index 000000000000..fc87f950b2f8 --- /dev/null +++ b/src/Processors/Formats/Impl/PuffinBlockInputFormat.h @@ -0,0 +1,83 @@ +#pragma once + +#include +#include +#include + +namespace DB +{ + +struct PuffinBlob +{ + String type; + Int64 snapshot_id = 0; + Int64 sequence_number = 0; + std::vector fields; + Int64 offset = 0; + Int64 length = 0; + String compression_codec; + std::map properties; +}; + +struct PuffinFooter +{ + std::vector blobs; + std::vector data; +}; + +class PuffinMetadataInputFormat : public IInputFormat +{ +public: + PuffinMetadataInputFormat(ReadBuffer & buf, SharedHeader header_, const FormatSettings & format_settings_); + + String getName() const override { return "PuffinMetadata"; } + + void resetParser() override; + +private: + Chunk read() override; + + bool seekable_read = true; + PuffinFooter footer; + bool initialized = false; + size_t blob_index = 0; +}; + +class PuffinInputFormat : public IInputFormat +{ +public: + PuffinInputFormat(ReadBuffer & buf, SharedHeader header_, const FormatSettings & format_settings_); + + String getName() const override { return "Puffin"; } + + void resetParser() override; + +private: + Chunk read() override; + + bool seekable_read = true; + bool need_deleted_rows = true; + PuffinFooter footer; + bool initialized = false; + size_t blob_index = 0; +}; + +class PuffinMetadataSchemaReader : public ISchemaReader +{ +public: + explicit PuffinMetadataSchemaReader(ReadBuffer & in_); + NamesAndTypesList readSchema() override; +}; + +class PuffinSchemaReader : public ISchemaReader +{ +public: + explicit PuffinSchemaReader(ReadBuffer & in_); + NamesAndTypesList readSchema() override; +}; + +class FormatFactory; +void registerInputFormatPuffin(FormatFactory & factory); +void registerPuffinSchemaReaders(FormatFactory & factory); + +} diff --git a/src/Processors/tests/gtest_parquet_v3_need_only_count_buckets.cpp b/src/Processors/tests/gtest_parquet_v3_need_only_count_buckets.cpp new file mode 100644 index 000000000000..7bdbea541e29 --- /dev/null +++ b/src/Processors/tests/gtest_parquet_v3_need_only_count_buckets.cpp @@ -0,0 +1,179 @@ +#include + +#include + +#if USE_PARQUET + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace DB; + +namespace DB::ErrorCodes +{ +extern const int LOGICAL_ERROR; +} + +namespace +{ + +void writeMultiRowGroupParquet(const String & path, size_t rows_per_group, size_t num_groups) +{ + FormatSettings format_settings; + format_settings.parquet.row_group_rows = rows_per_group; + format_settings.parquet.parallel_encoding = false; + + Block header; + header.insert(ColumnWithTypeAndName(std::make_shared(), "x")); + + Chunks chunks; + for (size_t group = 0; group < num_groups; ++group) + { + auto column = ColumnUInt64::create(); + for (size_t i = 0; i < rows_per_group; ++i) + column->insert(group * rows_per_group + i); + chunks.emplace_back(Columns{std::move(column)}, rows_per_group); + } + + auto source = std::make_shared(std::make_shared(header), std::move(chunks)); + QueryPipelineBuilder pipeline_builder; + pipeline_builder.init(Pipe(source)); + auto pipeline = QueryPipelineBuilder::getPipeline(std::move(pipeline_builder)); + + WriteBufferFromFile write_buffer(path); + auto output = std::make_shared(write_buffer, pipeline.getSharedHeader(), format_settings, nullptr); + pipeline.complete(output); + CompletedPipelineExecutor executor(pipeline); + executor.execute(); + output->finalize(); + write_buffer.finalize(); +} + +size_t readNeedOnlyCountTotal( + ReadBuffer & in, + const SharedHeader & header, + const FormatSettings & format_settings, + FormatParserSharedResourcesPtr parser_shared_resources, + const FileBucketInfoPtr & buckets) +{ + auto input = std::make_shared( + in, + header, + format_settings, + parser_shared_resources, + std::make_shared(), + /*min_bytes_for_seek=*/ 1024); + if (buckets) + input->setBucketsToRead(buckets); + input->needOnlyCount(); + + size_t total = 0; + while (true) + { + Chunk chunk = input->generate(); + if (!chunk) + break; + total += chunk.getNumRows(); + } + return total; +} + +std::vector> readNeedOnlyCountOffsetsAndRows( + ReadBuffer & in, + const SharedHeader & header, + const FormatSettings & format_settings, + FormatParserSharedResourcesPtr parser_shared_resources, + const FileBucketInfoPtr & buckets) +{ + auto input = std::make_shared( + in, + header, + format_settings, + parser_shared_resources, + std::make_shared(), + /*min_bytes_for_seek=*/ 1024); + if (buckets) + input->setBucketsToRead(buckets); + input->needOnlyCount(); + + std::vector> result; + while (true) + { + Chunk chunk = input->generate(); + if (!chunk) + break; + auto info = chunk.getChunkInfos().get(); + if (!info) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected ChunkInfoRowNumbers on need_only_count chunk"); + result.emplace_back(info->row_num_offset, chunk.getNumRows()); + } + return result; +} + +} + +TEST(ParquetV3NeedOnlyCountBuckets, CountsOnlyAssignedRowGroups) +{ + tryRegisterFormats(); + const auto context = getContext().context; + + const String path = "/home/iantonspb/ClickHouse/tmp/gtest_parquet_v3_need_only_count_buckets.parquet"; + std::filesystem::create_directories("/home/iantonspb/ClickHouse/tmp"); + constexpr size_t rows_per_group = 10; + constexpr size_t num_groups = 3; + writeMultiRowGroupParquet(path, rows_per_group, num_groups); + + Block header; + header.insert(ColumnWithTypeAndName(std::make_shared(), "x")); + auto shared_header = std::make_shared(header); + FormatSettings format_settings; + auto parser_shared_resources = FormatParserSharedResources::singleThreaded(context->getSettingsRef()); + + { + ReadBufferFromFile in(path); + EXPECT_EQ( + readNeedOnlyCountTotal(in, shared_header, format_settings, parser_shared_resources, /*buckets=*/ nullptr), + rows_per_group * num_groups); + } + + { + ReadBufferFromFile in(path); + auto buckets = std::make_shared(std::vector{1}); + EXPECT_EQ(readNeedOnlyCountTotal(in, shared_header, format_settings, parser_shared_resources, buckets), rows_per_group); + } + + { + ReadBufferFromFile in(path); + auto buckets = std::make_shared(std::vector{0, 2}); + const auto offsets_and_rows + = readNeedOnlyCountOffsetsAndRows(in, shared_header, format_settings, parser_shared_resources, buckets); + ASSERT_EQ(offsets_and_rows.size(), 2u); + EXPECT_EQ(offsets_and_rows[0].first, 0u); + EXPECT_EQ(offsets_and_rows[0].second, rows_per_group); + EXPECT_EQ(offsets_and_rows[1].first, 2 * rows_per_group); + EXPECT_EQ(offsets_and_rows[1].second, rows_per_group); + } + + { + ReadBufferFromFile in(path); + auto buckets = std::make_shared(std::vector{}); + EXPECT_EQ(readNeedOnlyCountTotal(in, shared_header, format_settings, parser_shared_resources, buckets), 0u); + } +} + +#endif diff --git a/src/Processors/tests/gtest_puffin_reset_parser.cpp b/src/Processors/tests/gtest_puffin_reset_parser.cpp new file mode 100644 index 000000000000..c5e9075df2be --- /dev/null +++ b/src/Processors/tests/gtest_puffin_reset_parser.cpp @@ -0,0 +1,130 @@ +#include + +#include +#include +#include +#include +#include +#include + +namespace DB +{ +namespace +{ + +/// Embedded copies of fixtures from tests/queries/0_stateless/data_puffin/. +/// Unit-test CI images do not ship the queries/ tree, so filesystem paths under +/// CLICKHOUSE_TESTS_CONFIG_DIR are unavailable at runtime. +constexpr UInt8 dense_range_100k_puffin[] = { + 0x50, 0x46, 0x41, 0x31, 0x00, 0x00, 0x00, 0x29, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3B, 0x30, 0x01, 0x00, 0x03, 0x00, 0x00, 0xFF, + 0xFF, 0x01, 0x00, 0x9F, 0x86, 0x01, 0x00, 0x00, 0x00, 0xFF, 0xFF, 0x01, 0x00, 0x00, 0x00, 0x9F, + 0x86, 0x66, 0x65, 0xF8, 0x5F, 0x50, 0x46, 0x41, 0x31, 0x7B, 0x22, 0x62, 0x6C, 0x6F, 0x62, 0x73, + 0x22, 0x3A, 0x20, 0x5B, 0x7B, 0x22, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x64, 0x65, + 0x6C, 0x65, 0x74, 0x69, 0x6F, 0x6E, 0x2D, 0x76, 0x65, 0x63, 0x74, 0x6F, 0x72, 0x2D, 0x76, 0x31, + 0x22, 0x2C, 0x20, 0x22, 0x66, 0x69, 0x65, 0x6C, 0x64, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x5D, 0x2C, + 0x20, 0x22, 0x73, 0x6E, 0x61, 0x70, 0x73, 0x68, 0x6F, 0x74, 0x2D, 0x69, 0x64, 0x22, 0x3A, 0x20, + 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x73, 0x65, 0x71, 0x75, 0x65, 0x6E, 0x63, 0x65, 0x2D, 0x6E, 0x75, + 0x6D, 0x62, 0x65, 0x72, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x6F, 0x66, 0x66, 0x73, + 0x65, 0x74, 0x22, 0x3A, 0x20, 0x34, 0x2C, 0x20, 0x22, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x22, + 0x3A, 0x20, 0x34, 0x39, 0x2C, 0x20, 0x22, 0x70, 0x72, 0x6F, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, + 0x73, 0x22, 0x3A, 0x20, 0x7B, 0x22, 0x72, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6E, 0x63, 0x65, 0x64, + 0x2D, 0x64, 0x61, 0x74, 0x61, 0x2D, 0x66, 0x69, 0x6C, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x2F, 0x64, + 0x61, 0x74, 0x61, 0x2F, 0x74, 0x61, 0x62, 0x6C, 0x65, 0x2F, 0x70, 0x61, 0x72, 0x74, 0x2D, 0x30, + 0x30, 0x30, 0x30, 0x30, 0x2E, 0x70, 0x61, 0x72, 0x71, 0x75, 0x65, 0x74, 0x22, 0x2C, 0x20, 0x22, + 0x63, 0x61, 0x72, 0x64, 0x69, 0x6E, 0x61, 0x6C, 0x69, 0x74, 0x79, 0x22, 0x3A, 0x20, 0x22, 0x31, + 0x30, 0x30, 0x30, 0x30, 0x30, 0x22, 0x7D, 0x7D, 0x5D, 0x7D, 0xE1, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x50, 0x46, 0x41, 0x31, +}; + +constexpr UInt8 mixed_blob_types_puffin[] = { + 0x50, 0x46, 0x41, 0x31, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x01, 0x00, 0x10, 0x00, 0x00, 0x00, 0x02, 0x00, 0x05, 0x00, 0x2C, 0xDB, 0x9F, 0xC1, + 0x50, 0x46, 0x41, 0x31, 0x7B, 0x22, 0x62, 0x6C, 0x6F, 0x62, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x7B, + 0x22, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x61, 0x70, 0x61, 0x63, 0x68, 0x65, 0x2D, + 0x64, 0x61, 0x74, 0x61, 0x73, 0x6B, 0x65, 0x74, 0x63, 0x68, 0x65, 0x73, 0x2D, 0x74, 0x68, 0x65, + 0x74, 0x61, 0x2D, 0x76, 0x31, 0x22, 0x2C, 0x20, 0x22, 0x66, 0x69, 0x65, 0x6C, 0x64, 0x73, 0x22, + 0x3A, 0x20, 0x5B, 0x31, 0x5D, 0x2C, 0x20, 0x22, 0x73, 0x6E, 0x61, 0x70, 0x73, 0x68, 0x6F, 0x74, + 0x2D, 0x69, 0x64, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x73, 0x65, 0x71, 0x75, 0x65, + 0x6E, 0x63, 0x65, 0x2D, 0x6E, 0x75, 0x6D, 0x62, 0x65, 0x72, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, + 0x20, 0x22, 0x6F, 0x66, 0x66, 0x73, 0x65, 0x74, 0x22, 0x3A, 0x20, 0x34, 0x2C, 0x20, 0x22, 0x6C, + 0x65, 0x6E, 0x67, 0x74, 0x68, 0x22, 0x3A, 0x20, 0x31, 0x36, 0x2C, 0x20, 0x22, 0x70, 0x72, 0x6F, + 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x22, 0x3A, 0x20, 0x7B, 0x7D, 0x7D, 0x2C, 0x20, 0x7B, + 0x22, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x64, 0x65, 0x6C, 0x65, 0x74, 0x69, 0x6F, + 0x6E, 0x2D, 0x76, 0x65, 0x63, 0x74, 0x6F, 0x72, 0x2D, 0x76, 0x31, 0x22, 0x2C, 0x20, 0x22, 0x66, + 0x69, 0x65, 0x6C, 0x64, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x5D, 0x2C, 0x20, 0x22, 0x73, 0x6E, 0x61, + 0x70, 0x73, 0x68, 0x6F, 0x74, 0x2D, 0x69, 0x64, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, + 0x73, 0x65, 0x71, 0x75, 0x65, 0x6E, 0x63, 0x65, 0x2D, 0x6E, 0x75, 0x6D, 0x62, 0x65, 0x72, 0x22, + 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x6F, 0x66, 0x66, 0x73, 0x65, 0x74, 0x22, 0x3A, 0x20, + 0x32, 0x30, 0x2C, 0x20, 0x22, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x22, 0x3A, 0x20, 0x34, 0x34, + 0x2C, 0x20, 0x22, 0x70, 0x72, 0x6F, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x22, 0x3A, 0x20, + 0x7B, 0x22, 0x72, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6E, 0x63, 0x65, 0x64, 0x2D, 0x64, 0x61, 0x74, + 0x61, 0x2D, 0x66, 0x69, 0x6C, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x2F, 0x64, 0x61, 0x74, 0x61, 0x2F, + 0x74, 0x61, 0x62, 0x6C, 0x65, 0x2F, 0x70, 0x61, 0x72, 0x74, 0x2D, 0x30, 0x30, 0x30, 0x30, 0x30, + 0x2E, 0x70, 0x61, 0x72, 0x71, 0x75, 0x65, 0x74, 0x22, 0x2C, 0x20, 0x22, 0x63, 0x61, 0x72, 0x64, + 0x69, 0x6E, 0x61, 0x6C, 0x69, 0x74, 0x79, 0x22, 0x3A, 0x20, 0x22, 0x32, 0x22, 0x7D, 0x7D, 0x5D, + 0x7D, 0x6D, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x50, 0x46, 0x41, 0x31, +}; + +Block makeSampleBlock(const String & format_name) +{ + EmptyReadBuffer schema_buf; + NamesAndTypesList schema; + if (format_name == "Puffin") + { + PuffinSchemaReader schema_reader(schema_buf); + schema = schema_reader.readSchema(); + } + else + { + PuffinMetadataSchemaReader schema_reader(schema_buf); + schema = schema_reader.readSchema(); + } + + ColumnsWithTypeAndName columns; + for (const auto & [name, type] : schema) + columns.emplace_back(type->createColumn(), type, name); + return Block(std::move(columns)); +} + +size_t executePuffinFormat(const String & format_name, const UInt8 * data, size_t size) +{ + const auto context = getContext().context; + const Block sample = makeSampleBlock(format_name); + + EmptyReadBuffer empty_buffer; + auto format = context->getInputFormat(format_name, empty_buffer, sample, 8192); + StreamingFormatExecutor executor(sample, format); + + ReadBufferFromMemory first_buffer(reinterpret_cast(data), size); + const size_t first_rows = executor.execute(first_buffer); + EXPECT_GT(first_rows, 0u); + + ReadBufferFromMemory second_buffer(reinterpret_cast(data), size); + const size_t second_rows = executor.execute(second_buffer); + EXPECT_EQ(second_rows, first_rows); + + return first_rows; +} + +} + +TEST(PuffinResetParser, PuffinReusedAcrossBuffers) +{ + tryRegisterFormats(); + EXPECT_EQ( + executePuffinFormat("Puffin", dense_range_100k_puffin, sizeof(dense_range_100k_puffin)), + 1u); +} + +TEST(PuffinResetParser, PuffinMetadataReusedAcrossBuffers) +{ + tryRegisterFormats(); + EXPECT_EQ( + executePuffinFormat("PuffinMetadata", mixed_blob_types_puffin, sizeof(mixed_blob_types_puffin)), + 2u); +} + +} diff --git a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp index 4517834d0dec..1d5dcfff1f1d 100644 --- a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp @@ -16,6 +16,7 @@ #include #include #include +#include namespace DB::ErrorCodes { @@ -242,6 +243,22 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE const auto record_count = getValueFromRowByName(row_index, c_data_file_record_count, TypeIndex::Int64).safeGet(); const auto file_size_in_bytes = getValueFromRowByName(row_index, c_data_file_file_size_in_bytes, TypeIndex::Int64).safeGet(); + std::optional content_offset; + if (hasPath(c_data_file_content_offset)) + { + const auto content_offset_value = getValueFromRowByName(row_index, c_data_file_content_offset); + if (!content_offset_value.isNull()) + content_offset = content_offset_value.safeGet(); + } + + std::optional content_size_in_bytes; + if (hasPath(c_data_file_content_size_in_bytes)) + { + const auto content_size_value = getValueFromRowByName(row_index, c_data_file_content_size_in_bytes); + if (!content_size_value.isNull()) + content_size_in_bytes = content_size_value.safeGet(); + } + switch (content_type) { case FileContentType::DATA: { @@ -292,6 +309,25 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE upper_reference_data_file_path.emplace(Iceberg::IcebergPathFromMetadata::deserialize(upper.safeGet())); } } + + if (Poco::toLower(file_format) == "puffin") + { + if (!content_offset.has_value() || !content_size_in_bytes.has_value()) + { + throw Exception( + DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Puffin deletion vector entry in manifest file '{}' is missing content_offset or content_size_in_bytes", + manifest_file_path); + } + if (!lower_reference_data_file_path.has_value()) + { + throw Exception( + DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Puffin deletion vector entry in manifest file '{}' is missing referenced_data_file", + manifest_file_path); + } + } + return std::make_shared( FileContentType::POSITION_DELETE, file_path_key, @@ -308,7 +344,9 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE /*equality_ids*/ std::nullopt, /*sort_order_id = */ std::nullopt, record_count, - file_size_in_bytes); + file_size_in_bytes, + content_offset, + content_size_in_bytes); } case FileContentType::EQUALITY_DELETE: { std::vector equality_ids; diff --git a/src/Storages/ObjectStorage/DataLakes/DataLakeObjectMetadata.h b/src/Storages/ObjectStorage/DataLakes/DataLakeObjectMetadata.h index fcc2c0495e18..566b1965e0df 100644 --- a/src/Storages/ObjectStorage/DataLakes/DataLakeObjectMetadata.h +++ b/src/Storages/ObjectStorage/DataLakes/DataLakeObjectMetadata.h @@ -1,6 +1,8 @@ #pragma once #include +#include + namespace DB { template @@ -17,4 +19,11 @@ struct DataLakeObjectMetadata ExcludedRowsPtr excluded_rows; }; +/// True when a deletion vector (or similar) will filter rows via DeletionVectorTransform. +/// Count-from-files cache must skip these objects: the cache key is data-file identity only, +/// while excluded_rows can change independently (Iceberg puffin DVs, DeltaLake selection vectors). +/// Cluster task serialization must also fail closed when these cannot be carried on the wire. +bool hasNonEmptyExcludedRows(const DataLakeObjectMetadata & metadata); +bool hasNonEmptyExcludedRows(const std::optional & metadata); + } diff --git a/src/Storages/ObjectStorage/DataLakes/DeletionVectorTransform.cpp b/src/Storages/ObjectStorage/DataLakes/DeletionVectorTransform.cpp index 3fd353ef99a1..9b21a7bb8b0b 100644 --- a/src/Storages/ObjectStorage/DataLakes/DeletionVectorTransform.cpp +++ b/src/Storages/ObjectStorage/DataLakes/DeletionVectorTransform.cpp @@ -13,6 +13,16 @@ namespace DB::ErrorCodes namespace DB { +bool hasNonEmptyExcludedRows(const DataLakeObjectMetadata & metadata) +{ + return metadata.excluded_rows && metadata.excluded_rows->size() > 0; +} + +bool hasNonEmptyExcludedRows(const std::optional & metadata) +{ + return metadata.has_value() && hasNonEmptyExcludedRows(*metadata); +} + DeletionVectorTransform::DeletionVectorTransform( const DB::SharedHeader & header_, ExcludedRowsPtr excluded_rows_) diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h index 6495b22f7cf0..0bb9b586326b 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h @@ -175,6 +175,8 @@ DEFINE_ICEBERG_FIELD_COMPOUND(data_file, null_value_counts); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, lower_bounds); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, upper_bounds); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, referenced_data_file); +DEFINE_ICEBERG_FIELD_COMPOUND(data_file, content_offset); +DEFINE_ICEBERG_FIELD_COMPOUND(data_file, content_size_in_bytes); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, sort_order_id); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, record_count); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, file_size_in_bytes); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.cpp index 4e161ca8c863..6e99248bcba4 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.cpp @@ -15,6 +15,7 @@ #include #include +#include namespace DB::ErrorCodes { @@ -25,6 +26,23 @@ extern const int UNKNOWN_PROTOCOL; using namespace DB::Iceberg; +namespace DB::Iceberg +{ + +void requireParquetDataFileForRowDeletes(const String & file_format, std::string_view feature_name) +{ + if (Poco::toUpper(file_format) != "PARQUET") + { + throw Exception( + DB::ErrorCodes::NOT_IMPLEMENTED, + "{} are only supported for data files of Parquet format in Iceberg, but got {}", + feature_name, + file_format); + } +} + +} + namespace DB { @@ -97,13 +115,7 @@ std::shared_ptr IcebergDataObjectInfo::getPositionDeleteTransf void IcebergDataObjectInfo::addPositionDeleteObject(Iceberg::ProcessedManifestFileEntryPtr position_delete_object, const String & resolved_storage_path) { - if (Poco::toUpper(info.file_format) != "PARQUET") - { - throw Exception( - ErrorCodes::NOT_IMPLEMENTED, - "Position deletes are only supported for data files of Parquet format in Iceberg, but got {}", - info.file_format); - } + Iceberg::requireParquetDataFileForRowDeletes(info.file_format, "Position deletes"); info.position_deletes_objects.emplace_back( resolved_storage_path, position_delete_object->parsed_entry->file_format, std::nullopt, position_delete_object->sequence_number); @@ -118,6 +130,26 @@ void IcebergDataObjectInfo::addEqualityDeleteObject(const Iceberg::ProcessedMani equality_delete_object->resolved_schema_id); } +ObjectInfoPtr IcebergDataObjectInfo::clone() const +{ + auto result = std::make_shared(relative_path_with_metadata, info); + result->data_lake_metadata = data_lake_metadata; + result->file_bucket_info = file_bucket_info; + return result; +} + +bool hasIcebergEqualityDeletes(const ObjectInfoPtr & object_info) +{ + const auto * iceberg = dynamic_cast(object_info.get()); + return iceberg && !iceberg->info.equality_deletes_objects.empty(); +} + +bool hasIcebergPositionDeletes(const ObjectInfoPtr & object_info) +{ + const auto * iceberg = dynamic_cast(object_info.get()); + return iceberg && !iceberg->info.position_deletes_objects.empty(); +} + #endif void IcebergObjectSerializableInfo::serializeForClusterFunctionProtocol(WriteBuffer & out, size_t protocol_version) const diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h index 6763d1013201..9e646eedd52c 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h @@ -11,12 +11,18 @@ #include #include +#include + namespace DB::Iceberg { String computePartitionId(const Row & partition_key_value); +/// Position deletes and deletion vectors need file-relative row numbers (`ChunkInfoRowNumbers`), +/// which Parquet readers emit. Reject other data-file formats early with NOT_IMPLEMENTED. +void requireParquetDataFileForRowDeletes(const String & file_format, std::string_view feature_name); + struct IcebergObjectSerializableInfo { @@ -84,10 +90,18 @@ struct IcebergDataObjectInfo : public ObjectInfo, std::enable_shared_from_this; + +/// Equality deletes need real column values — disable need_only_count / count-from-files cache. +bool hasIcebergEqualityDeletes(const ObjectInfoPtr & object_info); +/// Position-delete attachment changes independently of data-file identity — skip count-from-files cache. +bool hasIcebergPositionDeletes(const ObjectInfoPtr & object_info); } #endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp new file mode 100644 index 000000000000..4497a0aed812 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp @@ -0,0 +1,168 @@ +#include "config.h" + +#if USE_AVRO + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int ICEBERG_SPECIFICATION_VIOLATION; +} + +namespace Setting +{ +extern const SettingsBool use_puffin_files_cache; +} + +} + +namespace DB::Iceberg +{ + +namespace +{ + +DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( + ObjectStoragePtr object_storage, + const String & puffin_path, + Int64 content_offset, + Int64 content_size_in_bytes, + const IcebergPathFromMetadata & expected_data_file, + UInt64 expected_cardinality, + ContextPtr context, + LoggerPtr log, + bool disable_filesystem_cache) +{ + RelativePathWithMetadata puffin_object{puffin_path}; + auto read_settings = context->getReadSettings(); + if (disable_filesystem_cache) + read_settings.enable_filesystem_cache = false; + + auto read_buffer = createReadBuffer(puffin_object, object_storage, context, log, read_settings); + auto deleted_positions = readDeletionVectorFromPuffin( + *read_buffer, content_offset, content_size_in_bytes, expected_cardinality); + + if (deleted_positions.empty()) + return nullptr; + + auto bitmap = std::make_shared(); + for (UInt64 position : deleted_positions) + bitmap->add(static_cast(position)); + + LOG_DEBUG( + log, + "Loaded deletion vector from puffin file '{}' for data file '{}': {} deleted rows", + puffin_path, + expected_data_file.serialize(), + deleted_positions.size()); + + return bitmap; +} + +} + +DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVector( + ObjectStoragePtr object_storage, + const String & puffin_path, + Int64 content_offset, + Int64 content_size_in_bytes, + const IcebergPathFromMetadata & expected_data_file, + const std::optional & referenced_data_file, + Int64 expected_cardinality, + ContextPtr context, + LoggerPtr log) +{ + if (referenced_data_file.has_value() && referenced_data_file.value() != expected_data_file) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Deletion vector referenced_data_file '{}' does not match data file '{}'", + referenced_data_file->serialize(), + expected_data_file.serialize()); + } + + if (expected_cardinality < 0) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Deletion vector record_count {} must be non-negative", + expected_cardinality); + } + + const UInt64 expected_cardinality_u64 = static_cast(expected_cardinality); + + const bool use_cache = context->getSettingsRef()[Setting::use_puffin_files_cache]; + if (!use_cache) + { + LOG_TRACE(log, "Not using Puffin files cache for '{}', because the setting use_puffin_files_cache is false", puffin_path); + return loadDeletionVectorUncached( + object_storage, + puffin_path, + content_offset, + content_size_in_bytes, + expected_data_file, + expected_cardinality_u64, + context, + log, + false); + } + + RelativePathWithMetadata puffin_object{puffin_path}; + if (!puffin_object.metadata) + puffin_object.metadata = object_storage->getObjectMetadata(puffin_object.getPath(), /*with_tags=*/ false); + + auto cache_key = PuffinFilesCache::tryCreateKey( + puffin_path, + puffin_object.metadata->etag, + content_offset, + content_size_in_bytes, + expected_data_file.serialize()); + + if (!cache_key) + { + LOG_TRACE(log, "Not using Puffin files cache for '{}', because etag is empty", puffin_path); + return loadDeletionVectorUncached( + object_storage, + puffin_path, + content_offset, + content_size_in_bytes, + expected_data_file, + expected_cardinality_u64, + context, + log, + false); + } + + auto cache = context->getPuffinFilesCache(); + return cache->getOrSetDeletionVector(*cache_key, [&]() + { + return loadDeletionVectorUncached( + object_storage, + puffin_path, + content_offset, + content_size_in_bytes, + expected_data_file, + expected_cardinality_u64, + context, + log, + true); + }); +} + +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.h new file mode 100644 index 000000000000..59ec8005aed2 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.h @@ -0,0 +1,28 @@ +#pragma once +#include "config.h" + +#if USE_AVRO + +#include +#include +#include +#include +#include + +namespace DB::Iceberg +{ + +DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVector( + ObjectStoragePtr object_storage, + const String & puffin_path, + Int64 content_offset, + Int64 content_size_in_bytes, + const IcebergPathFromMetadata & expected_data_file, + const std::optional & referenced_data_file, + Int64 expected_cardinality, + ContextPtr context, + LoggerPtr log); + +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp index 39cc26a5fd6a..b624c2216aae 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp @@ -43,6 +43,7 @@ #include #include #include +#include #include #include @@ -71,6 +72,7 @@ namespace DB namespace ErrorCodes { extern const int LOGICAL_ERROR; +extern const int ICEBERG_SPECIFICATION_VIOLATION; } namespace Setting { @@ -270,6 +272,7 @@ IcebergIterator::IcebergIterator( persistent_components_) , blocking_queue(100) , callback(std::move(callback_)) + , local_context(local_context_) { auto delete_file = deletes_iterator.next(); while (delete_file.has_value()) @@ -278,15 +281,25 @@ IcebergIterator::IcebergIterator( { equality_deletes_files.emplace_back(std::move(delete_file.value())); } + else if (delete_file.value()->parsed_entry->isDeletionVector()) + { + deletion_vector_files.emplace_back(std::move(delete_file.value())); + } else { - position_deletes_files.emplace_back(std::move(delete_file.value())); + parquet_position_deletes_files.emplace_back(std::move(delete_file.value())); } delete_file = deletes_iterator.next(); } - LOG_DEBUG(logger, "Taken {} position deletes file and {} equality deletes files in iceberg iterator", position_deletes_files.size(), equality_deletes_files.size()); + LOG_DEBUG( + logger, + "Taken {} deletion vector files, {} parquet position delete files and {} equality delete files in iceberg iterator", + deletion_vector_files.size(), + parquet_position_deletes_files.size(), + equality_deletes_files.size()); std::sort(equality_deletes_files.begin(), equality_deletes_files.end()); - std::sort(position_deletes_files.begin(), position_deletes_files.end()); + std::sort(deletion_vector_files.begin(), deletion_vector_files.end()); + std::sort(parquet_position_deletes_files.begin(), parquet_position_deletes_files.end()); producer_task = std::make_unique( [this, thread_group = CurrentThread::getGroup()]() { @@ -333,42 +346,88 @@ ObjectInfoPtr IcebergIterator::next(size_t) manifest_file_entry, persistent_components.path_resolver.resolve(manifest_file_entry->parsed_entry->file_path_key), table_state_snapshot->schema_id); - for (const auto & position_delete : - defineDeletesSpan(manifest_file_entry, position_deletes_files, /* is_equality_delete */ false, logger)) + + const auto & data_file_path = object_info->info.data_object_file_path_key; + bool has_deletion_vector = false; + + for (const auto & deletion_vector : + defineDeletesSpan(manifest_file_entry, deletion_vector_files, /* is_equality_delete */ false, logger)) { - const auto & data_file_path = object_info->info.data_object_file_path_key; - const auto & lower = position_delete->parsed_entry->lower_reference_data_file_path; - const auto & upper = position_delete->parsed_entry->upper_reference_data_file_path; - bool can_contain_data_file_deletes - = (!lower.has_value() || *lower <= data_file_path) - && (!upper.has_value() || *upper >= data_file_path); - /// Skip position deletes that do not match the data file path. - if (!can_contain_data_file_deletes) + const auto & referenced_data_file = deletion_vector->parsed_entry->lower_reference_data_file_path; + if (!referenced_data_file.has_value() || referenced_data_file.value() != data_file_path) + continue; + + if (has_deletion_vector) { - ProfileEvents::increment(ProfileEvents::IcebergMinMaxPrunedDeleteFiles); - LOG_TEST( - logger, - "Skipping position delete file `{}` for data file `{}` because position delete has out of bounds reference data file " - "bounds: " - "(lower bound: `{}`, upper bound: `{}`)", - position_delete->parsed_entry->file_path_key, - data_file_path, - lower.has_value() ? lower->serialize() : "[no lower bound]", - upper.has_value() ? upper->serialize() : "[no upper bound]"); + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Multiple deletion vectors match data file '{}'", + data_file_path); } - else + + Iceberg::requireParquetDataFileForRowDeletes(object_info->info.file_format, "Deletion vectors"); + + const auto & parsed_entry = deletion_vector->parsed_entry; + auto excluded_rows = Iceberg::loadDeletionVector( + object_storage, + persistent_components.path_resolver.resolve(parsed_entry->file_path_key), + parsed_entry->content_offset.value(), + parsed_entry->content_size_in_bytes.value(), + data_file_path, + referenced_data_file, + parsed_entry->record_count, + local_context, + logger); + + object_info->data_lake_metadata.emplace(); + if (excluded_rows) + object_info->data_lake_metadata->excluded_rows = std::move(excluded_rows); + has_deletion_vector = true; + LOG_DEBUG( + logger, + "Attached deletion vector from puffin file `{}` to data file `{}`", + parsed_entry->file_path_key, + data_file_path); + } + + if (!has_deletion_vector) + { + for (const auto & position_delete : + defineDeletesSpan(manifest_file_entry, parquet_position_deletes_files, /* is_equality_delete */ false, logger)) { - ProfileEvents::increment(ProfileEvents::IcebergMinMaxNonPrunedDeleteFiles); - LOG_TEST( - logger, - "Processing position delete file `{}` for data file `{}` with reference data file bounds: " - "(lower bound: `{}`, upper bound: `{}`)", - position_delete->parsed_entry->file_path_key, - data_file_path, - lower.has_value() ? lower->serialize() : "[no lower bound]", - upper.has_value() ? upper->serialize() : "[no upper bound]"); - object_info->addPositionDeleteObject( - position_delete, persistent_components.path_resolver.resolve(position_delete->parsed_entry->file_path_key)); + const auto & lower = position_delete->parsed_entry->lower_reference_data_file_path; + const auto & upper = position_delete->parsed_entry->upper_reference_data_file_path; + bool can_contain_data_file_deletes + = (!lower.has_value() || *lower <= data_file_path) + && (!upper.has_value() || *upper >= data_file_path); + /// Skip position deletes that do not match the data file path. + if (!can_contain_data_file_deletes) + { + ProfileEvents::increment(ProfileEvents::IcebergMinMaxPrunedDeleteFiles); + LOG_TEST( + logger, + "Skipping position delete file `{}` for data file `{}` because position delete has out of bounds reference data file " + "bounds: " + "(lower bound: `{}`, upper bound: `{}`)", + position_delete->parsed_entry->file_path_key, + data_file_path, + lower.has_value() ? lower->serialize() : "[no lower bound]", + upper.has_value() ? upper->serialize() : "[no upper bound]"); + } + else + { + ProfileEvents::increment(ProfileEvents::IcebergMinMaxNonPrunedDeleteFiles); + LOG_TEST( + logger, + "Processing position delete file `{}` for data file `{}` with reference data file bounds: " + "(lower bound: `{}`, upper bound: `{}`)", + position_delete->parsed_entry->file_path_key, + data_file_path, + lower.has_value() ? lower->serialize() : "[no lower bound]", + upper.has_value() ? upper->serialize() : "[no upper bound]"); + object_info->addPositionDeleteObject( + position_delete, persistent_components.path_resolver.resolve(position_delete->parsed_entry->file_path_key)); + } } } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h index 09c21b8ec292..973c40440470 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h @@ -93,8 +93,10 @@ class IcebergIterator : public IObjectIterator ConcurrentBoundedQueue blocking_queue; std::unique_ptr producer_task; IDataLakeMetadata::FileProgressCallback callback; - std::vector position_deletes_files; + std::vector deletion_vector_files; + std::vector parquet_position_deletes_files; std::vector equality_deletes_files; + ContextPtr local_context; std::exception_ptr exception; std::mutex exception_mutex; }; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp index fd46034ee62a..3817e3f1e7d8 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp @@ -392,6 +392,7 @@ IcebergDataSnapshotPtr IcebergMetadata::createIcebergDataSnapshotFromSnapshotJSO std::optional total_rows; std::optional total_bytes; std::optional total_position_deletes; + std::optional total_equality_deletes; if (snapshot_object->has(f_summary)) { @@ -406,6 +407,9 @@ IcebergDataSnapshotPtr IcebergMetadata::createIcebergDataSnapshotFromSnapshotJSO { total_position_deletes = summary_object->getValue(f_total_position_deletes); } + + if (summary_object->has(f_total_equality_deletes)) + total_equality_deletes = summary_object->getValue(f_total_equality_deletes); } if (!snapshot_object->has(f_schema_id)) @@ -419,7 +423,8 @@ IcebergDataSnapshotPtr IcebergMetadata::createIcebergDataSnapshotFromSnapshotJSO schema_id, total_rows, total_bytes, - total_position_deletes); + total_position_deletes, + total_equality_deletes); } IcebergDataSnapshotPtr @@ -1047,13 +1052,22 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const return 0; } + /// Equality deletes remove data rows by value match; summary `total-equality-deletes` counts + /// rows in delete files, not deleted data rows. Fail closed when the field is present and > 0. + /// If the field is absent, skip the summary shortcut and scan manifests for EQUALITY_DELETE files. + if (actual_data_snapshot->total_equality_delete_rows.has_value() + && *actual_data_snapshot->total_equality_delete_rows > 0) + return {}; /// All these "hints" with total rows or bytes are optional both in /// metadata files and in manifest files, so we try all of them one by one - if (auto total_rows = actual_data_snapshot->getTotalRows(); total_rows.has_value()) + if (actual_data_snapshot->allowsSnapshotTotalRowsShortcut()) { - ProfileEvents::increment(ProfileEvents::IcebergTrivialCountOptimizationApplied); - return total_rows; + if (auto total_rows = actual_data_snapshot->getTotalRows(); total_rows.has_value()) + { + ProfileEvents::increment(ProfileEvents::IcebergTrivialCountOptimizationApplied); + return total_rows; + } } Int64 result = 0; @@ -1061,6 +1075,9 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const { auto manifest_file_ptr = getManifestFileEntriesHandle( object_storage, persistent_components, local_context, log, manifest_list_entry, actual_table_state_snapshot.schema_id); + if (!manifest_file_ptr.getFilesWithoutDeleted(FileContentType::EQUALITY_DELETE).empty()) + return {}; + auto data_count = manifest_file_ptr.getRowsCountInAllFilesExcludingDeleted(FileContentType::DATA); auto position_deletes_count = manifest_file_ptr.getRowsCountInAllFilesExcludingDeleted(FileContentType::POSITION_DELETE); if (!data_count.has_value() || !position_deletes_count.has_value()) diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h index f7c2ced00bf3..9e7a71f3ea64 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h @@ -15,6 +15,7 @@ #include #include +#include namespace DB::Iceberg { @@ -94,6 +95,17 @@ struct ParsedManifestFileEntry : boost::noncopyable Int64 record_count; Int64 file_size_in_bytes; + /// Iceberg v3 deletion vector metadata (position delete entries with puffin format) + std::optional content_offset; + std::optional content_size_in_bytes; + + bool isDeletionVector() const + { + return Poco::toLower(file_format) == "puffin" + && content_offset.has_value() + && content_size_in_bytes.has_value(); + } + ParsedManifestFileEntry( FileContentType content_type_, IcebergPathFromMetadata file_path_key_, @@ -110,7 +122,9 @@ struct ParsedManifestFileEntry : boost::noncopyable std::optional> equality_ids_, std::optional sort_order_id_, Int64 record_count_, - Int64 file_size_in_bytes_) + Int64 file_size_in_bytes_, + std::optional content_offset_ = std::nullopt, + std::optional content_size_in_bytes_ = std::nullopt) : content_type(content_type_) , file_path_key(std::move(file_path_key_)) , row_number(row_number_) @@ -127,6 +141,8 @@ struct ParsedManifestFileEntry : boost::noncopyable , sort_order_id(sort_order_id_) , record_count(record_count_) , file_size_in_bytes(file_size_in_bytes_) + , content_offset(content_offset_) + , content_size_in_bytes(content_size_in_bytes_) { } }; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h index 123a67ddc40b..a08e5dec87a7 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h @@ -19,6 +19,9 @@ struct IcebergDataSnapshot std::optional total_rows; std::optional total_bytes; std::optional total_position_delete_rows; + /// Rows in equality-delete files (snapshot summary). Not a count of deleted data rows; + /// used only to fail closed trivial COUNT when equality deletes are present. + std::optional total_equality_delete_rows; std::optional getTotalRows() const { @@ -26,6 +29,13 @@ struct IcebergDataSnapshot return *total_rows - *total_position_delete_rows; return std::nullopt; } + + /// Summary `total-equality-deletes` is optional. Only trust the cheap getTotalRows() shortcut + /// when the field is present and explicitly zero; absent or >0 must fall through / fail closed. + bool allowsSnapshotTotalRowsShortcut() const + { + return total_equality_delete_rows.has_value() && *total_equality_delete_rows == 0; + } }; using IcebergDataSnapshotPtr = std::shared_ptr; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp new file mode 100644 index 000000000000..5fe1ee441d9a --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp @@ -0,0 +1,66 @@ +#include + +#include "config.h" + +#if USE_AVRO + +#include +#include +#include + +using namespace DB; + +TEST(IcebergCountShortcuts, HasEqualityAndPositionDeleteHelpers) +{ + Iceberg::IcebergObjectSerializableInfo info; + info.data_object_file_path_key = Iceberg::IcebergPathFromMetadata::deserialize("s3://bucket/data/file.parquet"); + info.file_format = "PARQUET"; + + auto plain = std::make_shared(RelativePathWithMetadata{"data/file.parquet"}); + EXPECT_FALSE(hasIcebergEqualityDeletes(plain)); + EXPECT_FALSE(hasIcebergPositionDeletes(plain)); + + auto iceberg = std::make_shared(RelativePathWithMetadata{"data/file.parquet"}, info); + EXPECT_FALSE(hasIcebergEqualityDeletes(iceberg)); + EXPECT_FALSE(hasIcebergPositionDeletes(iceberg)); + + iceberg->info.equality_deletes_objects.push_back( + Iceberg::EqualityDeleteObject{ + .file_path = "s3://bucket/deletes/eq.parquet", + .file_format = "PARQUET", + .equality_ids = std::vector{1}, + .schema_id = 0, + }); + EXPECT_TRUE(hasIcebergEqualityDeletes(iceberg)); + EXPECT_FALSE(hasIcebergPositionDeletes(iceberg)); + + iceberg->info.position_deletes_objects.push_back( + Iceberg::PositionDeleteObject{ + .file_path = "s3://bucket/deletes/pos.parquet", + .file_format = "PARQUET", + .reference_data_file_path = std::nullopt, + .sequence_number = 1, + }); + EXPECT_TRUE(hasIcebergEqualityDeletes(iceberg)); + EXPECT_TRUE(hasIcebergPositionDeletes(iceberg)); +} + +TEST(IcebergCountShortcuts, SnapshotShortcutRequiresExplicitZeroEqualityDeletes) +{ + Iceberg::IcebergDataSnapshot snapshot; + snapshot.total_rows = 100; + snapshot.total_position_delete_rows = 10; + + /// Field absent: getTotalRows() would succeed, but shortcut must not be used. + ASSERT_TRUE(snapshot.getTotalRows().has_value()); + EXPECT_FALSE(snapshot.allowsSnapshotTotalRowsShortcut()); + + snapshot.total_equality_delete_rows = 5; + EXPECT_FALSE(snapshot.allowsSnapshotTotalRowsShortcut()); + + snapshot.total_equality_delete_rows = 0; + EXPECT_TRUE(snapshot.allowsSnapshotTotalRowsShortcut()); + EXPECT_EQ(*snapshot.getTotalRows(), 90u); +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_data_object_info_clone.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_data_object_info_clone.cpp new file mode 100644 index 000000000000..b863ee605446 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_data_object_info_clone.cpp @@ -0,0 +1,64 @@ +#include + +#include "config.h" + +#if USE_AVRO + +#include +#include +#include +#include + +using namespace DB; + +TEST(IcebergDataObjectInfoClone, PreservesEqualityAndPositionDeletes) +{ + Iceberg::IcebergObjectSerializableInfo info; + info.data_object_file_path_key = Iceberg::IcebergPathFromMetadata::deserialize("s3://bucket/data/file.parquet"); + info.file_format = "PARQUET"; + info.equality_deletes_objects.push_back( + Iceberg::EqualityDeleteObject{ + .file_path = "s3://bucket/deletes/eq.parquet", + .file_format = "PARQUET", + .equality_ids = std::vector{1, 2}, + .schema_id = 7, + }); + info.position_deletes_objects.push_back( + Iceberg::PositionDeleteObject{ + .file_path = "s3://bucket/deletes/pos.parquet", + .file_format = "PARQUET", + .reference_data_file_path = std::nullopt, + .sequence_number = 42, + }); + + auto original = std::make_shared( + RelativePathWithMetadata{"data/file.parquet"}, info); + original->data_lake_metadata.emplace(); + original->data_lake_metadata->excluded_rows = std::make_shared(); + original->data_lake_metadata->excluded_rows->add(11); + + /// Object-slicing (the old ObjectIteratorSplitByBuckets path) drops Iceberg metadata. + ObjectInfo sliced_value = *original; + auto sliced = std::make_shared(sliced_value); + EXPECT_FALSE(std::dynamic_pointer_cast(sliced)); + ASSERT_TRUE(sliced->data_lake_metadata.has_value()); + ASSERT_TRUE(sliced->data_lake_metadata->excluded_rows); + EXPECT_EQ(sliced->data_lake_metadata->excluded_rows->size(), 1u); + + auto cloned = original->clone(); + auto iceberg_cloned = std::dynamic_pointer_cast(cloned); + ASSERT_TRUE(iceberg_cloned); + ASSERT_EQ(iceberg_cloned->info.equality_deletes_objects.size(), 1u); + EXPECT_EQ(iceberg_cloned->info.equality_deletes_objects[0].file_path, "s3://bucket/deletes/eq.parquet"); + ASSERT_TRUE(iceberg_cloned->info.equality_deletes_objects[0].equality_ids.has_value()); + EXPECT_EQ(iceberg_cloned->info.equality_deletes_objects[0].equality_ids->size(), 2u); + ASSERT_EQ(iceberg_cloned->info.position_deletes_objects.size(), 1u); + EXPECT_EQ(iceberg_cloned->info.position_deletes_objects[0].file_path, "s3://bucket/deletes/pos.parquet"); + EXPECT_EQ(iceberg_cloned->info.position_deletes_objects[0].sequence_number, 42); + ASSERT_TRUE(iceberg_cloned->data_lake_metadata.has_value()); + ASSERT_TRUE(iceberg_cloned->data_lake_metadata->excluded_rows); + EXPECT_EQ(iceberg_cloned->data_lake_metadata->excluded_rows->size(), 1u); + EXPECT_TRUE(iceberg_cloned->data_lake_metadata->excluded_rows->rb_contains(11)); +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_parquet_row_deletes_guard.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_parquet_row_deletes_guard.cpp new file mode 100644 index 000000000000..ce5262728d1b --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_parquet_row_deletes_guard.cpp @@ -0,0 +1,47 @@ +#include + +#include +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int NOT_IMPLEMENTED; +} +} + +TEST(IcebergParquetRowDeletesGuard, AcceptsParquet) +{ + EXPECT_NO_THROW(Iceberg::requireParquetDataFileForRowDeletes("parquet", "Deletion vectors")); + EXPECT_NO_THROW(Iceberg::requireParquetDataFileForRowDeletes("PARQUET", "Position deletes")); + EXPECT_NO_THROW(Iceberg::requireParquetDataFileForRowDeletes("Parquet", "Deletion vectors")); +} + +TEST(IcebergParquetRowDeletesGuard, RejectsNonParquet) +{ + try + { + Iceberg::requireParquetDataFileForRowDeletes("ORC", "Deletion vectors"); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::NOT_IMPLEMENTED); + EXPECT_NE(e.message().find("Deletion vectors are only supported"), std::string::npos); + EXPECT_NE(e.message().find("ORC"), std::string::npos); + } + + try + { + Iceberg::requireParquetDataFileForRowDeletes("AVRO", "Position deletes"); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::NOT_IMPLEMENTED); + EXPECT_NE(e.message().find("Position deletes are only supported"), std::string::npos); + } +} diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp new file mode 100644 index 000000000000..4cf59f725ffe --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp @@ -0,0 +1,291 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include + +namespace ProfileEvents +{ +extern const Event PuffinFilesRead; +extern const Event PuffinFileReadMicroseconds; +} + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int BAD_ARGUMENTS; +} + +namespace +{ + +struct ScopedPuffinFileReadProfileEvent +{ + ProfileEventTimeIncrement watch; + + ScopedPuffinFileReadProfileEvent() + : watch(ProfileEvents::PuffinFileReadMicroseconds) + { + ProfileEvents::increment(ProfileEvents::PuffinFilesRead); + } +}; + +constexpr UInt8 DELETION_VECTOR_MAGIC[4] = {0xD1, 0xD3, 0x39, 0x64}; +/// Matches Iceberg DeleteLoader / Puffin format absolute blob-size ceiling. +constexpr size_t PUFFIN_DV_MAX_BLOB_SIZE = 2ULL * 1024 * 1024 * 1024; +constexpr UInt64 PUFFIN_DV_MAX_MATERIALIZED_POSITIONS = 100'000'000; +constexpr Int64 DELETION_VECTOR_MAX_POSITION = 0x7FFFFFFE80000000LL; +constexpr Int32 DELETION_VECTOR_MAX_KEY = std::numeric_limits::max() - 1; + +UInt32 readBigEndianUInt32(const UInt8 * data) +{ + return (static_cast(data[0]) << 24) + | (static_cast(data[1]) << 16) + | (static_cast(data[2]) << 8) + | static_cast(data[3]); +} + +UInt64 positionFromKeyAndSubPosition(UInt32 key, UInt32 sub_position) +{ + return (static_cast(key) << 32) | static_cast(sub_position); +} + +roaring::Roaring readRoaringPortableSafe(const char * data, size_t size, Int32 key) +{ + try + { + return roaring::Roaring::readSafe(data, size); + } + catch (const std::exception & e) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Failed to deserialize deletion vector roaring bitmap at key {}: {}", key, e.what()); + } +} + +std::vector deserializeRoaringPositionBitmap(std::string_view bytes, std::optional expected_cardinality) +{ + if (bytes.size() < sizeof(Int64)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap is too small"); + + const char * ptr = bytes.data(); + size_t remaining = bytes.size(); + + /// Iceberg deletion-vector roaring layout stores count and keys as little-endian. + const Int64 bitmap_count = unalignedLoadLittleEndian(ptr); + ptr += sizeof(Int64); + remaining -= sizeof(Int64); + + if (bitmap_count < 0 || bitmap_count > std::numeric_limits::max()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector bitmap count: {}", bitmap_count); + + std::vector positions; + Int32 last_key = -1; + Int32 remaining_count = static_cast(bitmap_count); + UInt64 running_cardinality = 0; + + while (remaining_count > 0) + { + if (remaining < sizeof(Int32)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap is truncated while reading key"); + + const Int32 key = unalignedLoadLittleEndian(ptr); + ptr += sizeof(Int32); + remaining -= sizeof(Int32); + + if (key < 0 || key > DELETION_VECTOR_MAX_KEY) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector bitmap key: {}", key); + if (key <= last_key) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap keys must be sorted in ascending order"); + + auto bitmap = readRoaringPortableSafe(ptr, remaining, key); + + const size_t bitmap_size = bitmap.getSizeInBytes(/*portable=*/true); + if (bitmap_size > remaining) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector roaring bitmap at key {} exceeds blob size", key); + + if (expected_cardinality.has_value()) + { + const UInt64 bitmap_cardinality = bitmap.cardinality(); + UInt64 new_running_cardinality = 0; + if (common::addOverflow(running_cardinality, bitmap_cardinality, new_running_cardinality)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality exceeds declared cardinality {}", + *expected_cardinality); + + if (new_running_cardinality > *expected_cardinality) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds declared cardinality {}", + new_running_cardinality, + *expected_cardinality); + + running_cardinality = new_running_cardinality; + } + + for (UInt32 sub_position : bitmap) + { + const UInt64 position = positionFromKeyAndSubPosition(static_cast(key), sub_position); + if (position > static_cast(DELETION_VECTOR_MAX_POSITION)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector position {} is out of supported range", position); + positions.push_back(position); + } + + ptr += bitmap_size; + remaining -= bitmap_size; + last_key = key; + --remaining_count; + } + + if (remaining != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap has {} trailing bytes", remaining); + + if (expected_cardinality.has_value() && running_cardinality != *expected_cardinality) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} does not match deserialized row count {}", + *expected_cardinality, + running_cardinality); + + return positions; +} + +std::string_view extractDeletionVectorPayload(std::string_view blob) +{ + if (blob.size() < 12) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob is too small"); + + const auto * blob_bytes = reinterpret_cast(blob.data()); + const UInt32 combined_length = readBigEndianUInt32(blob_bytes); + if (combined_length < sizeof(DELETION_VECTOR_MAGIC)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + const size_t vector_size = combined_length - sizeof(DELETION_VECTOR_MAGIC); + const size_t expected_blob_size = sizeof(UInt32) + combined_length + sizeof(UInt32); + if (blob.size() != expected_blob_size) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob size {} does not match combined length {}", blob.size(), combined_length); + + if (std::memcmp(blob_bytes + sizeof(UInt32), DELETION_VECTOR_MAGIC, sizeof(DELETION_VECTOR_MAGIC)) != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector magic"); + + const UInt8 * crc_input = blob_bytes + sizeof(UInt32); + const size_t crc_input_size = combined_length; + const UInt32 expected_crc = readBigEndianUInt32(blob_bytes + sizeof(UInt32) + combined_length); + const UInt32 actual_crc = static_cast(crc32_z(0L, reinterpret_cast(crc_input), crc_input_size)); + if (expected_crc != actual_crc) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector CRC mismatch"); + + return std::string_view(blob.data() + 2 * sizeof(UInt32), vector_size); +} + +} + +void validatePuffinBlobBounds(Int64 offset, Int64 length, size_t file_size, std::string_view context) +{ + if (offset < 0 || length < 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "{}: offset/length out of bounds", context); + + if (offset > static_cast(file_size) || length > static_cast(file_size)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "{}: offset/length out of bounds", context); + + Int64 end = 0; + if (common::addOverflow(offset, length, end)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "{}: offset/length out of bounds", context); + + if (static_cast(end) > file_size) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "{}: offset/length out of bounds", context); +} + +std::vector deserializeDeletionVectorV1Blob(std::string_view blob_bytes, std::optional expected_cardinality) +{ + if (expected_cardinality.has_value() && *expected_cardinality > PUFFIN_DV_MAX_MATERIALIZED_POSITIONS) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds materialization limit {}", + *expected_cardinality, + PUFFIN_DV_MAX_MATERIALIZED_POSITIONS); + + const std::string_view vector_bytes = extractDeletionVectorPayload(blob_bytes); + return deserializeRoaringPositionBitmap(vector_bytes, expected_cardinality); +} + +std::vector readDeletionVectorFromPuffin(ReadBuffer & file, Int64 offset, Int64 length, std::optional expected_cardinality) +{ + ScopedPuffinFileReadProfileEvent profile_event; + + if (length < 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob length is negative"); + + if (static_cast(length) > PUFFIN_DV_MAX_BLOB_SIZE) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector blob length {} exceeds absolute limit {}", + length, + PUFFIN_DV_MAX_BLOB_SIZE); + + /// Reject before envelope peek / full allocate: cardinality alone is enough to fail closed. + if (expected_cardinality.has_value() && *expected_cardinality > PUFFIN_DV_MAX_MATERIALIZED_POSITIONS) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds materialization limit {}", + *expected_cardinality, + PUFFIN_DV_MAX_MATERIALIZED_POSITIONS); + + if (auto file_size = tryGetFileSizeFromReadBuffer(file)) + validatePuffinBlobBounds(offset, length, *file_size); + else if (offset < 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid Puffin deletion vector offset {} or length {}", offset, length); + + if (static_cast(length) < 12) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob is too small"); + + auto * seekable = dynamic_cast(&file); + if (!seekable) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin deletion vector read requires a seekable buffer"); + + /// Peek combined_length + magic before allocating `length` (up to 2 GiB). Matches + /// `readDeletionVectorBlobBytes` in the SQL Puffin format path. + seekable->seek(offset, SEEK_SET); + + UInt8 header[8]; + file.readStrict(reinterpret_cast(header), sizeof(header)); + + const UInt32 combined_length = readBigEndianUInt32(header); + if (std::memcmp(header + sizeof(UInt32), DELETION_VECTOR_MAGIC, sizeof(DELETION_VECTOR_MAGIC)) != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector magic"); + + if (combined_length < sizeof(DELETION_VECTOR_MAGIC)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + UInt64 expected_blob_size = 0; + if (common::addOverflow(static_cast(combined_length), UInt64{8}, expected_blob_size)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + if (static_cast(length) != expected_blob_size) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector blob size {} does not match combined length {}", + length, + combined_length); + + String blob_data(static_cast(length), '\0'); + std::memcpy(blob_data.data(), header, sizeof(header)); + file.readStrict(blob_data.data() + sizeof(header), blob_data.size() - sizeof(header)); + + return deserializeDeletionVectorV1Blob(blob_data, expected_cardinality); +} + +} diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h new file mode 100644 index 000000000000..3cc1bfbd797a --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h @@ -0,0 +1,22 @@ +#pragma once + +#include +#include + +#include +#include +#include + +namespace DB +{ + +/// Validate that [offset, offset + length) fits within file_size. +void validatePuffinBlobBounds(Int64 offset, Int64 length, size_t file_size, std::string_view context = "Puffin deletion vector"); + +/// Deserialize a deletion-vector-v1 blob (magic + CRC wrapper + roaring bitmap payload). +std::vector deserializeDeletionVectorV1Blob(std::string_view blob_bytes, std::optional expected_cardinality = std::nullopt); + +/// Read a deletion-vector-v1 blob from a Puffin file at the given offset and length. +std::vector readDeletionVectorFromPuffin(ReadBuffer & file, Int64 offset, Int64 length, std::optional expected_cardinality = std::nullopt); + +} diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.cpp b/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.cpp new file mode 100644 index 000000000000..93bf606df368 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.cpp @@ -0,0 +1,110 @@ +#include + +#include +#include + +namespace CurrentMetrics +{ +extern const Metric PuffinFilesCacheBytes; +extern const Metric PuffinFilesCacheFiles; +} + +namespace ProfileEvents +{ +extern const Event PuffinFilesCacheWeightLost; +} + +namespace DB +{ + +DataLakeObjectMetadata::ExcludedRowsPtr PuffinFilesCache::cloneExcludedRows(const PuffinFilesCacheCell & cell) +{ + if (cell.is_empty_deletion_vector) + return nullptr; + + auto cloned = std::make_shared(); + cloned->merge(*cell.excluded_rows); + return cloned; +} + +bool PuffinFilesCacheKey::operator==(const PuffinFilesCacheKey & other) const +{ + return file_path == other.file_path + && etag == other.etag + && content_offset == other.content_offset + && content_size_in_bytes == other.content_size_in_bytes + && referenced_data_file == other.referenced_data_file; +} + +size_t PuffinFilesCacheKeyHash::operator()(const PuffinFilesCacheKey & key) const +{ + size_t hash = 0; + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.file_path.data(), key.file_path.size())); + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.etag.data(), key.etag.size())); + boost::hash_combine(hash, key.content_offset); + boost::hash_combine(hash, key.content_size_in_bytes); + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.referenced_data_file.data(), key.referenced_data_file.size())); + return hash; +} + +UInt64 PuffinFilesCacheCell::calculateMemorySize(bool is_empty_deletion_vector_, const DataLakeObjectMetadata::ExcludedRowsPtr & excluded_rows_) +{ + if (is_empty_deletion_vector_) + return EMPTY_DELETION_VECTOR_WEIGHT; + + if (!excluded_rows_) + return 0; + + return excluded_rows_->getAllocatedBytes(); +} + +PuffinFilesCacheCell::PuffinFilesCacheCell(DataLakeObjectMetadata::ExcludedRowsPtr excluded_rows_) + : excluded_rows(std::move(excluded_rows_)) + , is_empty_deletion_vector(!excluded_rows) + , memory_bytes( + calculateMemorySize(is_empty_deletion_vector, excluded_rows) + + (is_empty_deletion_vector ? 0 : SIZE_IN_MEMORY_OVERHEAD)) +{ +} + +size_t PuffinFilesCacheWeightFunction::operator()(const PuffinFilesCacheCell & cell) const +{ + return cell.memory_bytes; +} + +PuffinFilesCache::PuffinFilesCache( + const String & cache_policy, + size_t max_size_in_bytes, + size_t max_count, + double size_ratio) + : Base( + cache_policy, + CurrentMetrics::PuffinFilesCacheBytes, + CurrentMetrics::PuffinFilesCacheFiles, + max_size_in_bytes, + max_count, + size_ratio) + , log(getLogger("PuffinFilesCache")) +{ +} + +std::optional PuffinFilesCache::tryCreateKey( + const String & file_path, + const String & etag, + Int64 content_offset, + Int64 content_size_in_bytes, + const String & referenced_data_file) +{ + if (etag.empty()) + return std::nullopt; + + return PuffinFilesCacheKey{file_path, etag, content_offset, content_size_in_bytes, referenced_data_file}; +} + +void PuffinFilesCache::onEntryRemoval(const size_t weight_loss, const MappedPtr &) +{ + LOG_TRACE(log, "Puffin files cache eviction"); + ProfileEvents::increment(ProfileEvents::PuffinFilesCacheWeightLost, weight_loss); +} + +} diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.h b/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.h new file mode 100644 index 000000000000..1dd12205994d --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.h @@ -0,0 +1,138 @@ +#pragma once + +#include +#include + +#include +#include +#include +#include +#include +#include + +#include + +namespace ProfileEvents +{ +extern const Event PuffinFilesCacheHits; +extern const Event PuffinFilesCacheMisses; +extern const Event PuffinFilesCacheWeightLost; +} + +namespace DB +{ + +struct PuffinFilesCacheKey +{ + String file_path; + String etag; + Int64 content_offset = 0; + Int64 content_size_in_bytes = 0; + String referenced_data_file; + + bool operator==(const PuffinFilesCacheKey & other) const; +}; + +struct PuffinFilesCacheKeyHash +{ + size_t operator()(const PuffinFilesCacheKey & key) const; +}; + +struct PuffinFilesCacheCell : private boost::noncopyable +{ + DataLakeObjectMetadata::ExcludedRowsPtr excluded_rows; + bool is_empty_deletion_vector = false; + UInt64 memory_bytes = 0; + + explicit PuffinFilesCacheCell(DataLakeObjectMetadata::ExcludedRowsPtr excluded_rows_); + +private: + static constexpr UInt64 EMPTY_DELETION_VECTOR_WEIGHT = 1; + static constexpr size_t SIZE_IN_MEMORY_OVERHEAD = 200; + + static UInt64 calculateMemorySize(bool is_empty_deletion_vector_, const DataLakeObjectMetadata::ExcludedRowsPtr & excluded_rows_); +}; + +struct PuffinFilesCacheWeightFunction +{ + size_t operator()(const PuffinFilesCacheCell & cell) const; +}; + +/// Cache for parsed content loaded from Puffin files (deletion vectors today, indexes later). +class PuffinFilesCache : public CacheBase +{ +public: + using Base = CacheBase; + + PuffinFilesCache(const String & cache_policy, size_t max_size_in_bytes, size_t max_count, double size_ratio); + + static std::optional tryCreateKey( + const String & file_path, + const String & etag, + Int64 content_offset, + Int64 content_size_in_bytes, + const String & referenced_data_file); + + template + DataLakeObjectMetadata::ExcludedRowsPtr getOrSetDeletionVector(const PuffinFilesCacheKey & key, LoadFunc && load_fn) + { + auto load_fn_wrapper = [&]() + { + auto excluded_rows = load_fn(); + const bool is_empty_deletion_vector = !excluded_rows; + if (is_empty_deletion_vector) + { + LOG_TRACE( + log, + "Cached empty puffin deletion vector for {} | {} at offset {} length {} for data file {}", + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + } + else + { + LOG_TRACE( + log, + "Loaded puffin deletion vector into cache for {} | {} at offset {} length {} for data file {}", + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + } + return std::make_shared(std::move(excluded_rows)); + }; + + auto result = Base::getOrSet(key, load_fn_wrapper); + if (result.second) + { + LOG_TRACE(log, "Puffin files cache miss for {} | {} at offset {} length {} for data file {}", key.file_path, key.etag, key.content_offset, key.content_size_in_bytes, key.referenced_data_file); + ProfileEvents::increment(ProfileEvents::PuffinFilesCacheMisses); + } + else if (result.first->is_empty_deletion_vector) + { + LOG_TRACE(log, "Puffin files cache hit (empty deletion vector) for {} | {} at offset {} length {} for data file {}", key.file_path, key.etag, key.content_offset, key.content_size_in_bytes, key.referenced_data_file); + ProfileEvents::increment(ProfileEvents::PuffinFilesCacheHits); + } + else + { + LOG_TRACE(log, "Puffin files cache hit for {} | {} at offset {} length {} for data file {}", key.file_path, key.etag, key.content_offset, key.content_size_in_bytes, key.referenced_data_file); + ProfileEvents::increment(ProfileEvents::PuffinFilesCacheHits); + } + + return cloneExcludedRows(*result.first); + } + +private: + static DataLakeObjectMetadata::ExcludedRowsPtr cloneExcludedRows(const PuffinFilesCacheCell & cell); + + LoggerPtr log; + + void onEntryRemoval(size_t weight_loss, const MappedPtr &) override; +}; + +using PuffinFilesCachePtr = std::shared_ptr; + +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_before_equality_filter.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_before_equality_filter.cpp new file mode 100644 index 000000000000..1d17d21d7d2b --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_before_equality_filter.cpp @@ -0,0 +1,68 @@ +#include + +#include +#include +#include +#include + +using namespace DB; + +namespace +{ + +Chunk makeChunkWithFileRowNumbers(const std::vector & values, size_t row_num_offset = 0) +{ + auto column = ColumnUInt64::create(); + for (UInt64 value : values) + column->insert(value); + + Chunk chunk(Columns{std::move(column)}, values.size()); + chunk.getChunkInfos().add(std::make_shared(row_num_offset)); + return chunk; +} + +/// Mimics Iceberg equality-delete `FilterTransform`: shrink columns, leave `applied_filter` unset. +void shrinkWithoutAppliedFilter(Chunk & chunk, const IColumn::Filter & filter) +{ + size_t result_size = 0; + for (UInt8 keep : filter) + result_size += keep != 0; + + auto columns = chunk.detachColumns(); + for (auto & column : columns) + column = column->filter(filter, -1); + chunk.setColumns(std::move(columns), result_size); +} + +std::vector readValues(const Chunk & chunk) +{ + const auto & column = assert_cast(*chunk.getColumns().at(0)); + std::vector values; + values.reserve(column.size()); + for (size_t i = 0; i < column.size(); ++i) + values.push_back(column.getData()[i]); + return values; +} + +std::shared_ptr makeExcludedRows(std::initializer_list positions) +{ + auto excluded_rows = std::make_shared(); + for (UInt64 position : positions) + excluded_rows->add(position); + return excluded_rows; +} + +} + +/// Equality removes file row 1; DV deletes file position 2. Survivors must be {0, 3}. +TEST(DeletionVectorBeforeEqualityFilter, CorrectOrderKeepsFileRowMapping) +{ + Chunk chunk = makeChunkWithFileRowNumbers({0, 1, 2, 3}); + + DeletionVectorTransform::transform(chunk, *makeExcludedRows({2})); + ASSERT_EQ(readValues(chunk), (std::vector{0, 1, 3})); + + /// Drop equality-deleted value 1 without updating applied_filter (as FilterTransform does). + shrinkWithoutAppliedFilter(chunk, IColumn::Filter{1, 0, 1}); + EXPECT_EQ(readValues(chunk), (std::vector{0, 3})); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_has_non_empty_excluded_rows.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_has_non_empty_excluded_rows.cpp new file mode 100644 index 000000000000..9903493c3413 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_has_non_empty_excluded_rows.cpp @@ -0,0 +1,28 @@ +#include + +#include +#include + +using namespace DB; + +TEST(HasNonEmptyExcludedRows, EmptyOptionalIsFalse) +{ + EXPECT_FALSE(hasNonEmptyExcludedRows(std::nullopt)); +} + +TEST(HasNonEmptyExcludedRows, MissingOrEmptyBitmapIsFalse) +{ + DataLakeObjectMetadata metadata; + EXPECT_FALSE(hasNonEmptyExcludedRows(metadata)); + + metadata.excluded_rows = std::make_shared(); + EXPECT_FALSE(hasNonEmptyExcludedRows(metadata)); +} + +TEST(HasNonEmptyExcludedRows, NonEmptyBitmapIsTrue) +{ + DataLakeObjectMetadata metadata; + metadata.excluded_rows = std::make_shared(); + metadata.excluded_rows->add(7); + EXPECT_TRUE(hasNonEmptyExcludedRows(metadata)); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_bounds.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_bounds.cpp new file mode 100644 index 000000000000..0c036309b73b --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_bounds.cpp @@ -0,0 +1,63 @@ +#include + +#include +#include +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int BAD_ARGUMENTS; +} +} + +TEST(PuffinDeletionVectorBounds, RejectsLengthExceedingFileSize) +{ + const String data(64, '\0'); + ReadBufferFromOutsideMemoryFile file("test.puffin", data); + + try + { + readDeletionVectorFromPuffin(file, 0, 1000); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("offset/length out of bounds"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorBounds, RejectsOffsetPlusLengthOverflow) +{ + const String data(64, '\0'); + ReadBufferFromOutsideMemoryFile file("test.puffin", data); + + try + { + readDeletionVectorFromPuffin(file, 60, 12); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("offset/length out of bounds"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorBounds, ValidatePuffinBlobBoundsRejectsNegativeOffset) +{ + try + { + validatePuffinBlobBounds(-1, 10, 64); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("offset/length out of bounds"), std::string::npos); + } +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_cardinality.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_cardinality.cpp new file mode 100644 index 000000000000..9bae4c4511f5 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_cardinality.cpp @@ -0,0 +1,96 @@ +#include + +#include +#include +#include + +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int BAD_ARGUMENTS; +} +} + +namespace +{ + +/// deletion-vector-v1 blob for positions {2, 5} (cardinality 2). +constexpr UInt8 two_position_dv_blob[] = { + 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, + 0x10, 0x00, 0x00, 0x00, 0x02, 0x00, 0x05, 0x00, 0x2C, 0xDB, 0x9F, 0xC1, +}; + +constexpr UInt64 PUFFIN_DV_MAX_MATERIALIZED_POSITIONS = 100'000'000; +constexpr Int64 large_declared_length = 64 * 1024 * 1024; + +} + +TEST(PuffinDeletionVectorCardinality, RejectsCardinalityAboveMaterializationLimitBeforeParse) +{ + /// Ceiling is checked before payload validation, so even an empty blob must fail closed. + try + { + deserializeDeletionVectorV1Blob(std::string_view{}, PUFFIN_DV_MAX_MATERIALIZED_POSITIONS + 1); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("exceeds materialization limit"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorCardinality, RejectsCardinalityAboveLimitBeforeFullAllocate) +{ + /// Huge declared length with only a tiny buffer: without an early ceiling check this would + /// allocate `large_declared_length` (or fail mid-read after that allocate). ReadBufferFromMemory + /// does not expose file size, so bounds checks alone do not stop this. + const char header[8] = {}; + ReadBufferFromMemory file(header, sizeof(header)); + + try + { + readDeletionVectorFromPuffin( + file, 0, large_declared_length, PUFFIN_DV_MAX_MATERIALIZED_POSITIONS + 1); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("exceeds materialization limit"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorCardinality, RejectsBitmapExceedingDeclaredCardinality) +{ + const std::string_view blob( + reinterpret_cast(two_position_dv_blob), sizeof(two_position_dv_blob)); + + try + { + deserializeDeletionVectorV1Blob(blob, /*expected_cardinality=*/1); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("exceeds declared cardinality"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorCardinality, AcceptsMatchingCardinality) +{ + const std::string_view blob( + reinterpret_cast(two_position_dv_blob), sizeof(two_position_dv_blob)); + + const auto positions = deserializeDeletionVectorV1Blob(blob, /*expected_cardinality=*/2); + ASSERT_EQ(positions.size(), 2u); + EXPECT_EQ(positions[0], 2u); + EXPECT_EQ(positions[1], 5u); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_envelope.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_envelope.cpp new file mode 100644 index 000000000000..89c0f97fce7f --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_envelope.cpp @@ -0,0 +1,98 @@ +#include + +#include +#include +#include + +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int BAD_ARGUMENTS; +} +} + +namespace +{ + +/// deletion-vector-v1 blob for positions {2, 5} (cardinality 2). +constexpr UInt8 two_position_dv_blob[] = { + 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, + 0x10, 0x00, 0x00, 0x00, 0x02, 0x00, 0x05, 0x00, 0x2C, 0xDB, 0x9F, 0xC1, +}; + +/// Large declared length that would force a huge allocate if peeked after full read. +constexpr Int64 large_declared_length = 64 * 1024 * 1024; + +} + +TEST(PuffinDeletionVectorEnvelope, RejectsLengthBelowEnvelopeMinimum) +{ + const char zeros[16] = {}; + ReadBufferFromMemory file(zeros, sizeof(zeros)); + + try + { + readDeletionVectorFromPuffin(file, 0, 11); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("too small"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorEnvelope, RejectsInvalidMagicBeforeFullAllocate) +{ + /// Only 8 bytes available; declared length is huge. Without envelope peek this would allocate + /// `large_declared_length` (or fail mid-read after that allocate). ReadBufferFromMemory does not + /// expose file size, so the absolute 2 GiB / bounds checks alone do not stop this. + const UInt8 header[8] = {0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x00}; // wrong magic + ReadBufferFromMemory file(header, sizeof(header)); + + try + { + readDeletionVectorFromPuffin(file, 0, large_declared_length); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("Invalid deletion vector magic"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorEnvelope, RejectsCombinedLengthMismatchBeforeFullAllocate) +{ + /// Valid magic, but combined_length implies blob size 0x24+8=44, while caller length is huge. + const UInt8 header[8] = {0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64}; + ReadBufferFromMemory file(header, sizeof(header)); + + try + { + readDeletionVectorFromPuffin(file, 0, large_declared_length); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("does not match combined length"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorEnvelope, ReadsValidBlobAfterEnvelopePeek) +{ + ReadBufferFromMemory file(two_position_dv_blob, sizeof(two_position_dv_blob)); + const auto positions = readDeletionVectorFromPuffin( + file, 0, static_cast(sizeof(two_position_dv_blob)), /*expected_cardinality=*/2); + + ASSERT_EQ(positions.size(), 2u); + EXPECT_EQ(positions[0], 2u); + EXPECT_EQ(positions[1], 5u); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_clone.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_clone.cpp new file mode 100644 index 000000000000..ff0193cb9206 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_clone.cpp @@ -0,0 +1,87 @@ +#include + +#include +#include +#include + +using namespace DB; + +namespace CurrentMetrics +{ +extern const Metric PuffinFilesCacheBytes; +extern const Metric PuffinFilesCacheFiles; +} + +namespace +{ + +DataLakeObjectMetadata::ExcludedRowsPtr makeExcludedRows(const std::vector & positions) +{ + auto excluded_rows = std::make_shared(); + for (size_t position : positions) + excluded_rows->add(position); + return excluded_rows; +} + +} + +TEST(PuffinFilesCacheClone, CacheHitReturnsIndependentCopy) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key = PuffinFilesCache::tryCreateKey("puffin.bin", "etag-1", 100, 200, "data/file-a.parquet"); + ASSERT_TRUE(key.has_value()); + + size_t load_calls = 0; + auto load_fn = [&]() + { + ++load_calls; + return makeExcludedRows({1, 5, 10}); + }; + + auto first = cache.getOrSetDeletionVector(*key, load_fn); + auto second = cache.getOrSetDeletionVector(*key, load_fn); + + ASSERT_EQ(load_calls, 1); + ASSERT_NE(first, second); + EXPECT_TRUE(first->rb_contains(1)); + EXPECT_TRUE(first->rb_contains(5)); + EXPECT_TRUE(first->rb_contains(10)); + EXPECT_FALSE(first->rb_contains(99)); + + first->add(99); + + EXPECT_FALSE(second->rb_contains(99)); + + auto third = cache.getOrSetDeletionVector(*key, load_fn); + ASSERT_NE(third, first); + EXPECT_FALSE(third->rb_contains(99)); + EXPECT_TRUE(third->rb_contains(10)); +} + +TEST(PuffinFilesCacheClone, EmptyExcludedRowsReturnsNullptr) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key = PuffinFilesCache::tryCreateKey("puffin.bin", "etag-1", 100, 200, "data/file-a.parquet"); + ASSERT_TRUE(key.has_value()); + + const auto files_before = CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheFiles); + const auto bytes_before = CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheBytes); + + size_t load_calls = 0; + auto load_fn = [&]() + { + ++load_calls; + return DataLakeObjectMetadata::ExcludedRowsPtr{}; + }; + + auto first = cache.getOrSetDeletionVector(*key, load_fn); + auto second = cache.getOrSetDeletionVector(*key, load_fn); + + EXPECT_EQ(first, nullptr); + EXPECT_EQ(second, nullptr); + EXPECT_EQ(load_calls, 1); + EXPECT_EQ(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheFiles), files_before + 1); + EXPECT_EQ(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheBytes), bytes_before + 1); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_key.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_key.cpp new file mode 100644 index 000000000000..0252c8b61d75 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_key.cpp @@ -0,0 +1,33 @@ +#include + +#include + +using namespace DB; + +TEST(PuffinFilesCacheKey, SamePuffinSliceDifferentReferencedDataFile) +{ + const auto key1 = PuffinFilesCache::tryCreateKey("puffin.bin", "etag-1", 100, 200, "data/file-a.parquet"); + const auto key2 = PuffinFilesCache::tryCreateKey("puffin.bin", "etag-1", 100, 200, "data/file-b.parquet"); + + ASSERT_TRUE(key1.has_value()); + ASSERT_TRUE(key2.has_value()); + EXPECT_NE(*key1, *key2); + EXPECT_NE(PuffinFilesCacheKeyHash{}(*key1), PuffinFilesCacheKeyHash{}(*key2)); +} + +TEST(PuffinFilesCacheKey, SameReferencedDataFileProducesEqualKeys) +{ + const auto key1 = PuffinFilesCache::tryCreateKey("puffin.bin", "etag-1", 100, 200, "data/file-a.parquet"); + const auto key2 = PuffinFilesCache::tryCreateKey("puffin.bin", "etag-1", 100, 200, "data/file-a.parquet"); + + ASSERT_TRUE(key1.has_value()); + ASSERT_TRUE(key2.has_value()); + EXPECT_EQ(*key1, *key2); + EXPECT_EQ(PuffinFilesCacheKeyHash{}(*key1), PuffinFilesCacheKeyHash{}(*key2)); +} + +TEST(PuffinFilesCacheKey, EmptyEtagReturnsNullopt) +{ + const auto key = PuffinFilesCache::tryCreateKey("puffin.bin", "", 100, 200, "data/file-a.parquet"); + EXPECT_FALSE(key.has_value()); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_weight.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_weight.cpp new file mode 100644 index 000000000000..f1d307b90f9c --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_weight.cpp @@ -0,0 +1,46 @@ +#include + +#include +#include +#include +#include + +using namespace DB; + +namespace CurrentMetrics +{ +extern const Metric PuffinFilesCacheBytes; +} + +namespace +{ + +DataLakeObjectMetadata::ExcludedRowsPtr makeLargeSparseExcludedRows() +{ + auto excluded_rows = std::make_shared(); + for (size_t i = 0; i < 33; ++i) + excluded_rows->add((static_cast(i) + 1) << 32); + return excluded_rows; +} + +} + +TEST(RoaringBitmapWithSmallSetMemory, LargeSparseBitmapAllocatedBytesExceedCardinalityEstimate) +{ + const auto excluded_rows = makeLargeSparseExcludedRows(); + + ASSERT_TRUE(excluded_rows->isLarge()); + EXPECT_GT(excluded_rows->getAllocatedBytes(), excluded_rows->size() * sizeof(size_t)); +} + +TEST(PuffinFilesCacheWeight, UsesRoaringAllocatedBytesForWeight) +{ + const auto excluded_rows = makeLargeSparseExcludedRows(); + const auto key = PuffinFilesCache::tryCreateKey("puffin.bin", "etag-1", 100, 200, "data/file-a.parquet"); + ASSERT_TRUE(key.has_value()); + + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + cache.getOrSetDeletionVector(*key, [&]() { return excluded_rows; }); + + EXPECT_GE(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheBytes), excluded_rows->getAllocatedBytes()); +} diff --git a/src/Storages/ObjectStorage/IObjectIterator.cpp b/src/Storages/ObjectStorage/IObjectIterator.cpp index 7ae3d1cbb1b2..b03ca71b325c 100644 --- a/src/Storages/ObjectStorage/IObjectIterator.cpp +++ b/src/Storages/ObjectStorage/IObjectIterator.cpp @@ -170,19 +170,19 @@ ObjectInfoPtr ObjectIteratorSplitByBuckets::next(size_t id) auto file_bucket_infos = splitter->splitToBuckets(bucket_size, *buffer, format_settings); for (const auto & file_bucket : file_bucket_infos) { - auto copy_object_info = *last_object_info; + auto copy_object_info = last_object_info->clone(); if (has_cache_entry) { auto filtered = file_bucket->filterByMatchingRowGroups(matching_row_groups); if (!filtered) continue; - copy_object_info.file_bucket_info = std::move(filtered); + copy_object_info->file_bucket_info = std::move(filtered); } else { - copy_object_info.file_bucket_info = file_bucket; + copy_object_info->file_bucket_info = file_bucket; } - pending_objects_info.push(std::make_shared(copy_object_info)); + pending_objects_info.push(std::move(copy_object_info)); } } } diff --git a/src/Storages/ObjectStorage/IObjectIterator.h b/src/Storages/ObjectStorage/IObjectIterator.h index 47febf269ee4..c922f0a70243 100644 --- a/src/Storages/ObjectStorage/IObjectIterator.h +++ b/src/Storages/ObjectStorage/IObjectIterator.h @@ -53,6 +53,10 @@ struct ObjectInfo FileBucketInfoPtr file_bucket_info; + /// Polymorphic copy. Used when splitting files into buckets so derived metadata + /// (e.g. Iceberg equality / position deletes) is not sliced away. + virtual ObjectInfoPtr clone() const { return std::make_shared(*this); } + String getIdentifier() const; }; diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp index 8521f6c309af..c10ff9a4ebe6 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp @@ -122,6 +122,19 @@ static void logIcebergFileStats(const ObjectInfoPtr & object_info, const LoggerP #endif } +/// Count-from-files cache key is data-file identity only. Skip when row filtering can change +/// independently (DVs / selection vectors, Iceberg eq/pos deletes) or when the task is a bucket subset. +static bool canUseCountFromFilesCache(const ObjectInfoPtr & object_info) +{ + if (hasNonEmptyExcludedRows(object_info->data_lake_metadata) || object_info->file_bucket_info) + return false; +#if USE_AVRO + if (hasIcebergEqualityDeletes(object_info) || hasIcebergPositionDeletes(object_info)) + return false; +#endif + return true; +} + StorageObjectStorageSource::StorageObjectStorageSource( const StorageID & storage_id_, String name_, @@ -585,7 +598,8 @@ Chunk StorageObjectStorageSource::generate() } if (reader.getInputFormat() && read_context->getSettingsRef()[Setting::use_cache_for_count_from_files] - && !format_filter_info->filter_actions_dag) + && !format_filter_info->filter_actions_dag + && canUseCountFromFilesCache(reader.getObjectInfo())) addNumRowsToCache(*reader.getObjectInfo(), total_rows_in_file); total_rows_in_file = 0; @@ -747,8 +761,24 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade return schema_cache->tryGetNumRows(cache_key, get_last_mod_time); }; - std::optional num_rows_from_cache - = need_only_count && context_->getSettingsRef()[Setting::use_cache_for_count_from_files] ? try_get_num_rows_from_cache() : std::nullopt; + /// The count-from-cache shortcut builds a `ConstChunkGenerator` without opening the read buffer, so a + /// requested `_headers` virtual column (the HTTP response headers of the data `GET`) would have to fall + /// back to the metadata-probe headers (usually a `HEAD`), which can differ from the actual `GET` + /// response. Skip the shortcut when `_headers` is requested so the real `GET` headers are used. + const bool headers_requested = read_from_format_info.requested_virtual_columns.contains("_headers"); + +#if USE_AVRO + /// Equality deletes filter by column values; need_only_count emits default-filled chunks. + const bool effective_need_only_count = need_only_count && !hasIcebergEqualityDeletes(object_info); +#else + const bool effective_need_only_count = need_only_count; +#endif + + const bool can_use_count_cache = effective_need_only_count && !headers_requested + && context_->getSettingsRef()[Setting::use_cache_for_count_from_files] + && canUseCountFromFilesCache(object_info); + + std::optional num_rows_from_cache = can_use_count_cache ? try_get_num_rows_from_cache() : std::nullopt; if (num_rows_from_cache) { @@ -919,7 +949,7 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade filter_info, true /* is_remote_fs */, compression_method, - need_only_count, + effective_need_only_count, std::nullopt /*min_block_size_bytes*/, std::nullopt /*min_block_size_rows*/, std::nullopt /*max_block_size_bytes*/); @@ -937,22 +967,22 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade filter_info, true /* is_remote_fs */, compression_method, - need_only_count); + effective_need_only_count); } input_format->setBucketsToRead(object_info->file_bucket_info); input_format->setSerializationHints(read_from_format_info.serialization_hints); - if (need_only_count) + if (effective_need_only_count) input_format->needOnlyCount(); builder.init(Pipe(input_format)); - configuration->addDeleteTransformers(object_info, builder, format_settings, parser_shared_resources, context_); - - if (object_info->data_lake_metadata - && object_info->data_lake_metadata->excluded_rows - && object_info->data_lake_metadata->excluded_rows->size() > 0) + /// Deletion vectors (and selection vectors) address absolute file row numbers via + /// `ChunkInfoRowNumbers`. Iceberg equality deletes use a plain `FilterTransform` that + /// shrinks the chunk without maintaining `applied_filter`, so DV must run first — + /// otherwise later DV filtering maps dense post-equality indices to the wrong file rows. + if (hasNonEmptyExcludedRows(object_info->data_lake_metadata)) { builder.addSimpleTransform([&](const SharedHeader & header) { @@ -960,6 +990,8 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade }); } + configuration->addDeleteTransformers(object_info, builder, format_settings, parser_shared_resources, context_); + std::optional schema_transform; if (object_info->data_lake_metadata && object_info->data_lake_metadata->schema_transform) { diff --git a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp index b00c1d609fa1..fdd534a1278f 100644 --- a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp +++ b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp @@ -1,9 +1,26 @@ #include +#include +#include #include #include +#include +#include + +#if USE_PARQUET +#include +#include +#endif using namespace DB; +namespace DB +{ +namespace ErrorCodes +{ +extern const int UNKNOWN_PROTOCOL; +} +} + namespace { class TestIterator : public IObjectIterator @@ -231,3 +248,225 @@ TEST(RendezvousHashing, MultipleNodesReducedClusterOneByOne) ASSERT_TRUE(checkHead(paths0, {1, 5, 6, 7, 8, 9})); ASSERT_TRUE(checkHead(paths1, {0, 2, 3, 4})); } + +TEST(ClusterFunctionReadTaskResponse, RejectsNonEmptyExcludedRowsOnOldProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.excluded_rows = std::make_shared(); + response.data_lake_metadata.excluded_rows->add(7); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("excluded_rows"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, AllowsEmptyExcludedRowsOnOldProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.excluded_rows = std::make_shared(); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO); + out.finalize(); + EXPECT_FALSE(serialized.empty()); +} + +TEST(ClusterFunctionReadTaskResponse, RoundTripsExcludedRowsOnSupportedProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.excluded_rows = std::make_shared(); + response.data_lake_metadata.excluded_rows->add(3); + response.data_lake_metadata.excluded_rows->add(9); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_EXCLUDED_ROWS); + out.finalize(); + + ReadBufferFromString in(serialized); + ClusterFunctionReadTaskResponse deserialized; + deserialized.deserialize(in); + + ASSERT_TRUE(deserialized.data_lake_metadata.excluded_rows); + EXPECT_EQ(deserialized.data_lake_metadata.excluded_rows->size(), 2u); + EXPECT_TRUE(deserialized.data_lake_metadata.excluded_rows->rb_contains(3)); + EXPECT_TRUE(deserialized.data_lake_metadata.excluded_rows->rb_contains(9)); +} + +TEST(ClusterFunctionReadTaskResponse, RejectsEqualityDeletesOnProtocolBeforeIcebergMetadata) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.iceberg_info = Iceberg::IcebergObjectSerializableInfo{}; + response.iceberg_info->equality_deletes_objects.push_back( + Iceberg::EqualityDeleteObject{ + .file_path = "/path/eq.parquet", + .file_format = "PARQUET", + .equality_ids = std::vector{1}, + .schema_id = 0, + }); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("iceberg_info"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, RejectsPositionDeletesOnProtocolBeforeIcebergMetadata) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.iceberg_info = Iceberg::IcebergObjectSerializableInfo{}; + response.iceberg_info->position_deletes_objects.push_back( + Iceberg::PositionDeleteObject{ + .file_path = "/path/pos.parquet", + .file_format = "PARQUET", + .reference_data_file_path = std::nullopt, + .sequence_number = 1, + }); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("iceberg_info"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, AllowsIcebergInfoWithoutDeletesOnProtocolBeforeIcebergMetadata) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.iceberg_info = Iceberg::IcebergObjectSerializableInfo{}; + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA); + out.finalize(); + EXPECT_FALSE(serialized.empty()); +} + +TEST(ClusterFunctionReadTaskResponse, RoundTripsIcebergDeletesOnSupportedProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.iceberg_info = Iceberg::IcebergObjectSerializableInfo{}; + response.iceberg_info->data_object_file_path_key + = Iceberg::IcebergPathFromMetadata::deserialize("s3://bucket/path/file.parquet"); + response.iceberg_info->file_format = "PARQUET"; + response.iceberg_info->equality_deletes_objects.push_back( + Iceberg::EqualityDeleteObject{ + .file_path = "/path/eq.parquet", + .file_format = "PARQUET", + .equality_ids = std::vector{1, 2}, + .schema_id = 7, + }); + response.iceberg_info->position_deletes_objects.push_back( + Iceberg::PositionDeleteObject{ + .file_path = "/path/pos.parquet", + .file_format = "PARQUET", + .reference_data_file_path = std::nullopt, + .sequence_number = 42, + }); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA); + out.finalize(); + + ReadBufferFromString in(serialized); + ClusterFunctionReadTaskResponse deserialized; + deserialized.deserialize(in); + + ASSERT_TRUE(deserialized.iceberg_info.has_value()); + ASSERT_EQ(deserialized.iceberg_info->equality_deletes_objects.size(), 1u); + EXPECT_EQ(deserialized.iceberg_info->equality_deletes_objects[0].file_path, "/path/eq.parquet"); + ASSERT_EQ(deserialized.iceberg_info->position_deletes_objects.size(), 1u); + EXPECT_EQ(deserialized.iceberg_info->position_deletes_objects[0].file_path, "/path/pos.parquet"); +} + +#if USE_PARQUET + +TEST(ClusterFunctionReadTaskResponse, RejectsFileBucketInfoOnProtocolBeforeFileBuckets) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.file_bucket_info = std::make_shared(std::vector{0, 1}); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("file_bucket_info"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, AllowsMissingFileBucketInfoOnProtocolBeforeFileBuckets) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA); + out.finalize(); + EXPECT_FALSE(serialized.empty()); +} + +TEST(ClusterFunctionReadTaskResponse, RoundTripsFileBucketInfoOnSupportedProtocol) +{ + tryRegisterFormats(); + + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.file_bucket_info = std::make_shared(std::vector{2, 5}); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO); + out.finalize(); + + ReadBufferFromString in(serialized); + ClusterFunctionReadTaskResponse deserialized; + deserialized.deserialize(in); + + ASSERT_TRUE(deserialized.file_bucket_info); + auto * parquet_buckets = dynamic_cast(deserialized.file_bucket_info.get()); + ASSERT_TRUE(parquet_buckets); + EXPECT_EQ(parquet_buckets->row_group_ids, (std::vector{2, 5})); +} + +#endif diff --git a/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py b/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py new file mode 100644 index 000000000000..127df6ab993d --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py @@ -0,0 +1,265 @@ +import uuid + +import pytest + +from helpers.iceberg_utils import ( + default_upload_directory, + get_uuid_str, + get_creation_expression, +) + + +def get_array(query_result: str): + return sorted([int(x) for x in query_result.strip().split("\n") if x]) + + +def expected_complex_ids(): + ids = list(range(20, 90)) + list(range(100, 150)) + ids += [x for x in range(200, 250) if x not in {205, 210, 220}] + return sorted(ids) + + +def upload_table(cluster, storage_type, table_name): + default_upload_directory( + cluster, + storage_type, + f"/iceberg_data/default/{table_name}/", + f"/iceberg_data/default/{table_name}/", + ) + + +@pytest.mark.parametrize("run_on_cluster", [False, True]) +@pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +def test_deletion_vectors(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): + if storage_type == "local" and run_on_cluster: + pytest.skip("Local storage with cluster execution is not supported") + + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_deletion_vectors_" + storage_type + "_" + get_uuid_str() + deleted_ids = [2, 5, 7, 100] + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint) USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql(f"INSERT INTO {table_name} SELECT id FROM range(0, 200)") + spark.sql( + f"DELETE FROM {table_name} WHERE id IN ({', '.join(str(x) for x in deleted_ids)})" + ) + + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + run_on_cluster=run_on_cluster, + table_function=True, + ) + + assert int(instance.query(f"SELECT count() FROM {expression}")) == 200 - len(deleted_ids) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == [ + x for x in range(200) if x not in deleted_ids + ] + + +@pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +def test_deletion_vectors_complex(started_cluster_iceberg_with_spark, storage_type): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_deletion_vectors_complex_" + storage_type + "_" + get_uuid_str() + expected_ids = expected_complex_ids() + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint, data string) USING iceberg + PARTITIONED BY (bucket(5, id)) + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql( + f"INSERT INTO {table_name} SELECT id, char(id + ascii('a')) FROM range(10, 100)" + ) + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + table_function=True, + ) + + assert int(instance.query(f"SELECT count(id) FROM {expression}")) == 90 + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(10, 100)) + + spark.sql(f"DELETE FROM {table_name} WHERE id < 20") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 100)) + + spark.sql(f"DELETE FROM {table_name} WHERE id >= 90") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 90)) + + spark.sql( + f"INSERT INTO {table_name} SELECT id, char(id + ascii('a')) FROM range(100, 200)" + ) + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 90)) + list( + range(100, 200) + ) + + spark.sql(f"DELETE FROM {table_name} WHERE id >= 150") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 90)) + list( + range(100, 150) + ) + + spark.sql(f"ALTER TABLE {table_name} ADD COLUMNS (label string)") + spark.sql( + f""" + INSERT INTO {table_name} + SELECT id, char(id + ascii('a')), 'new' + FROM range(200, 250) + """ + ) + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 90)) + list( + range(100, 150) + ) + list(range(200, 250)) + assert int(instance.query(f"SELECT count(id) FROM {expression} WHERE label = 'new'")) == 50 + + spark.sql(f"DELETE FROM {table_name} WHERE id IN (205, 210, 220)") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == expected_ids + assert int(instance.query(f"SELECT count(id) FROM {expression}")) == len(expected_ids) + + spark.sql(f"UPDATE {table_name} SET label = 'updated' WHERE id = 25") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert instance.query(f"SELECT label FROM {expression} WHERE id = 25").strip() == "updated" + assert int(instance.query(f"SELECT count(id) FROM {expression} WHERE label = 'updated'")) == 1 + + spark.sql(f"CALL system.rewrite_data_files(table => '{table_name}')") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == expected_ids + + assert get_array( + instance.query( + f"SELECT id FROM {expression} WHERE id % 3 = 0" + ) + ) == sorted([x for x in expected_ids if x % 3 == 0]) + + +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_deletion_vectors_puffin_files_cache(started_cluster_iceberg_with_spark, storage_type): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_deletion_vectors_cache_" + storage_type + "_" + get_uuid_str() + deleted_ids = [2, 5, 7, 100] + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint) USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql(f"INSERT INTO {table_name} SELECT id FROM range(0, 200)") + spark.sql( + f"DELETE FROM {table_name} WHERE id IN ({', '.join(str(x) for x in deleted_ids)})" + ) + + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + table_function=True, + ) + + instance.query("SYSTEM DROP PUFFIN_FILES_CACHE") + + query_id1 = f"{table_name}-{uuid.uuid4()}" + query_id2 = f"{table_name}-{uuid.uuid4()}" + query_id3 = f"{table_name}-{uuid.uuid4()}" + + assert int( + instance.query( + f"SELECT count(id) FROM {expression}", + query_id=query_id1, + settings={"use_puffin_files_cache": 1}, + ) + ) == 200 - len(deleted_ids) + + assert int( + instance.query( + f"SELECT count(id) FROM {expression}", + query_id=query_id2, + settings={"use_puffin_files_cache": 1}, + ) + ) == 200 - len(deleted_ids) + + instance.query("SYSTEM FLUSH LOGS") + + assert int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesCacheMisses'] FROM system.query_log WHERE query_id = '{query_id1}' AND type = 'QueryFinish'" + ) + ) > 0 + assert int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesCacheHits'] FROM system.query_log WHERE query_id = '{query_id2}' AND type = 'QueryFinish'" + ) + ) > 0 + + puffin_reads_first = int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesRead'] FROM system.query_log WHERE query_id = '{query_id1}' AND type = 'QueryFinish'" + ) + ) + puffin_reads_second = int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesRead'] FROM system.query_log WHERE query_id = '{query_id2}' AND type = 'QueryFinish'" + ) + ) + assert puffin_reads_first > 0 + assert puffin_reads_second == 0 + + instance.query("SYSTEM DROP PUFFIN_FILES_CACHE") + + assert int( + instance.query( + f"SELECT count(id) FROM {expression}", + query_id=query_id3, + settings={"use_puffin_files_cache": 1}, + ) + ) == 200 - len(deleted_ids) + + instance.query("SYSTEM FLUSH LOGS") + + assert int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesCacheMisses'] FROM system.query_log WHERE query_id = '{query_id3}' AND type = 'QueryFinish'" + ) + ) > int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesCacheMisses'] FROM system.query_log WHERE query_id = '{query_id2}' AND type = 'QueryFinish'" + ) + ) diff --git a/tests/queries/0_stateless/01271_show_privileges.reference b/tests/queries/0_stateless/01271_show_privileges.reference index 11fae0771496..7fc845218de4 100644 --- a/tests/queries/0_stateless/01271_show_privileges.reference +++ b/tests/queries/0_stateless/01271_show_privileges.reference @@ -128,6 +128,7 @@ SYSTEM DROP MARK CACHE ['SYSTEM CLEAR MARK CACHE','SYSTEM DROP MARK','DROP MARK SYSTEM DROP ICEBERG METADATA CACHE ['SYSTEM CLEAR ICEBERG_METADATA_CACHE','SYSTEM DROP ICEBERG_METADATA_CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP AVRO SCHEMA CACHE ['SYSTEM CLEAR AVRO SCHEMA CACHE','SYSTEM DROP AVRO SCHEMA CACHE','DROP AVRO SCHEMA CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP PARQUET METADATA CACHE ['SYSTEM DROP PARQUET_METADATA_CACHE'] GLOBAL SYSTEM DROP CACHE +SYSTEM DROP PUFFIN FILES CACHE ['SYSTEM DROP PUFFIN_FILES_CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM PREWARM PRIMARY INDEX CACHE ['SYSTEM PREWARM PRIMARY INDEX','PREWARM PRIMARY INDEX CACHE','PREWARM PRIMARY INDEX'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP PRIMARY INDEX CACHE ['SYSTEM CLEAR PRIMARY INDEX CACHE','SYSTEM DROP PRIMARY INDEX','DROP PRIMARY INDEX CACHE','DROP PRIMARY INDEX'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP UNCOMPRESSED CACHE ['SYSTEM CLEAR UNCOMPRESSED CACHE','SYSTEM DROP UNCOMPRESSED','DROP UNCOMPRESSED CACHE','DROP UNCOMPRESSED'] GLOBAL SYSTEM DROP CACHE diff --git a/tests/queries/0_stateless/04077_puffin_happy_path.reference b/tests/queries/0_stateless/04077_puffin_happy_path.reference new file mode 100644 index 000000000000..3c855ff323d9 --- /dev/null +++ b/tests/queries/0_stateless/04077_puffin_happy_path.reference @@ -0,0 +1,34 @@ +--- spark_deletion_vector.puffin --- +deletion-vector-v1 -1 -1 4 58 ['cardinality','referenced-data-file'] ['5','/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet'] +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet [2,5,7,100,65536] +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 2 +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 5 +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 7 +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 100 +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 65536 +--- compressed_footer.puffin --- +deletion-vector-v1 -1 -1 4 58 ['cardinality','referenced-data-file'] ['5','/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet'] +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet [2,5,7,100,65536] +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 2 +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 5 +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 7 +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 100 +/tmp/clickhouse_spark_puffin_9eiw_i48/default/spark_puffin_source/data/00000-0-15387e93-06e3-48bb-ab87-ea4bfedb0c1d-0-00001.parquet 65536 +--- mixed_blob_types.puffin --- +apache-datasketches-theta-v1 -1 -1 4 16 [] [] +deletion-vector-v1 -1 -1 20 44 ['cardinality','referenced-data-file'] ['2','/data/table/part-00000.parquet'] +/data/table/part-00000.parquet [2,5] +/data/table/part-00000.parquet 2 +/data/table/part-00000.parquet 5 +--- file_properties_ok.puffin --- +deletion-vector-v1 -1 -1 4 44 ['cardinality','referenced-data-file'] ['2','/data/table/part-00000.parquet'] +/data/table/part-00000.parquet [2,5] +/data/table/part-00000.parquet 2 +/data/table/part-00000.parquet 5 +--- dense_range_100k.puffin --- +100000 0 99999 +--- subset without deleted_rows --- +/data/table/part-00000.parquet +/data/table/part-00000.parquet +/data/table/part-00000.parquet +/data/table/part-00000.parquet diff --git a/tests/queries/0_stateless/04077_puffin_happy_path.sh b/tests/queries/0_stateless/04077_puffin_happy_path.sh new file mode 100755 index 000000000000..f41522da71f8 --- /dev/null +++ b/tests/queries/0_stateless/04077_puffin_happy_path.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# Tags: no-fasttest + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +DATA="$CURDIR/data_puffin" + +run_happy_path() { + local name="$1" + local puffin="$2" + echo "--- $name ---" + $CLICKHOUSE_LOCAL -q " + SELECT blob_type, snapshot_id, sequence_number, offset, length, compression_codec, mapKeys(properties), mapValues(properties) + FROM file('$puffin', PuffinMetadata) + ORDER BY all + " + $CLICKHOUSE_LOCAL -q " + SELECT referenced_data_file, deleted_rows + FROM file('$puffin', Puffin) + ORDER BY all + " + $CLICKHOUSE_LOCAL -q " + SELECT referenced_data_file, row_number + FROM file('$puffin', Puffin) + ARRAY JOIN deleted_rows AS row_number + ORDER BY referenced_data_file, row_number + " +} + +run_happy_path "spark_deletion_vector.puffin" "$DATA/spark_deletion_vector.puffin" +run_happy_path "compressed_footer.puffin" "$DATA/compressed_footer.puffin" +run_happy_path "mixed_blob_types.puffin" "$DATA/mixed_blob_types.puffin" +run_happy_path "file_properties_ok.puffin" "$DATA/file_properties_ok.puffin" + +echo "--- dense_range_100k.puffin ---" +$CLICKHOUSE_LOCAL -q " +SELECT length(deleted_rows), deleted_rows[1], deleted_rows[100000] +FROM file('$DATA/dense_range_100k.puffin', Puffin) +" + +echo "--- subset without deleted_rows ---" +$CLICKHOUSE_LOCAL -q " +SELECT referenced_data_file +FROM file('$DATA/dense_range_100k.puffin', Puffin) +" +$CLICKHOUSE_LOCAL -q " +SELECT referenced_data_file +FROM file('$DATA/cardinality_exceeds_materialization_limit.puffin', Puffin) +" +# Corrupt payload is still readable when deleted_rows is not projected (payload checks are skipped). +$CLICKHOUSE_LOCAL -q " +SELECT referenced_data_file +FROM file('$DATA/dv_envelope_crc_mismatch.puffin', Puffin) +" +$CLICKHOUSE_LOCAL -q " +SELECT referenced_data_file +FROM file('$DATA/dense_range_100k.puffin', Puffin) +SETTINGS input_format_allow_seeks = 0 +" diff --git a/tests/queries/0_stateless/04117_parser_system_query_variants.reference b/tests/queries/0_stateless/04117_parser_system_query_variants.reference index 2ffe7e9fb91f..037f0692ae0e 100644 --- a/tests/queries/0_stateless/04117_parser_system_query_variants.reference +++ b/tests/queries/0_stateless/04117_parser_system_query_variants.reference @@ -15,6 +15,7 @@ SYSTEM CLEAR QUERY CONDITION CACHE SYSTEM CLEAR COMPILED EXPRESSION CACHE SYSTEM CLEAR ICEBERG METADATA CACHE SYSTEM CLEAR PARQUET METADATA CACHE +SYSTEM CLEAR PUFFIN FILES CACHE SYSTEM CLEAR FILESYSTEM CACHE SYSTEM CLEAR DISTRIBUTED CACHE SYSTEM CLEAR DISTRIBUTED CACHE CONNECTIONS ON CLUSTER cluster diff --git a/tests/queries/0_stateless/04117_parser_system_query_variants.sql b/tests/queries/0_stateless/04117_parser_system_query_variants.sql index c48bb44537f9..e52cb9f54caa 100644 --- a/tests/queries/0_stateless/04117_parser_system_query_variants.sql +++ b/tests/queries/0_stateless/04117_parser_system_query_variants.sql @@ -25,6 +25,7 @@ EXPLAIN SYNTAX SYSTEM DROP QUERY CONDITION CACHE; EXPLAIN SYNTAX SYSTEM DROP COMPILED EXPRESSION CACHE; EXPLAIN SYNTAX SYSTEM DROP ICEBERG METADATA CACHE; EXPLAIN SYNTAX SYSTEM DROP PARQUET METADATA CACHE; +EXPLAIN SYNTAX SYSTEM DROP PUFFIN_FILES_CACHE; EXPLAIN SYNTAX SYSTEM DROP FILESYSTEM CACHE; EXPLAIN SYNTAX SYSTEM DROP DISTRIBUTED CACHE; EXPLAIN SYNTAX SYSTEM DROP DISTRIBUTED CACHE CONNECTIONS ON CLUSTER cluster; diff --git a/tests/queries/0_stateless/04257_puffin_errors.reference b/tests/queries/0_stateless/04257_puffin_errors.reference new file mode 100644 index 000000000000..f8b8279a584d --- /dev/null +++ b/tests/queries/0_stateless/04257_puffin_errors.reference @@ -0,0 +1,38 @@ +--- overflow_offset_length.puffin --- +Puffin blob 0: offset/length out of bounds +--- negative_offset.puffin --- +Puffin blob 0: offset/length out of bounds +--- length_exceeds_file.puffin --- +Puffin blob 0: offset/length out of bounds +--- blob_overlaps_footer.puffin --- +Puffin blob 0: offset/length out of bounds +--- invalid_roaring_bitmap.puffin --- +Failed to deserialize deletion vector roaring bitmap +BAD_ARGUMENTS +--- invalid_bitmap_key.puffin --- +Invalid deletion vector bitmap key +--- dv_envelope_length_mismatch.puffin --- +does not match combined length +--- dv_envelope_bad_magic.puffin --- +Invalid deletion vector magic +--- dv_envelope_crc_mismatch.puffin --- +Deletion vector CRC mismatch +--- cardinality_mismatch_large_bitmap.puffin --- +exceeds declared cardinality +--- cardinality_exceeds_materialization_limit.puffin --- +exceeds materialization limit +BAD_ARGUMENTS +--- oversized_dv_blob.puffin --- +exceeds absolute limit +--- invalid_cardinality_non_numeric.puffin --- +property 'cardinality' must be an unsigned integer +--- invalid_cardinality_negative.puffin --- +property 'cardinality' must be an unsigned integer +--- invalid_cardinality_non_numeric_subset --- +property 'cardinality' must be an unsigned integer +--- invalid_cardinality_negative_subset --- +property 'cardinality' must be an unsigned integer +--- puffin_wrong_type --- +Unexpected type +--- puffin_unknown_column --- +Unexpected column diff --git a/tests/queries/0_stateless/04257_puffin_errors.sh b/tests/queries/0_stateless/04257_puffin_errors.sh new file mode 100755 index 000000000000..1ef976a39770 --- /dev/null +++ b/tests/queries/0_stateless/04257_puffin_errors.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# Tags: no-fasttest +# +# Deletion-vector / Puffin payload error cases. Split from the former monolithic +# 04257 suite so slow CI builds stay under the 300s timeout. Cases still run in +# a small parallel pool because clickhouse-local startup dominates runtime. + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh +# shellcheck source=./data_puffin/puffin_errors_common.sh +. "$CURDIR"/data_puffin/puffin_errors_common.sh + +for f in overflow_offset_length negative_offset length_exceeds_file blob_overlaps_footer +do + launch "$id" puffin "$DATA/$f.puffin" 'Puffin blob 0: offset/length out of bounds' + id=$((id + 1)) +done + +launch "$id" puffin "$DATA/invalid_roaring_bitmap.puffin" 'Failed to deserialize deletion vector roaring bitmap' 'BAD_ARGUMENTS' +id=$((id + 1)) +launch "$id" puffin "$DATA/invalid_bitmap_key.puffin" 'Invalid deletion vector bitmap key' +id=$((id + 1)) +launch "$id" puffin "$DATA/dv_envelope_length_mismatch.puffin" 'does not match combined length' +id=$((id + 1)) +launch "$id" puffin "$DATA/dv_envelope_bad_magic.puffin" 'Invalid deletion vector magic' +id=$((id + 1)) +launch "$id" puffin "$DATA/dv_envelope_crc_mismatch.puffin" 'Deletion vector CRC mismatch' +id=$((id + 1)) +launch "$id" puffin "$DATA/cardinality_mismatch_large_bitmap.puffin" 'exceeds declared cardinality' +id=$((id + 1)) +launch "$id" puffin "$DATA/cardinality_exceeds_materialization_limit.puffin" 'exceeds materialization limit' 'BAD_ARGUMENTS' +id=$((id + 1)) + +# Oversized DV blob region is generated at runtime (sparse) to avoid committing a 2 GiB fixture. +OVERSIZE_DV_BLOB="$TMP/oversized_dv_blob.puffin" +python3 - "$OVERSIZE_DV_BLOB" <<'PY' +import json +import struct +import sys + +path = sys.argv[1] +magic = b"PFA1" +blob_length = 2 * 1024 * 1024 * 1024 + 1 +footer = { + "blobs": [ + { + "type": "deletion-vector-v1", + "fields": [], + "snapshot-id": -1, + "sequence-number": -1, + "offset": 4, + "length": blob_length, + "properties": { + "referenced-data-file": "/data/table/part-00000.parquet", + "cardinality": "1", + }, + } + ] +} +footer_json = json.dumps(footer, separators=(", ", ": ")).encode("utf-8") +flags = b"\x00\x00\x00\x00" +with open(path, "wb") as f: + f.write(magic) + f.seek(4 + blob_length - 1, 0) + f.write(b"\x00") + f.write(magic) + f.write(footer_json) + f.write(struct.pack(" "$NOT_PUFFIN" +dd if=/dev/zero bs=1024 count=1024 status=none >> "$NOT_PUFFIN" +$CLICKHOUSE_LOCAL -q " +SELECT blob_type +FROM file('$NOT_PUFFIN', PuffinMetadata) +SETTINGS input_format_allow_seeks = 0 +" 2>&1 | grep -oF 'Invalid Puffin magic (header)' diff --git a/tests/queries/0_stateless/04612_puffin_metadata_footer_errors.reference b/tests/queries/0_stateless/04612_puffin_metadata_footer_errors.reference new file mode 100644 index 000000000000..ee92652feb71 --- /dev/null +++ b/tests/queries/0_stateless/04612_puffin_metadata_footer_errors.reference @@ -0,0 +1,40 @@ +--- inflated_lz4_content_size.puffin --- +Puffin footer LZ4 content size +--- lz4_content_size_over_absolute_cap.puffin --- +absolute decompression limit +--- missing_lz4_content_size.puffin --- +Puffin footer LZ4 frame must declare content size +--- lz4_trailing_bytes.puffin --- +trailing bytes +--- incomplete_lz4_footer.puffin --- +Puffin footer LZ4 frame is incomplete +--- missing_type.puffin --- +missing required field +--- missing_blobs.puffin --- +missing required field 'blobs' +--- null_blobs.puffin --- +missing required field 'blobs' +--- null_blob_entry.puffin --- +must be an object +--- invalid_blob_entry.puffin --- +must be an object +--- missing_properties.puffin --- +missing required field +--- missing_referenced_data_file.puffin --- +missing required property +--- missing_cardinality.puffin --- +missing required property +--- invalid_properties_array.puffin --- +field 'properties' must be an object +--- null_properties.puffin --- +field 'properties' must be an object +--- invalid_file_properties_array.puffin --- +Puffin footer field 'properties' must be an object +--- null_file_properties.puffin --- +Puffin footer field 'properties' must be an object +--- invalid_file_property_number.puffin --- +Puffin footer property +--- missing_footer_leading_magic.puffin --- +Invalid Puffin footer length +--- oversized_raw_footer.puffin --- +exceeds absolute limit diff --git a/tests/queries/0_stateless/04612_puffin_metadata_footer_errors.sh b/tests/queries/0_stateless/04612_puffin_metadata_footer_errors.sh new file mode 100755 index 000000000000..ca6bac8941c0 --- /dev/null +++ b/tests/queries/0_stateless/04612_puffin_metadata_footer_errors.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +# Tags: no-fasttest +# +# PuffinMetadata footer / required-field error cases. Split from 04257 so CI +# stays under the 300s timeout; cases run in a small parallel pool. + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh +# shellcheck source=./data_puffin/puffin_errors_common.sh +. "$CURDIR"/data_puffin/puffin_errors_common.sh + +launch "$id" meta "$DATA/inflated_lz4_content_size.puffin" 'Puffin footer LZ4 content size' +id=$((id + 1)) +launch "$id" meta "$DATA/lz4_content_size_over_absolute_cap.puffin" 'absolute decompression limit' +id=$((id + 1)) +launch "$id" meta "$DATA/missing_lz4_content_size.puffin" 'Puffin footer LZ4 frame must declare content size' +id=$((id + 1)) +launch "$id" meta "$DATA/lz4_trailing_bytes.puffin" 'trailing bytes' +id=$((id + 1)) +launch "$id" meta "$DATA/incomplete_lz4_footer.puffin" 'Puffin footer LZ4 frame is incomplete' +id=$((id + 1)) + +# One missing blob field covers requireBlobMetadataField for all required keys. +launch "$id" meta "$DATA/missing_type.puffin" 'missing required field' +id=$((id + 1)) + +launch "$id" meta "$DATA/missing_blobs.puffin" "missing required field 'blobs'" +id=$((id + 1)) +launch "$id" meta "$DATA/null_blobs.puffin" "missing required field 'blobs'" +id=$((id + 1)) + +for f in null_blob_entry invalid_blob_entry +do + launch "$id" meta "$DATA/$f.puffin" 'must be an object' + id=$((id + 1)) +done + +for f in missing_properties missing_referenced_data_file missing_cardinality +do + launch "$id" meta_re "$DATA/$f.puffin" 'missing required (field|property)' + id=$((id + 1)) +done + +for f in invalid_properties_array null_properties +do + launch "$id" meta "$DATA/$f.puffin" "field 'properties' must be an object" + id=$((id + 1)) +done + +for f in invalid_file_properties_array null_file_properties +do + launch "$id" meta "$DATA/$f.puffin" "Puffin footer field 'properties' must be an object" + id=$((id + 1)) +done + +launch "$id" meta "$DATA/invalid_file_property_number.puffin" "Puffin footer property" +id=$((id + 1)) + +launch "$id" meta "$DATA/missing_footer_leading_magic.puffin" 'Invalid Puffin footer length' +id=$((id + 1)) + +# Oversized raw footer is generated at runtime (sparse) to avoid committing a 16 MiB fixture. +OVERSIZE_FOOTER="$TMP/oversized_raw_footer.puffin" +python3 - "$OVERSIZE_FOOTER" <<'PY' +import struct +import sys + +path = sys.argv[1] +magic = b"PFA1" +footer_length = 16 * 1024 * 1024 + 1 +flags = b"\x00\x00\x00\x00" +with open(path, "wb") as f: + f.write(magic) + f.write(magic) + f.seek(footer_length - 1, 1) + f.write(b"{") + f.write(struct.pack("= 90` +- Insert `id` 100-199, delete `id >= 150` +- Schema evolution (`label` column), insert `id` 200-249 with `label = 'new'` +- Delete `id` in (205, 210, 220), update `id = 25` to `label = 'updated'` +- Spark `rewrite_data_files` compaction + +Regenerate: + +```bash +python3 generate_iceberg_dv_fixture.py +``` + +Query paths for tests: + +```text +data_minio/dv_puffin_warehouse/default/dv_puffin_source +data_minio/dv_puffin_warehouse/default/dv_puffin_complex +``` diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-14-ba106366-1379-49df-a585-047546b341d9-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-14-ba106366-1379-49df-a585-047546b341d9-00001-deletes.puffin new file mode 100644 index 000000000000..956042e717a5 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-14-ba106366-1379-49df-a585-047546b341d9-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-18-0da8af70-7e2e-4ced-935b-d2b8ef79eb76-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-18-0da8af70-7e2e-4ced-935b-d2b8ef79eb76-00001-deletes.puffin new file mode 100644 index 000000000000..94aa1644cfd8 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-18-0da8af70-7e2e-4ced-935b-d2b8ef79eb76-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001-deletes.puffin new file mode 100644 index 000000000000..ce4a05ab2431 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-6-74340066-9c57-43b9-9112-c569f91929b9-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-6-74340066-9c57-43b9-9112-c569f91929b9-00001-deletes.puffin new file mode 100644 index 000000000000..7bb98d451f28 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-6-74340066-9c57-43b9-9112-c569f91929b9-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-9-dcc19f77-3570-494e-b336-f4d3301291b4-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-9-dcc19f77-3570-494e-b336-f4d3301291b4-00001-deletes.puffin new file mode 100644 index 000000000000..7df1cec9ff2a Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-9-dcc19f77-3570-494e-b336-f4d3301291b4-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00005.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00005.parquet new file mode 100644 index 000000000000..fb364a125c2f Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00005.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00005.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00005.parquet new file mode 100644 index 000000000000..d9b655282bf8 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00005.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001.parquet new file mode 100644 index 000000000000..6dfde23d8eae Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-25-ee6fe6d5-2f4f-47e6-ad97-00c53284ff87-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-25-ee6fe6d5-2f4f-47e6-ad97-00c53284ff87-0-00001.parquet new file mode 100644 index 000000000000..97c37bb1ab5d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-25-ee6fe6d5-2f4f-47e6-ad97-00c53284ff87-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00005.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00005.parquet new file mode 100644 index 000000000000..f5da4d7105d6 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00005.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00001.parquet new file mode 100644 index 000000000000..43f7e9c338c6 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00001.parquet new file mode 100644 index 000000000000..ada984ceed8d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-24-0b9f04c6-aebb-4cfc-a3b7-3f2a1034a6dc-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-24-0b9f04c6-aebb-4cfc-a3b7-3f2a1034a6dc-0-00001.parquet new file mode 100644 index 000000000000..0f39927f6c7b Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-24-0b9f04c6-aebb-4cfc-a3b7-3f2a1034a6dc-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00001.parquet new file mode 100644 index 000000000000..76b701bf1e2b Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00004.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00004.parquet new file mode 100644 index 000000000000..8b4c38ecebe0 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00004.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00004.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00004.parquet new file mode 100644 index 000000000000..cb037bcb4efa Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00004.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-23-292ac3e8-88a4-482a-9a88-e08ad8bdd2e6-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-23-292ac3e8-88a4-482a-9a88-e08ad8bdd2e6-0-00001.parquet new file mode 100644 index 000000000000..690316c0bced Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-23-292ac3e8-88a4-482a-9a88-e08ad8bdd2e6-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00004.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00004.parquet new file mode 100644 index 000000000000..56882346e595 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00004.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00002.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00002.parquet new file mode 100644 index 000000000000..f9661fe1b8f3 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00002.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00002.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00002.parquet new file mode 100644 index 000000000000..c9baf49eb9c9 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00002.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-21-55307a4a-f583-48b6-983a-c2993f261fec-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-21-55307a4a-f583-48b6-983a-c2993f261fec-0-00001.parquet new file mode 100644 index 000000000000..4a5ea28ead0c Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-21-55307a4a-f583-48b6-983a-c2993f261fec-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00002.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00002.parquet new file mode 100644 index 000000000000..82ba5ea76d37 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00002.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00003.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00003.parquet new file mode 100644 index 000000000000..852ad728cd69 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00003.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00003.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00003.parquet new file mode 100644 index 000000000000..4d8d2c59eee2 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00003.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-22-cbc4c1d2-88dd-4d8a-a31a-6db198e852d4-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-22-cbc4c1d2-88dd-4d8a-a31a-6db198e852d4-0-00001.parquet new file mode 100644 index 000000000000..9fcb8cee529f Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-22-cbc4c1d2-88dd-4d8a-a31a-6db198e852d4-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00003.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00003.parquet new file mode 100644 index 000000000000..50b6c3ed85e6 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00003.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/0c03abfe-66a8-43ab-a765-a9e8072b59ed-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/0c03abfe-66a8-43ab-a765-a9e8072b59ed-m0.avro new file mode 100644 index 000000000000..50388b8a2ebb Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/0c03abfe-66a8-43ab-a765-a9e8072b59ed-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34-m0.avro new file mode 100644 index 000000000000..2289c0aaeea3 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m0.avro new file mode 100644 index 000000000000..0cce769e50f0 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m1.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m1.avro new file mode 100644 index 000000000000..c9facfde4c04 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m1.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da-m0.avro new file mode 100644 index 000000000000..2f84bfe01b74 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m0.avro new file mode 100644 index 000000000000..3f7df28d60e9 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m1.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m1.avro new file mode 100644 index 000000000000..59d5e1f574b4 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m1.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m0.avro new file mode 100644 index 000000000000..9084391f0e62 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m1.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m1.avro new file mode 100644 index 000000000000..c42c203e2b30 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m1.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m2.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m2.avro new file mode 100644 index 000000000000..97a79ed3f6d9 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m2.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m3.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m3.avro new file mode 100644 index 000000000000..e96733a33279 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m3.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m4.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m4.avro new file mode 100644 index 000000000000..1825a92f8fa4 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m4.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/6a19b33f-2be5-4073-820c-82cb1c7d76cb-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/6a19b33f-2be5-4073-820c-82cb1c7d76cb-m0.avro new file mode 100644 index 000000000000..b41643648c61 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/6a19b33f-2be5-4073-820c-82cb1c7d76cb-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/b68e784a-6b9c-4894-827d-c56978599ecd-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/b68e784a-6b9c-4894-827d-c56978599ecd-m0.avro new file mode 100644 index 000000000000..2dbdc71cc3cd Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/b68e784a-6b9c-4894-827d-c56978599ecd-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/c21c4572-6006-4649-85b1-47e82e73a396-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/c21c4572-6006-4649-85b1-47e82e73a396-m0.avro new file mode 100644 index 000000000000..4f57b9fff1fe Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/c21c4572-6006-4649-85b1-47e82e73a396-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro new file mode 100644 index 000000000000..1c6208295989 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro new file mode 100644 index 000000000000..619989a8a328 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro new file mode 100644 index 000000000000..62699c0aa26d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro new file mode 100644 index 000000000000..151ec4194d68 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro new file mode 100644 index 000000000000..9ca00ca319c2 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro new file mode 100644 index 000000000000..929e6598ea9f Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro new file mode 100644 index 000000000000..070f738c8a08 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8541702041949031574-1-5b66c606-c296-4fe4-a5f8-d8be592f6b96.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8541702041949031574-1-5b66c606-c296-4fe4-a5f8-d8be592f6b96.avro new file mode 100644 index 000000000000..8911f605816f Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8541702041949031574-1-5b66c606-c296-4fe4-a5f8-d8be592f6b96.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro new file mode 100644 index 000000000000..09ad8a80111b Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json new file mode 100644 index 000000000000..ee75c0491a42 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":0,"last-updated-ms":1783931355607,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":null,"next-row-id":0,"refs":{},"snapshots":[],"statistics":[],"partition-statistics":[],"snapshot-log":[],"metadata-log":[]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v10.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v10.metadata.json new file mode 100644 index 000000000000..e6b202777546 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v10.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":8,"last-updated-ms":1783931358480,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":9221420601809522049,"next-row-id":241,"refs":{"main":{"snapshot-id":9221420601809522049,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0},{"sequence-number":6,"snapshot-id":2216958009875447676,"parent-snapshot-id":212327459647764229,"timestamp-ms":1783931357776,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"50","added-files-size":"5103","changed-partition-count":"5","total-records":"240","total-files-size":"13293","total-data-files":"15","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro","schema-id":1,"first-row-id":190,"added-rows":50},{"sequence-number":7,"snapshot-id":399037466005519109,"parent-snapshot-id":2216958009875447676,"timestamp-ms":1783931358118,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"126","added-position-deletes":"3","changed-partition-count":"3","total-records":"240","total-files-size":"13419","total-data-files":"15","total-delete-files":"12","total-position-deletes":"73","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro","schema-id":1,"first-row-id":240,"added-rows":0},{"sequence-number":8,"snapshot-id":9221420601809522049,"parent-snapshot-id":399037466005519109,"timestamp-ms":1783931358480,"summary":{"operation":"overwrite","spark.app.id":"local-1783931350618","added-data-files":"1","added-delete-files":"1","added-dvs":"1","added-records":"1","added-files-size":"1011","added-position-deletes":"1","changed-partition-count":"1","total-records":"241","total-files-size":"14430","total-data-files":"16","total-delete-files":"13","total-position-deletes":"74","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro","schema-id":1,"first-row-id":240,"added-rows":1}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229},{"timestamp-ms":1783931357776,"snapshot-id":2216958009875447676},{"timestamp-ms":1783931358118,"snapshot-id":399037466005519109},{"timestamp-ms":1783931358480,"snapshot-id":9221420601809522049}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"},{"timestamp-ms":1783931357478,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json"},{"timestamp-ms":1783931357776,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json"},{"timestamp-ms":1783931358118,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v11.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v11.metadata.json new file mode 100644 index 000000000000..a29d02234e63 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v11.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":9,"last-updated-ms":1783931359048,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":8541702041949031574,"next-row-id":408,"refs":{"main":{"snapshot-id":8541702041949031574,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0},{"sequence-number":6,"snapshot-id":2216958009875447676,"parent-snapshot-id":212327459647764229,"timestamp-ms":1783931357776,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"50","added-files-size":"5103","changed-partition-count":"5","total-records":"240","total-files-size":"13293","total-data-files":"15","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro","schema-id":1,"first-row-id":190,"added-rows":50},{"sequence-number":7,"snapshot-id":399037466005519109,"parent-snapshot-id":2216958009875447676,"timestamp-ms":1783931358118,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"126","added-position-deletes":"3","changed-partition-count":"3","total-records":"240","total-files-size":"13419","total-data-files":"15","total-delete-files":"12","total-position-deletes":"73","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro","schema-id":1,"first-row-id":240,"added-rows":0},{"sequence-number":8,"snapshot-id":9221420601809522049,"parent-snapshot-id":399037466005519109,"timestamp-ms":1783931358480,"summary":{"operation":"overwrite","spark.app.id":"local-1783931350618","added-data-files":"1","added-delete-files":"1","added-dvs":"1","added-records":"1","added-files-size":"1011","added-position-deletes":"1","changed-partition-count":"1","total-records":"241","total-files-size":"14430","total-data-files":"16","total-delete-files":"13","total-position-deletes":"74","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro","schema-id":1,"first-row-id":240,"added-rows":1},{"sequence-number":9,"snapshot-id":8541702041949031574,"parent-snapshot-id":9221420601809522049,"timestamp-ms":1783931359048,"summary":{"operation":"replace","added-data-files":"5","deleted-data-files":"16","added-records":"167","deleted-records":"241","added-files-size":"5886","removed-files-size":"13895","changed-partition-count":"5","total-records":"167","total-files-size":"6421","total-data-files":"5","total-delete-files":"13","total-position-deletes":"74","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8541702041949031574-1-5b66c606-c296-4fe4-a5f8-d8be592f6b96.avro","schema-id":1,"first-row-id":241,"added-rows":167}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229},{"timestamp-ms":1783931357776,"snapshot-id":2216958009875447676},{"timestamp-ms":1783931358118,"snapshot-id":399037466005519109},{"timestamp-ms":1783931358480,"snapshot-id":9221420601809522049},{"timestamp-ms":1783931359048,"snapshot-id":8541702041949031574}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"},{"timestamp-ms":1783931357478,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json"},{"timestamp-ms":1783931357776,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json"},{"timestamp-ms":1783931358118,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json"},{"timestamp-ms":1783931358480,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v10.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json new file mode 100644 index 000000000000..be4f8957698b --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":1,"last-updated-ms":1783931356056,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":8352556540874311077,"next-row-id":90,"refs":{"main":{"snapshot-id":8352556540874311077,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json new file mode 100644 index 000000000000..ec655a2cfe44 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":2,"last-updated-ms":1783931356399,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":2245381059480343696,"next-row-id":90,"refs":{"main":{"snapshot-id":2245381059480343696,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json new file mode 100644 index 000000000000..7de46c7bff57 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":3,"last-updated-ms":1783931356769,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":7131460395349610904,"next-row-id":90,"refs":{"main":{"snapshot-id":7131460395349610904,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json new file mode 100644 index 000000000000..c118c7839b1d --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":4,"last-updated-ms":1783931357067,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":243132775900853210,"next-row-id":190,"refs":{"main":{"snapshot-id":243132775900853210,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json new file mode 100644 index 000000000000..0efae360a449 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":5,"last-updated-ms":1783931357425,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":212327459647764229,"next-row-id":190,"refs":{"main":{"snapshot-id":212327459647764229,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json new file mode 100644 index 000000000000..83d3f5ec91f6 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":5,"last-updated-ms":1783931357478,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":212327459647764229,"next-row-id":190,"refs":{"main":{"snapshot-id":212327459647764229,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json new file mode 100644 index 000000000000..3479448d1bf0 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":6,"last-updated-ms":1783931357776,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":2216958009875447676,"next-row-id":240,"refs":{"main":{"snapshot-id":2216958009875447676,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0},{"sequence-number":6,"snapshot-id":2216958009875447676,"parent-snapshot-id":212327459647764229,"timestamp-ms":1783931357776,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"50","added-files-size":"5103","changed-partition-count":"5","total-records":"240","total-files-size":"13293","total-data-files":"15","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro","schema-id":1,"first-row-id":190,"added-rows":50}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229},{"timestamp-ms":1783931357776,"snapshot-id":2216958009875447676}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"},{"timestamp-ms":1783931357478,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json new file mode 100644 index 000000000000..c317c0743b1b --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":7,"last-updated-ms":1783931358118,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":399037466005519109,"next-row-id":240,"refs":{"main":{"snapshot-id":399037466005519109,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0},{"sequence-number":6,"snapshot-id":2216958009875447676,"parent-snapshot-id":212327459647764229,"timestamp-ms":1783931357776,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"50","added-files-size":"5103","changed-partition-count":"5","total-records":"240","total-files-size":"13293","total-data-files":"15","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro","schema-id":1,"first-row-id":190,"added-rows":50},{"sequence-number":7,"snapshot-id":399037466005519109,"parent-snapshot-id":2216958009875447676,"timestamp-ms":1783931358118,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"126","added-position-deletes":"3","changed-partition-count":"3","total-records":"240","total-files-size":"13419","total-data-files":"15","total-delete-files":"12","total-position-deletes":"73","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro","schema-id":1,"first-row-id":240,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229},{"timestamp-ms":1783931357776,"snapshot-id":2216958009875447676},{"timestamp-ms":1783931358118,"snapshot-id":399037466005519109}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"},{"timestamp-ms":1783931357478,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json"},{"timestamp-ms":1783931357776,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/version-hint.text b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/version-hint.text new file mode 100644 index 000000000000..9d607966b721 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/version-hint.text @@ -0,0 +1 @@ +11 \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-0-e150e3cb-5697-43f9-b2ea-9894139e9373-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-0-e150e3cb-5697-43f9-b2ea-9894139e9373-0-00001.parquet new file mode 100644 index 000000000000..4a32f9743fe7 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-0-e150e3cb-5697-43f9-b2ea-9894139e9373-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-2-ee31c25d-1223-465e-b4d7-6366f5ebda40-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-2-ee31c25d-1223-465e-b4d7-6366f5ebda40-00001-deletes.puffin new file mode 100644 index 000000000000..64c688f02ccd Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-2-ee31c25d-1223-465e-b4d7-6366f5ebda40-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57-m0.avro new file mode 100644 index 000000000000..1f1d617f028d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/c34940c4-6b84-4560-85a3-35e17802d85d-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/c34940c4-6b84-4560-85a3-35e17802d85d-m0.avro new file mode 100644 index 000000000000..684c55f993d7 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/c34940c4-6b84-4560-85a3-35e17802d85d-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro new file mode 100644 index 000000000000..a968682b674d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-8278841256334127309-1-0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-8278841256334127309-1-0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57.avro new file mode 100644 index 000000000000..ef9e9c5fc2f3 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-8278841256334127309-1-0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json new file mode 100644 index 000000000000..761494ac6f45 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"1d4ead84-5f74-4062-a8d2-46b419a5650d","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source","last-sequence-number":0,"last-updated-ms":1783931352802,"last-column-id":1,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":null,"next-row-id":0,"refs":{},"snapshots":[],"statistics":[],"partition-statistics":[],"snapshot-log":[],"metadata-log":[]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v2.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v2.metadata.json new file mode 100644 index 000000000000..6470e30e9314 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v2.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"1d4ead84-5f74-4062-a8d2-46b419a5650d","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source","last-sequence-number":1,"last-updated-ms":1783931354428,"last-column-id":1,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":1455293623190430422,"next-row-id":200,"refs":{"main":{"snapshot-id":1455293623190430422,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":1455293623190430422,"timestamp-ms":1783931354428,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"1","added-records":"200","added-files-size":"734","changed-partition-count":"1","total-records":"200","total-files-size":"734","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro","schema-id":0,"first-row-id":0,"added-rows":200}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931354428,"snapshot-id":1455293623190430422}],"metadata-log":[{"timestamp-ms":1783931352802,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v3.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v3.metadata.json new file mode 100644 index 000000000000..d3e24bdc9b16 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v3.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"1d4ead84-5f74-4062-a8d2-46b419a5650d","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source","last-sequence-number":2,"last-updated-ms":1783931355547,"last-column-id":1,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":8278841256334127309,"next-row-id":200,"refs":{"main":{"snapshot-id":8278841256334127309,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":1455293623190430422,"timestamp-ms":1783931354428,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"1","added-records":"200","added-files-size":"734","changed-partition-count":"1","total-records":"200","total-files-size":"734","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro","schema-id":0,"first-row-id":0,"added-rows":200},{"sequence-number":2,"snapshot-id":8278841256334127309,"parent-snapshot-id":1455293623190430422,"timestamp-ms":1783931355547,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"1","added-dvs":"1","added-files-size":"48","added-position-deletes":"4","changed-partition-count":"1","total-records":"200","total-files-size":"782","total-data-files":"1","total-delete-files":"1","total-position-deletes":"4","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-8278841256334127309-1-0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57.avro","schema-id":0,"first-row-id":200,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931354428,"snapshot-id":1455293623190430422},{"timestamp-ms":1783931355547,"snapshot-id":8278841256334127309}],"metadata-log":[{"timestamp-ms":1783931352802,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json"},{"timestamp-ms":1783931354428,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v2.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/version-hint.text b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/version-hint.text new file mode 100644 index 000000000000..e440e5c84258 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/version-hint.text @@ -0,0 +1 @@ +3 \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/generate_iceberg_dv_fixture.py b/tests/queries/0_stateless/data_minio/generate_iceberg_dv_fixture.py new file mode 100644 index 000000000000..86e4ffd08804 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/generate_iceberg_dv_fixture.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""Generate Iceberg v3 table fixtures with Puffin deletion vectors for ClickHouse tests. + +Requirements: + - Java 11+ + - pyspark + - Network access on first run (downloads iceberg-spark-runtime) + +Usage: + python3 generate_iceberg_dv_fixture.py [warehouse_dir] + python3 generate_iceberg_dv_fixture.py --complex-only [warehouse_dir] +""" + +from __future__ import annotations + +import shutil +import sys +from pathlib import Path + +try: + from pyspark.sql import SparkSession +except ImportError as exc: # pragma: no cover - helper script + raise SystemExit("pyspark is required") from exc + +DEFAULT_WAREHOUSE = Path(__file__).resolve().parent / "dv_puffin_warehouse" +ICEBERG_PACKAGE = "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0" +SIMPLE_TABLE_NAME = "dv_puffin_source" +COMPLEX_TABLE_NAME = "dv_puffin_complex" +SIMPLE_DELETED_IDS = [2, 5, 7, 100] +COMPLEX_DELETED_IDS = [205, 210, 220] + + +def build_spark_session(warehouse: Path) -> SparkSession: + return ( + SparkSession.builder.appName("generate_iceberg_dv_fixture") + .master("local[1]") + .config("spark.jars.packages", ICEBERG_PACKAGE) + .config( + "spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + ) + .config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog") + .config("spark.sql.catalog.spark_catalog.type", "hadoop") + .config("spark.sql.catalog.spark_catalog.warehouse", str(warehouse)) + .config("spark.ui.enabled", "false") + .getOrCreate() + ) + + +def generate_simple_fixture(spark: SparkSession) -> None: + spark.sql( + f""" + CREATE TABLE default.{SIMPLE_TABLE_NAME} (id BIGINT) + USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql(f"INSERT INTO default.{SIMPLE_TABLE_NAME} SELECT id FROM range(0, 200)") + spark.sql( + f"DELETE FROM default.{SIMPLE_TABLE_NAME} " + f"WHERE id IN ({', '.join(str(x) for x in SIMPLE_DELETED_IDS)})" + ) + + +def generate_complex_fixture(spark: SparkSession) -> None: + spark.sql( + f""" + CREATE TABLE default.{COMPLEX_TABLE_NAME} (id BIGINT, data STRING) + USING iceberg + PARTITIONED BY (bucket(5, id)) + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql( + f"INSERT INTO default.{COMPLEX_TABLE_NAME} " + f"SELECT id, char(id + ascii('a')) FROM range(10, 100)" + ) + spark.sql(f"DELETE FROM default.{COMPLEX_TABLE_NAME} WHERE id < 20") + spark.sql(f"DELETE FROM default.{COMPLEX_TABLE_NAME} WHERE id >= 90") + spark.sql( + f"INSERT INTO default.{COMPLEX_TABLE_NAME} " + f"SELECT id, char(id + ascii('a')) FROM range(100, 200)" + ) + spark.sql(f"DELETE FROM default.{COMPLEX_TABLE_NAME} WHERE id >= 150") + spark.sql(f"ALTER TABLE default.{COMPLEX_TABLE_NAME} ADD COLUMNS (label STRING)") + spark.sql( + f""" + INSERT INTO default.{COMPLEX_TABLE_NAME} + SELECT id, char(id + ascii('a')), 'new' + FROM range(200, 250) + """ + ) + spark.sql( + f"DELETE FROM default.{COMPLEX_TABLE_NAME} " + f"WHERE id IN ({', '.join(str(x) for x in COMPLEX_DELETED_IDS)})" + ) + spark.sql(f"UPDATE default.{COMPLEX_TABLE_NAME} SET label = 'updated' WHERE id = 25") + spark.sql(f"CALL system.rewrite_data_files(table => 'default.{COMPLEX_TABLE_NAME}')") + + +def cleanup_crc_files(table_dir: Path) -> None: + for crc_file in table_dir.rglob("*.crc"): + crc_file.unlink() + + +def generate_fixture(warehouse: Path, *, simple: bool, complex_table: bool) -> None: + spark = build_spark_session(warehouse) + try: + if simple: + generate_simple_fixture(spark) + if complex_table: + generate_complex_fixture(spark) + finally: + spark.stop() + + if simple: + simple_dir = warehouse / "default" / SIMPLE_TABLE_NAME + if not simple_dir.exists(): + raise RuntimeError(f"Expected table directory at {simple_dir}") + cleanup_crc_files(simple_dir) + + if complex_table: + complex_dir = warehouse / "default" / COMPLEX_TABLE_NAME + if not complex_dir.exists(): + raise RuntimeError(f"Expected table directory at {complex_dir}") + cleanup_crc_files(complex_dir) + + +def main() -> None: + args = sys.argv[1:] + complex_only = False + if args and args[0] == "--complex-only": + complex_only = True + args = args[1:] + + warehouse = Path(args[0]).resolve() if args else DEFAULT_WAREHOUSE.resolve() + if warehouse.exists(): + shutil.rmtree(warehouse) + warehouse.mkdir(parents=True) + + generate_fixture(warehouse, simple=not complex_only, complex_table=True) + + if complex_only: + print(f"Wrote complex Iceberg v3 deletion vector fixture to {warehouse / 'default' / COMPLEX_TABLE_NAME}") + else: + print(f"Wrote Iceberg v3 deletion vector fixtures to {warehouse / 'default'}") + + +if __name__ == "__main__": + main() diff --git a/tests/queries/0_stateless/data_puffin/blob_overlaps_footer.puffin b/tests/queries/0_stateless/data_puffin/blob_overlaps_footer.puffin new file mode 100644 index 000000000000..99774a3ed01b Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/blob_overlaps_footer.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/bool_fields_element.puffin b/tests/queries/0_stateless/data_puffin/bool_fields_element.puffin new file mode 100644 index 000000000000..6a8f27b86807 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/bool_fields_element.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/bool_offset.puffin b/tests/queries/0_stateless/data_puffin/bool_offset.puffin new file mode 100644 index 000000000000..cf8c90dbc0f0 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/bool_offset.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/cardinality_exceeds_materialization_limit.puffin b/tests/queries/0_stateless/data_puffin/cardinality_exceeds_materialization_limit.puffin new file mode 100644 index 000000000000..ec6fd4addda8 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/cardinality_exceeds_materialization_limit.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/cardinality_mismatch_large_bitmap.puffin b/tests/queries/0_stateless/data_puffin/cardinality_mismatch_large_bitmap.puffin new file mode 100644 index 000000000000..e4f14a998f5d Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/cardinality_mismatch_large_bitmap.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/compressed_footer.puffin b/tests/queries/0_stateless/data_puffin/compressed_footer.puffin new file mode 100644 index 000000000000..ec4fe3ac3ab3 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/compressed_footer.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/compression_codec_null.puffin b/tests/queries/0_stateless/data_puffin/compression_codec_null.puffin new file mode 100644 index 000000000000..d15038ed8c45 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/compression_codec_null.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/compression_codec_number.puffin b/tests/queries/0_stateless/data_puffin/compression_codec_number.puffin new file mode 100644 index 000000000000..8f826a7477fb Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/compression_codec_number.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/dense_range_100k.puffin b/tests/queries/0_stateless/data_puffin/dense_range_100k.puffin new file mode 100644 index 000000000000..8030beafbe5a Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/dense_range_100k.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/dv_envelope_bad_magic.puffin b/tests/queries/0_stateless/data_puffin/dv_envelope_bad_magic.puffin new file mode 100644 index 000000000000..d445aab49a0a Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/dv_envelope_bad_magic.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/dv_envelope_crc_mismatch.puffin b/tests/queries/0_stateless/data_puffin/dv_envelope_crc_mismatch.puffin new file mode 100644 index 000000000000..941d40183c97 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/dv_envelope_crc_mismatch.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/dv_envelope_length_mismatch.puffin b/tests/queries/0_stateless/data_puffin/dv_envelope_length_mismatch.puffin new file mode 100644 index 000000000000..eefdd6fe794f Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/dv_envelope_length_mismatch.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/dv_nonzero_snapshot_id.puffin b/tests/queries/0_stateless/data_puffin/dv_nonzero_snapshot_id.puffin new file mode 100644 index 000000000000..3a186d83bf47 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/dv_nonzero_snapshot_id.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/dv_with_compression_codec.puffin b/tests/queries/0_stateless/data_puffin/dv_with_compression_codec.puffin new file mode 100644 index 000000000000..8a2ccd88e81c Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/dv_with_compression_codec.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/fields_element_out_of_int32_range.puffin b/tests/queries/0_stateless/data_puffin/fields_element_out_of_int32_range.puffin new file mode 100644 index 000000000000..613e15ebba46 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/fields_element_out_of_int32_range.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/fields_element_out_of_int64_range.puffin b/tests/queries/0_stateless/data_puffin/fields_element_out_of_int64_range.puffin new file mode 100644 index 000000000000..61945ceadde8 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/fields_element_out_of_int64_range.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/file_properties_ok.puffin b/tests/queries/0_stateless/data_puffin/file_properties_ok.puffin new file mode 100644 index 000000000000..4c91e051344c Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/file_properties_ok.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/float_fields_element.puffin b/tests/queries/0_stateless/data_puffin/float_fields_element.puffin new file mode 100644 index 000000000000..901491254815 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/float_fields_element.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/float_offset.puffin b/tests/queries/0_stateless/data_puffin/float_offset.puffin new file mode 100644 index 000000000000..922f7ec13ce9 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/float_offset.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/footer_integer_overflow.puffin b/tests/queries/0_stateless/data_puffin/footer_integer_overflow.puffin new file mode 100644 index 000000000000..365325e36b2a Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/footer_integer_overflow.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/footer_root_array.puffin b/tests/queries/0_stateless/data_puffin/footer_root_array.puffin new file mode 100644 index 000000000000..bfb8f877841b Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/footer_root_array.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/generate_puffin_fixtures.py b/tests/queries/0_stateless/data_puffin/generate_puffin_fixtures.py new file mode 100644 index 000000000000..b9000a523cf7 --- /dev/null +++ b/tests/queries/0_stateless/data_puffin/generate_puffin_fixtures.py @@ -0,0 +1,745 @@ +#!/usr/bin/env python3 +"""Generate synthetic puffin test fixtures for ClickHouse stateless tests.""" + +from __future__ import annotations + +import json +import struct +import subprocess +import sys +import zlib +from pathlib import Path + +try: + import lz4.frame + import xxhash +except ImportError: + subprocess.check_call([sys.executable, "-m", "pip", "install", "lz4", "xxhash", "-q"]) + import lz4.frame + import xxhash + +try: + import pyroaring +except ImportError: + subprocess.check_call([sys.executable, "-m", "pip", "install", "pyroaring", "-q"]) + import pyroaring + +OUTPUT_DIR = Path(__file__).parent +PUFFIN_MAGIC = b"PFA1" +DELETION_VECTOR_MAGIC = bytes([0xD1, 0xD3, 0x39, 0x64]) +INFLATED_CONTENT_SIZE = 0x40000000 +# Must stay in sync with PUFFIN_FOOTER_MAX_PAYLOAD_SIZE in PuffinBlockInputFormat.cpp. +FOOTER_MAX_PAYLOAD_SIZE = 16 * 1024 * 1024 +FOOTER_LZ4_MAX_RATIO = 255 +INVALID_KEY = 0x7FFFFFFF +LARGE_KEY = 1_000_000 +SPARSE_SUB_POSITION = 42 +BLOB_PLACEHOLDER = b"\x00" * 58 +DEFAULT_REFERENCED_DATA_FILE = "/data/table/part-00000.parquet" + + +def default_dv_properties(cardinality: str = "0") -> dict[str, str]: + return { + "referenced-data-file": DEFAULT_REFERENCED_DATA_FILE, + "cardinality": cardinality, + } + + +def wrap_deletion_vector_blob(vector: bytes) -> bytes: + combined_length = len(DELETION_VECTOR_MAGIC) + len(vector) + crc_input = DELETION_VECTOR_MAGIC + vector + crc = zlib.crc32(crc_input) & 0xFFFFFFFF + return struct.pack(">I", combined_length) + crc_input + struct.pack(">I", crc) + + +def footer_json_for_blob(blob: bytes, properties: dict[str, str] | None = None) -> bytes: + payload = { + "blobs": [ + { + "type": "deletion-vector-v1", + "fields": [], + "snapshot-id": -1, + "sequence-number": -1, + "offset": 4, + "length": len(blob), + "properties": properties if properties is not None else default_dv_properties(), + } + ] + } + return json.dumps(payload, separators=(", ", ": ")).encode("utf-8") + + +def build_puffin_file( + blob: bytes, + footer_json: bytes, + *, + compressed: bool = False, + lz4_declare_content_size: bool = True, +) -> bytes: + return build_puffin_file_from_blobs( + [blob], + footer_json, + compressed=compressed, + lz4_declare_content_size=lz4_declare_content_size, + ) + + +def build_puffin_file_from_blobs( + blobs: list[bytes], + footer_json: bytes, + *, + compressed: bool = False, + lz4_declare_content_size: bool = True, +) -> bytes: + if compressed: + # Puffin requires a single LZ4 frame; Content Size must be present for valid files. + footer_payload = lz4.frame.compress(footer_json, store_size=lz4_declare_content_size) + flags = bytes([0x01, 0x00, 0x00, 0x00]) + else: + footer_payload = footer_json + flags = b"\x00\x00\x00\x00" + + footer_length = struct.pack(" tuple[bytes, bytes]: + blob_end = puffin.index(PUFFIN_MAGIC, 4) + blob = puffin[4:blob_end] + footer_len = struct.unpack(" int: + return (xxhash.xxh32(descriptor, seed=0).intdigest() >> 8) & 0xFF + + +def set_lz4_content_size(compressed: bytes, content_size: int) -> bytes: + """Replace the Content Size field in an LZ4 frame that already declares it. + + FLG bit 0x08 is Content Size; the previous implementation wrongly used 0x10 + (Block Checksum), which produced frames that fail decompression. + """ + data = bytearray(compressed) + if len(data) < 15: + raise ValueError("LZ4 frame is too short to contain a Content Size field") + if (data[4] & 0x08) == 0: + raise ValueError("LZ4 frame does not declare Content Size") + struct.pack_into(" bytes: + return set_lz4_content_size(compressed, fake_size) + + +def write_fixture(name: str, content: bytes) -> None: + path = OUTPUT_DIR / name + path.write_bytes(content) + print(f"Wrote {path} ({len(content)} bytes)") + + +def generate_compressed_footer(source: Path) -> None: + puffin = source.read_bytes() + blob, footer_json = extract_blob_and_footer_json(puffin) + write_fixture("compressed_footer.puffin", build_puffin_file(blob, footer_json, compressed=True)) + + +def generate_invalid_blob_bounds() -> None: + cases = { + "overflow_offset_length.puffin": (9223372036854775797, 20), + "negative_offset.puffin": (-1, 10), + "length_exceeds_file.puffin": (4, 10_000), + # Offset/length stay within the total file size, but the blob would extend past the + # end of the blob region (i.e. into the footer payload), which must also be rejected. + "blob_overlaps_footer.puffin": (4, len(BLOB_PLACEHOLDER) + 8), + } + for name, (offset, length) in cases.items(): + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + payload = json.loads(footer_json.decode("utf-8")) + payload["blobs"][0]["offset"] = offset + payload["blobs"][0]["length"] = length + write_fixture(name, build_puffin_file(BLOB_PLACEHOLDER, json.dumps(payload, separators=(", ", ": ")).encode("utf-8"))) + + +def generate_invalid_roaring_bitmap() -> None: + vector = struct.pack(" None: + vector = struct.pack(" None: + """Corrupt deletion-vector-v1 framing so extractDeletionVectorPayload cannot silently regress.""" + bitmap = pyroaring.BitMap([2, 5]) + vector = struct.pack("I", len(DELETION_VECTOR_MAGIC) + len(vector) + 1) + write_fixture( + "dv_envelope_length_mismatch.puffin", + build_puffin_file(bytes(length_mismatch), footer_json_for_blob(bytes(length_mismatch), properties)), + ) + + bad_magic = bytearray(base) + bad_magic[4:8] = b"XXXX" + write_fixture( + "dv_envelope_bad_magic.puffin", + build_puffin_file(bytes(bad_magic), footer_json_for_blob(bytes(bad_magic), properties)), + ) + + crc_mismatch = bytearray(base) + crc_mismatch[-1] ^= 0xFF + write_fixture( + "dv_envelope_crc_mismatch.puffin", + build_puffin_file(bytes(crc_mismatch), footer_json_for_blob(bytes(crc_mismatch), properties)), + ) + + +def generate_inflated_lz4_content_size(source: Path) -> None: + puffin = source.read_bytes() + blob, footer_json = extract_blob_and_footer_json(puffin) + footer_payload = set_lz4_content_size(lz4.frame.compress(footer_json, store_size=True), INFLATED_CONTENT_SIZE) + footer_length = struct.pack(" None: + """contentSize passes size*255 but exceeds the absolute decompressed-footer cap. + + Pads the compressed footer payload so the ratio guard would allow the forged size; + the parser must reject on the absolute ceiling before allocating. + """ + puffin = source.read_bytes() + blob, footer_json = extract_blob_and_footer_json(puffin) + forged_content_size = FOOTER_MAX_PAYLOAD_SIZE + 1 + min_payload_for_ratio = (forged_content_size + FOOTER_LZ4_MAX_RATIO - 1) // FOOTER_LZ4_MAX_RATIO + footer_payload = bytearray( + set_lz4_content_size(lz4.frame.compress(footer_json, store_size=True), forged_content_size) + ) + if len(footer_payload) < min_payload_for_ratio: + footer_payload.extend(b"\x00" * (min_payload_for_ratio - len(footer_payload))) + footer_length = struct.pack(" None: + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + write_fixture( + "missing_lz4_content_size.puffin", + build_puffin_file( + BLOB_PLACEHOLDER, + footer_json, + compressed=True, + lz4_declare_content_size=False, + ), + ) + + +def generate_lz4_trailing_bytes() -> None: + """Valid single LZ4 frame plus trailing garbage; FooterPayloadSize includes the junk.""" + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + footer_payload = lz4.frame.compress(footer_json, store_size=True) + b"GARBAGE" + footer_length = struct.pack(" None: + """Truncated LZ4 frame: header parses with content size, body is incomplete (must not hang).""" + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + full_frame = lz4.frame.compress(footer_json, store_size=True) + # Keep enough bytes for LZ4F_getFrameInfo to succeed, but truncate the body. + footer_payload = full_frame[:19] + footer_length = struct.pack(" None: + # One missing blob field covers requireBlobMetadataField; keep distinct blobs / + # properties / DV-property cases that hit different branches. + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + payload = json.loads(footer_json.decode("utf-8")) + del payload["blobs"][0]["type"] + write_fixture( + "missing_type.puffin", + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + footer_field_cases = { + "missing_blobs.puffin": {}, + "null_blobs.puffin": {"blobs": None}, + "null_blob_entry.puffin": {"blobs": [None]}, + "invalid_blob_entry.puffin": {"blobs": ["not-an-object"]}, + } + for name, footer_payload in footer_field_cases.items(): + write_fixture( + name, + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(footer_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + dv_property_cases = { + "missing_properties.puffin": None, + "null_properties.puffin": "null", + "missing_referenced_data_file.puffin": {"cardinality": "0"}, + "missing_cardinality.puffin": {"referenced-data-file": DEFAULT_REFERENCED_DATA_FILE}, + # Array covers the non-object branch; string would be the same check. + "invalid_properties_array.puffin": [], + } + for name, properties in dv_property_cases.items(): + case_payload = json.loads(footer_json.decode("utf-8")) + if properties is None: + del case_payload["blobs"][0]["properties"] + elif properties == "null": + case_payload["blobs"][0]["properties"] = None + else: + case_payload["blobs"][0]["properties"] = properties + write_fixture( + name, + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(case_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + # Present compression-codec (including null) must be rejected for DV; one fixture is enough. + case_payload = json.loads(footer_json.decode("utf-8")) + case_payload["blobs"][0]["compression-codec"] = "lz4" + write_fixture( + "dv_with_compression_codec.puffin", + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(case_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + +def generate_invalid_property_value_types() -> None: + """Property maps must have string values; non-strings must fail with BAD_ARGUMENTS.""" + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + base = json.loads(footer_json.decode("utf-8")) + # One extra-key non-string covers the shared !isString check; also require a wrong-typed + # required DV key so that path stays covered. + cases = { + "invalid_property_number.puffin": {**default_dv_properties(), "ndv": 5}, + "invalid_property_cardinality_number.puffin": { + "referenced-data-file": DEFAULT_REFERENCED_DATA_FILE, + "cardinality": 5, + }, + } + for name, properties in cases.items(): + case_payload = json.loads(json.dumps(base)) + case_payload["blobs"][0]["properties"] = properties + write_fixture( + name, + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(case_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + +def generate_invalid_integer_fields() -> None: + """BlobMetadata integer fields must be JSON integers, not floats, strings, or booleans. + + Poco reports JSON booleans as integers (`std::numeric_limits::is_integer`), so + boolean cases must be rejected explicitly in the reader. Keep one scalar and one fields[] + example per JSON type / range branch rather than one file per field name. + """ + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + base = json.loads(footer_json.decode("utf-8")) + cases = { + "float_offset.puffin": ("offset", 5.1), + "float_fields_element.puffin": ("fields", [1.9]), + "string_offset.puffin": ("offset", "4"), + "bool_offset.puffin": ("offset", True), + "bool_fields_element.puffin": ("fields", [True]), + "fields_element_out_of_int32_range.puffin": ("fields", [2**40]), + # Poco stores integers that fail signed Int64 parse as UInt64 (2**63 wraps in tryParse64). + "offset_out_of_int64_range.puffin": ("offset", 2**64 - 1), + "fields_element_out_of_int64_range.puffin": ("fields", [2**64 - 1]), + } + for name, (field, value) in cases.items(): + case_payload = json.loads(json.dumps(base)) + case_payload["blobs"][0][field] = value + write_fixture( + name, + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(case_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + +def generate_invalid_string_fields() -> None: + """BlobMetadata type / compression-codec must be JSON strings.""" + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + base = json.loads(footer_json.decode("utf-8")) + + case_payload = json.loads(json.dumps(base)) + case_payload["blobs"][0]["type"] = 123 + write_fixture( + "type_number.puffin", + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(case_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + # Non-DV blob so DV omit-codec logic does not hide the type error. + # Number covers !isString; null covers present-null for the optional codec field. + theta_cases = { + "compression_codec_number.puffin": 1, + "compression_codec_null.puffin": None, + } + for name, codec_value in theta_cases.items(): + case_payload = { + "blobs": [ + { + "type": "apache-datasketches-theta-v1", + "fields": [], + "snapshot-id": -1, + "sequence-number": -1, + "offset": 4, + "length": len(BLOB_PLACEHOLDER), + "compression-codec": codec_value, + } + ] + } + write_fixture( + name, + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(case_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + +def generate_cardinality_mismatch_large_bitmap() -> None: + bitmap = pyroaring.BitMap([2, 5, 7, 100, 65536]) + vector = struct.pack(" None: + """RLE-dense roaring: tiny on-disk blob, 100k consecutive positions — must remain readable.""" + bitmap = pyroaring.BitMap() + bitmap.add_range(0, 100_000) + vector = struct.pack(" None: + """Declared cardinality above the absolute materialization ceiling must be rejected early.""" + bitmap = pyroaring.BitMap([0]) + vector = struct.pack(" None: + """cardinality must parse as UInt64; invalid strings must fail with BAD_ARGUMENTS.""" + bitmap = pyroaring.BitMap([2, 5]) + vector = struct.pack(" None: + """Iceberg requires deletion-vector-v1 snapshot-id and sequence-number to be -1.""" + footer_json = footer_json_for_blob(BLOB_PLACEHOLDER) + base = json.loads(footer_json.decode("utf-8")) + # Shared check rejects either field; one nonzero fixture is enough. + case_payload = json.loads(json.dumps(base)) + case_payload["blobs"][0]["snapshot-id"] = 1 + write_fixture( + "dv_nonzero_snapshot_id.puffin", + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(case_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + +def generate_sparse_large_key() -> None: + bitmap = pyroaring.BitMap() + bitmap.add(SPARSE_SUB_POSITION) + vector = struct.pack(" None: + theta_blob = b"\x00" * 16 + bitmap = pyroaring.BitMap([2, 5]) + vector = struct.pack(" None: + theta_blob = b"\x00" * 16 + bitmap = pyroaring.BitMap([2, 5]) + vector = struct.pack(" None: + flags = b"\x00\x00\x00\x00" + footer_length = struct.pack(" None: + """FileMetadata must be a JSON object. + + Poco's RFC 4627 parser accepts array as a top-level value, so this exercises the + Object::Ptr guard. String/number roots fail earlier inside the parser itself. + """ + write_raw_footer_fixture("footer_root_array.puffin", b"[1, 2, 3]") + + +def generate_missing_footer_leading_magic() -> None: + """Footer must start with its own Magic; header Magic must not double as footer-open Magic. + + Layout Magic | FooterPayload | Size | Flags | Magic is invalid even when blobs is empty. + """ + footer_payload = b'{"blobs": []}' + flags = b"\x00\x00\x00\x00" + footer_length = struct.pack(" None: + """Optional FileMetadata.properties must be a JSON object with string values.""" + bitmap = pyroaring.BitMap([2, 5]) + vector = struct.pack(" None: + """Malformed JSON / oversize integers must fail with BAD_ARGUMENTS, not STD_EXCEPTION.""" + write_raw_footer_fixture("malformed_footer_json.puffin", b"{") + # Larger than UInt64::max; Poco NumberParser throws SyntaxException. + write_raw_footer_fixture( + "footer_integer_overflow.puffin", + b'{"blobs": [{"type": "apache-datasketches-theta-v1", "fields": [], ' + b'"snapshot-id": 99999999999999999999999999999999, "sequence-number": 1, ' + b'"offset": 4, "length": 4, "properties": {}}]}', + ) + + +def main() -> None: + spark_fixture = OUTPUT_DIR / "spark_deletion_vector.puffin" + if not spark_fixture.exists(): + raise SystemExit(f"Missing {spark_fixture}; run generate_spark_puffin.py first") + + generate_compressed_footer(spark_fixture) + generate_invalid_blob_bounds() + generate_invalid_roaring_bitmap() + generate_invalid_bitmap_key() + generate_invalid_dv_envelope() + generate_inflated_lz4_content_size(spark_fixture) + generate_lz4_content_size_within_ratio_over_absolute_cap(spark_fixture) + generate_missing_lz4_content_size() + generate_lz4_trailing_bytes() + generate_incomplete_lz4_footer() + generate_missing_required_fields() + generate_invalid_property_value_types() + generate_invalid_integer_fields() + generate_invalid_string_fields() + generate_invalid_footer_root() + generate_missing_footer_leading_magic() + generate_invalid_file_metadata_properties() + generate_unparseable_footer_json() + generate_mixed_blob_types() + generate_invalid_non_dv_properties() + + generate_cardinality_mismatch_large_bitmap() + generate_dense_range_100k() + generate_cardinality_exceeds_materialization_limit() + generate_invalid_cardinality_strings() + generate_invalid_dv_snapshot_sequence() + generate_sparse_large_key() + + +if __name__ == "__main__": + main() diff --git a/tests/queries/0_stateless/data_puffin/generate_spark_puffin.py b/tests/queries/0_stateless/data_puffin/generate_spark_puffin.py new file mode 100644 index 000000000000..375b15b9b9be --- /dev/null +++ b/tests/queries/0_stateless/data_puffin/generate_spark_puffin.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +"""Generate a Puffin deletion-vector file using Apache Spark + Iceberg v3. + +Requirements: + - Java 11+ + - pyspark + - Network access on first run (downloads iceberg-spark-runtime) + +Usage: + python3 generate_spark_puffin.py + python3 generate_puffin_fixtures.py +""" + +from __future__ import annotations + +import shutil +import tempfile +from pathlib import Path + +try: + from pyspark.sql import SparkSession +except ImportError as exc: # pragma: no cover - helper script + raise SystemExit("pyspark is required") from exc + +OUTPUT = Path(__file__).with_name("spark_deletion_vector.puffin") +ICEBERG_PACKAGE = "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0" +DELETED_IDS = [2, 5, 7, 100, 65536] + + +def find_puffin_file(warehouse: Path) -> Path: + puffin_files = sorted(warehouse.rglob("*-deletes.puffin")) + if len(puffin_files) != 1: + raise RuntimeError(f"Expected exactly one puffin file, found {len(puffin_files)}: {puffin_files}") + return puffin_files[0] + + +def build_spark_session(warehouse: Path) -> SparkSession: + return ( + SparkSession.builder.appName("generate_spark_puffin") + .master("local[1]") + .config("spark.jars.packages", ICEBERG_PACKAGE) + .config( + "spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + ) + .config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog") + .config("spark.sql.catalog.spark_catalog.type", "hadoop") + .config("spark.sql.catalog.spark_catalog.warehouse", str(warehouse)) + .config("spark.ui.enabled", "false") + .getOrCreate() + ) + + +def generate_puffin(warehouse: Path) -> Path: + spark = build_spark_session(warehouse) + try: + spark.sql( + """ + CREATE TABLE default.spark_puffin_source (id BIGINT) + USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql("INSERT INTO default.spark_puffin_source SELECT id FROM range(0, 70000)") + spark.sql( + "DELETE FROM default.spark_puffin_source " + f"WHERE id IN ({', '.join(str(x) for x in DELETED_IDS)})" + ) + finally: + spark.stop() + + return find_puffin_file(warehouse) + + +def main() -> None: + with tempfile.TemporaryDirectory(prefix="clickhouse_spark_puffin_") as tmp: + warehouse = Path(tmp) + puffin_path = generate_puffin(warehouse) + shutil.copyfile(puffin_path, OUTPUT) + print(f"Wrote {OUTPUT} ({OUTPUT.stat().st_size} bytes) from {puffin_path}") + + +if __name__ == "__main__": + main() diff --git a/tests/queries/0_stateless/data_puffin/incomplete_lz4_footer.puffin b/tests/queries/0_stateless/data_puffin/incomplete_lz4_footer.puffin new file mode 100644 index 000000000000..2e45030e78b6 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/incomplete_lz4_footer.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/inflated_lz4_content_size.puffin b/tests/queries/0_stateless/data_puffin/inflated_lz4_content_size.puffin new file mode 100644 index 000000000000..27f2545a8ff9 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/inflated_lz4_content_size.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_bitmap_key.puffin b/tests/queries/0_stateless/data_puffin/invalid_bitmap_key.puffin new file mode 100644 index 000000000000..2937b0d8529f Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_bitmap_key.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_blob_entry.puffin b/tests/queries/0_stateless/data_puffin/invalid_blob_entry.puffin new file mode 100644 index 000000000000..d5d6ee863001 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_blob_entry.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_cardinality_negative.puffin b/tests/queries/0_stateless/data_puffin/invalid_cardinality_negative.puffin new file mode 100644 index 000000000000..7b7e3bb2dbee Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_cardinality_negative.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_cardinality_non_numeric.puffin b/tests/queries/0_stateless/data_puffin/invalid_cardinality_non_numeric.puffin new file mode 100644 index 000000000000..4e784be4e8e9 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_cardinality_non_numeric.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_file_properties_array.puffin b/tests/queries/0_stateless/data_puffin/invalid_file_properties_array.puffin new file mode 100644 index 000000000000..aa959c4e57f9 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_file_properties_array.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_file_property_number.puffin b/tests/queries/0_stateless/data_puffin/invalid_file_property_number.puffin new file mode 100644 index 000000000000..90fe9010006c Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_file_property_number.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_non_dv_properties_array.puffin b/tests/queries/0_stateless/data_puffin/invalid_non_dv_properties_array.puffin new file mode 100644 index 000000000000..0e570e10a1a8 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_non_dv_properties_array.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_properties_array.puffin b/tests/queries/0_stateless/data_puffin/invalid_properties_array.puffin new file mode 100644 index 000000000000..3768217c0996 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_properties_array.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_property_cardinality_number.puffin b/tests/queries/0_stateless/data_puffin/invalid_property_cardinality_number.puffin new file mode 100644 index 000000000000..e9dedbbd75cc Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_property_cardinality_number.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_property_number.puffin b/tests/queries/0_stateless/data_puffin/invalid_property_number.puffin new file mode 100644 index 000000000000..e61998faf095 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_property_number.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/invalid_roaring_bitmap.puffin b/tests/queries/0_stateless/data_puffin/invalid_roaring_bitmap.puffin new file mode 100644 index 000000000000..fb818b09610c Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/invalid_roaring_bitmap.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/length_exceeds_file.puffin b/tests/queries/0_stateless/data_puffin/length_exceeds_file.puffin new file mode 100644 index 000000000000..f06f02e694d1 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/length_exceeds_file.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/lz4_content_size_over_absolute_cap.puffin b/tests/queries/0_stateless/data_puffin/lz4_content_size_over_absolute_cap.puffin new file mode 100644 index 000000000000..4cebadaaa633 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/lz4_content_size_over_absolute_cap.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/lz4_trailing_bytes.puffin b/tests/queries/0_stateless/data_puffin/lz4_trailing_bytes.puffin new file mode 100644 index 000000000000..18763d94df2d Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/lz4_trailing_bytes.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/malformed_footer_json.puffin b/tests/queries/0_stateless/data_puffin/malformed_footer_json.puffin new file mode 100644 index 000000000000..cace309ac9f5 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/malformed_footer_json.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/missing_blobs.puffin b/tests/queries/0_stateless/data_puffin/missing_blobs.puffin new file mode 100644 index 000000000000..dc1a9e6dd3ff Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/missing_blobs.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/missing_cardinality.puffin b/tests/queries/0_stateless/data_puffin/missing_cardinality.puffin new file mode 100644 index 000000000000..b3e85363ddb6 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/missing_cardinality.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/missing_footer_leading_magic.puffin b/tests/queries/0_stateless/data_puffin/missing_footer_leading_magic.puffin new file mode 100644 index 000000000000..59370e33b82e Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/missing_footer_leading_magic.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/missing_lz4_content_size.puffin b/tests/queries/0_stateless/data_puffin/missing_lz4_content_size.puffin new file mode 100644 index 000000000000..ab820d3be7ca Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/missing_lz4_content_size.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/missing_properties.puffin b/tests/queries/0_stateless/data_puffin/missing_properties.puffin new file mode 100644 index 000000000000..e1308a29a6ce Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/missing_properties.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/missing_referenced_data_file.puffin b/tests/queries/0_stateless/data_puffin/missing_referenced_data_file.puffin new file mode 100644 index 000000000000..c3fb17427250 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/missing_referenced_data_file.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/missing_type.puffin b/tests/queries/0_stateless/data_puffin/missing_type.puffin new file mode 100644 index 000000000000..d3a741b6e0ad Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/missing_type.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/mixed_blob_types.puffin b/tests/queries/0_stateless/data_puffin/mixed_blob_types.puffin new file mode 100644 index 000000000000..2dbbdd9dcd2c Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/mixed_blob_types.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/negative_offset.puffin b/tests/queries/0_stateless/data_puffin/negative_offset.puffin new file mode 100644 index 000000000000..94a381891579 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/negative_offset.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/null_blob_entry.puffin b/tests/queries/0_stateless/data_puffin/null_blob_entry.puffin new file mode 100644 index 000000000000..4382f188975f Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/null_blob_entry.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/null_blobs.puffin b/tests/queries/0_stateless/data_puffin/null_blobs.puffin new file mode 100644 index 000000000000..96175e04ed7b Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/null_blobs.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/null_file_properties.puffin b/tests/queries/0_stateless/data_puffin/null_file_properties.puffin new file mode 100644 index 000000000000..6bc40e32cf85 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/null_file_properties.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/null_non_dv_properties.puffin b/tests/queries/0_stateless/data_puffin/null_non_dv_properties.puffin new file mode 100644 index 000000000000..76ba8edb5c21 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/null_non_dv_properties.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/null_properties.puffin b/tests/queries/0_stateless/data_puffin/null_properties.puffin new file mode 100644 index 000000000000..de1afa0c0426 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/null_properties.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/offset_out_of_int64_range.puffin b/tests/queries/0_stateless/data_puffin/offset_out_of_int64_range.puffin new file mode 100644 index 000000000000..a37fb4825108 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/offset_out_of_int64_range.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/overflow_offset_length.puffin b/tests/queries/0_stateless/data_puffin/overflow_offset_length.puffin new file mode 100644 index 000000000000..049f06493b63 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/overflow_offset_length.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/puffin_errors_common.sh b/tests/queries/0_stateless/data_puffin/puffin_errors_common.sh new file mode 100644 index 000000000000..1d6ab16eaadb --- /dev/null +++ b/tests/queries/0_stateless/data_puffin/puffin_errors_common.sh @@ -0,0 +1,87 @@ +# Shared parallel harness for Puffin error suites (sourced by 04257 / 04612 / 04613). +# Caller must set CURDIR and source shell_config.sh first. + +DATA="$CURDIR/data_puffin" +PUFFIN="$DATA/spark_deletion_vector.puffin" +PARALLEL="${PUFFIN_ERRORS_PARALLEL:-4}" +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +pids=() +id=0 + +wait_one() +{ + [[ ${#pids[@]} -eq 0 ]] && return + wait "${pids[0]}" || true + pids=("${pids[@]:1}") +} + +run_case() +{ + local case_id="$1" + local kind="$2" + local path_or_label="$3" + local needle="$4" + local extra="${5:-}" + local out="$TMP/$case_id.out" + local err + + { + case "$kind" in + puffin) + echo "--- $(basename "$path_or_label") ---" + err=$($CLICKHOUSE_LOCAL -q "SELECT deleted_rows FROM file('$path_or_label', Puffin)" 2>&1) || true + echo "$err" | grep -oF "$needle" || true + if [[ -n "$extra" ]]; then + echo "$err" | grep -oF "$extra" || true + fi + ;; + raw_puffin) + echo "--- $path_or_label ---" + err=$($CLICKHOUSE_LOCAL -q "$extra" 2>&1) || true + echo "$err" | grep -oF "$needle" || true + ;; + meta) + echo "--- $(basename "$path_or_label") ---" + $CLICKHOUSE_LOCAL -q "SELECT blob_type FROM file('$path_or_label', PuffinMetadata)" 2>&1 \ + | grep -oF "$needle" || true + ;; + meta_re) + echo "--- $(basename "$path_or_label") ---" + $CLICKHOUSE_LOCAL -q "SELECT blob_type FROM file('$path_or_label', PuffinMetadata)" 2>&1 \ + | grep -oE "$needle" || true + ;; + raw_meta) + echo "--- $path_or_label ---" + err=$($CLICKHOUSE_LOCAL -q "$extra" 2>&1) || true + echo "$err" | grep -oF "$needle" || true + ;; + *) + echo "unknown puffin error case kind: $kind" >&2 + return 1 + ;; + esac + } > "$out" +} + +launch() +{ + run_case "$@" & + pids+=($!) + while [[ ${#pids[@]} -ge $PARALLEL ]]; do + wait_one + done +} + +finish_puffin_errors() +{ + while [[ ${#pids[@]} -gt 0 ]]; do + wait_one + done + + local i + for ((i = 0; i < id; i++)); do + cat "$TMP/$i.out" + done +} diff --git a/tests/queries/0_stateless/data_puffin/spark_deletion_vector.puffin b/tests/queries/0_stateless/data_puffin/spark_deletion_vector.puffin new file mode 100644 index 000000000000..4a0d3e8a7e10 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/spark_deletion_vector.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/sparse_large_key.puffin b/tests/queries/0_stateless/data_puffin/sparse_large_key.puffin new file mode 100644 index 000000000000..e927acc5628d Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/sparse_large_key.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/string_offset.puffin b/tests/queries/0_stateless/data_puffin/string_offset.puffin new file mode 100644 index 000000000000..704412f86fa0 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/string_offset.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/type_number.puffin b/tests/queries/0_stateless/data_puffin/type_number.puffin new file mode 100644 index 000000000000..6d279cd6938d Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/type_number.puffin differ