[EPIC] Deletion vector read support (`deletion-vector-v1` Puffin blobs)

[mbutrovich]

Is your feature request related to a problem or challenge?

Iceberg format version 3 stores positional deletes as deletion vectors instead of position delete files. A deletion vector is a roaring bitmap of deleted row positions, written as a deletion-vector-v1 Puffin blob and referenced from a delete manifest entry by content_offset and content_size_in_bytes.

iceberg-rust does not apply them on read. CachingDeleteFileLoader::load_file_for_task branches only on DataContentType (PositionDeletes / EqualityDeletes / Data), and the PositionDeletes arm unconditionally calls parquet_to_batch_stream(...) on the file path (crates/iceberg/src/arrow/caching_delete_file_loader.rs). A V3 deletion vector arrives as PositionDeletes content but points at a Puffin file, so the read fails when the loader parses that file as Parquet. The DV branch is stubbed: DeleteFileContext carries // TODO: Delete Vector loader from Puffin files (caching_delete_file_loader.rs:54). Any V3 table with positional deletes is therefore unreadable.

This issue tracks the read path under the V3 epic #2411, scoped to land independently of the DV write path.

What already exists on main

• DataFile parses the V3 manifest fields referenced_data_file, content_offset, content_size_in_bytes (crates/iceberg/src/spec/manifest/data_file.rs).
• DeleteVector (a RoaringTreemap wrapper) and the ParsedDeleteFileContext::DelVecs loader variant exist (crates/iceberg/src/delete_vector.rs, caching_delete_file_loader.rs).
• Positional deletes already flow DeleteVector -> DeleteFilter -> ArrowReader, so DV application reuses that path unchanged.
• PuffinReader, PuffinWriter, and the DELETION_VECTOR_V1 blob-type constant exist (crates/iceberg/src/puffin/).

Describe the solution you'd like

Implement DV read as a series of small PRs that depend only on each other, not on the write path (#2678 / #2203). DV writing is out of scope here; it belongs with #2678 under #2411.

Status: all tasks below are implemented on the dv-read branch and validated end to end against Spark / Iceberg-Java-written deletion vectors via a draft DataFusion Comet PR (apache/datafusion-comet#4887), which reads real V3 merge-on-read tables through this code and confirms the deleted rows match Spark. They are now being peeled into small, independently reviewable PRs, tracked per task below. The codec PR supersedes #2414, which has been idle since review feedback.

Task list

• Task 1: DV blob codec (decode). Parse a deletion-vector-v1 blob into deleted positions. No scan or reader API changes, so it merges first. PR: feat(delete-vector): [1/N] decode deletion-vector-v1 puffin blobs #2866 (supersedes feat(delete-vector): parse deletion-vector-v1 puffin blob - read support #2414).
• Task 2: Carry DV coordinates on the scan task. Add content_offset, content_size_in_bytes, and referenced_data_file to FileScanTaskDeleteFile (crates/iceberg/src/scan/task.rs), populated from DataFile during scan planning. This matches Iceberg-Java, where a DV is a DeleteFile (content=POSITION_DELETES, format=PUFFIN) attached to the task via FileScanTask.deletes(). PR: feat(scan): [2/N] carry deletion-vector coordinates on FileScanTaskDeleteFile #2868.
• Task 3: Loader branch, apply rules, end-to-end read. In load_file_for_task, route entries with content_offset.is_some() to a DV path: read the blob at content_offset / content_size_in_bytes via PuffinReader, decode with Task 1, and return ParsedDeleteFileContext::DelVecs. Apply the spec rules below (DeleteFileIndex supersede logic as in feat(scan): apply V3 deletion vectors on read (stacked on #2678) #2681, decoupled from write). Validate that the decoded cardinality equals record_count and the blob length equals content_size_in_bytes. Add an end-to-end test reading a V3 table whose deletes are DVs. PR: feat(scan): [3/N] read and apply V3 deletion vectors #3035.

Two read-path rules from the table spec must hold. A DV applies to a data file when the file path equals the DV's referenced_data_file, the file's data sequence number is <= the DV's, and the partitions match. When a DV applies to a data file, readers ignore any position delete files that would otherwise match it, because the DV subsumes them. At most one DV exists per data file per snapshot.

References

• Iceberg table spec format/spec.md, "Deletion Vectors": semantics, at-most-one-per-data-file, DV-subsumes-position-deletes, apply-scope rules, and the referenced_data_file / content_offset / content_size_in_bytes fields (required for DVs, must match the Puffin footer).
• Iceberg Puffin spec format/puffin-spec.md: the deletion-vector-v1 blob byte layout.
• Iceberg-Java: core/.../deletes/BitmapPositionDeleteIndex.java (framing, length, magic, CRC) and core/.../deletes/RoaringPositionBitmap.java (portable 64-bit layout).

Willingness to contribute

I can contribute to this feature independently.

[mbutrovich]

I have it working end-to-end in a draft PR in Comet pointing to my feature branch (with all DV tasks complete) apache/datafusion-comet#4887. The tasks will be submitted as separate, stacked PRs.

[raghav-reglobe]

Nice — that was fast. Since you have it end-to-end already, the most useful thing we can offer is validation coverage: we run V3 tables whose DVs are written by DuckDB's iceberg extension, which today omits referenced_data_file from the manifest entry while writing it correctly in the Puffin blob. Binding DVs purely from manifest metadata silently misses those — binding via the scan task's delete entries handles both. Happy to run your stacked PRs against our DuckDB-DV fixtures once they're up, and we have CRC/framing test vectors for the blob layout if useful.

On overlap: once your read stack lands we'll rebase #2678 (DV write / RowDelta) onto it and retire the read half of #2681 — the write path staying decoupled from read should make both easier to review.

[mbutrovich]

Noting an interaction with the encryption epic, found while reviewing #2822 (Puffin encryption) for context.

Task 3's loader (implemented locally, not yet a PR) reads the deletion-vector-v1 blob via a raw range read straight off FileIO, in load_deletion_vector in caching_delete_file_loader.rs:

let blob = basic_delete_file_loader.file_io().new_input(&task.file_path)?
    .reader().await?.read(start..start + len).await?;

It doesn't check task.key_metadata on FileScanTaskDeleteFile, even though that field already exists and the equality/positional-delete branches right above it in the same function already use it to decrypt Parquet delete files. So a V3 table combining deletion vectors with table encryption won't read correctly today. It'll fail loudly, since DeleteVector::deserialize's length/CRC/magic checks catch the still-encrypted bytes, not silently return wrong deletes, but the combination is unsupported and nothing currently flags it.

#2822 validated that EncryptedInputFile/AesGcmFileRead (already on main, independent of #2822/#2701/#2918 landing) decrypt over plaintext-relative byte ranges, which is exactly the coordinate space content_offset/content_size_in_bytes already use. So the fix for Task 3 doesn't need PuffinReader::new_from_encrypted; it can decode task.key_metadata via StandardKeyMetadata::decode(...) and wrap the input in EncryptedInputFile::new(input, key_metadata) before the range read, the same pattern ManifestFile::load_manifest already uses (crates/iceberg/src/spec/manifest_list/manifest_file.rs:181-199), which #2918's manifest-decryption fix depends on.

Scope check: only Task 3 needs to change for this. Task 1 (#2866, blob codec) has no I/O, so it's unaffected. Task 2 (#2868) already copies data_file.key_metadata() onto FileScanTaskDeleteFile unconditionally, regardless of content type, so DV entries already carry the right key metadata; that PR needs no change. I'll fold the decrypt step into Task 3 when I open that PR.