[format] Reuse one source stream for consecutive BlobRef copies#8725
Open
XiaoHongbo-Hope wants to merge 1 commit into
Open
[format] Reuse one source stream for consecutive BlobRef copies#8725XiaoHongbo-Hope wants to merge 1 commit into
XiaoHongbo-Hope wants to merge 1 commit into
Conversation
XiaoHongbo-Hope
marked this pull request as ready for review
July 19, 2026 13:44
XiaoHongbo-Hope
force-pushed
the
open_seek_fix
branch
from
July 21, 2026 04:24
ddb0f52 to
98e8c03
Compare
BLOB compaction opens a fresh stream (file open / HTTP GET) per BlobRef -- thousands of redundant opens when one compacted file draws from one source. Reuse one stream across references into the same source (same UriReader + URI), seeking to each offset and copying exactly descriptor.length bytes, bounded to the descriptor window so a caller never reads past the blob. Reuse state lives in an internal BlobReuseSource (paimon-common) that only ever hands back a stream bounded to a reference's descriptor; the raw UriReader never leaves the class. A stale view, a mismatched ref, an unknown length, or a BlobRef subclass are rejected; a source that can't rewind is reopened, and its cleanup errors surface instead of becoming a NULL write. Non-BlobRef / unknown-length values keep the open-per-blob path, so output bytes and CRC are unchanged, and BlobRef.equals keeps exact-class semantics. Behavior change: a source shorter than the descriptor throws EOFException instead of writing a truncated blob.
XiaoHongbo-Hope
force-pushed
the
open_seek_fix
branch
from
July 21, 2026 04:45
98e8c03 to
dd5cf33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
BLOB compaction opens a fresh stream (file open / HTTP GET) per
BlobRef — thousands of redundant opens when one compacted file draws
from one source. This reuses one stream across references into the
same source (same UriReader + URI), seeking to each offset and
copying exactly descriptor.length bytes, bounded to the descriptor
window: N references cost one open. Non-BlobRef / unknown-length
values keep the old path, so bytes and CRC are unchanged; only exact
BlobRef takes the fast path, so subclass newInputStream() and the
exact-class equals contract are preserved.
Tests
Verified on JDK 8:
BlobReuseSourceTestBlobFormatWriterTestOffsetSeekableInputStreamTestBlobFileFormatTestBlobTest