fix: validate numBytes in CompressedVSizeColumnarIntsSupplier - #20174
Open
dfengliu wants to merge 1 commit into
Open
fix: validate numBytes in CompressedVSizeColumnarIntsSupplier#20174dfengliu wants to merge 1 commit into
dfengliu wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CompressedVSizeColumnarIntsSupplier.fromByteBuffer()trusts thenumBytesbyte in the serialized column header. Writers only ever emit values 1..4 (VSizeColumnarInts.getNumBytesForMax()), but the read path never verifies it: the byte directly drives the shift/mask constants (bigEndianShift,littleEndianMask) and the per-value absolute reads (buffer.getInt(bufferIndex * numBytes)).A segment carrying any other value reads back silently corrupted data. Using a serialized supplier with the
numBytesbyte tampered to 100,get(0)returns 50462976 instead of 0 andget(4)returns 117835012 instead of 4, with no error raised. WithnumBytes=0every value reads as 0. This reader backs every dictionary-encoded column, multi-value columns, and nested variant columns, so malformed storage can silently poison query results. JVM bounds checks contain reads to the decompressed buffer, so this is an integrity/robustness issue rather than a memory-safety one.Fixed the bug ...
Validate
numBytesin theCompressedVSizeColumnarIntsSupplierconstructor withPreconditions.checkArgument(range[1, Integer.BYTES]), so malformed headers fail fast with a descriptive message on both the read (fromByteBuffer) and construction (fromList) paths, before any shift/mask constants or offsets are computed.Added
testInvalidNumBytesRejectedtoCompressedVSizeColumnarIntsSupplierTest: it serializes a real supplier, tampers thenumBytesheader byte with 0, 5, 100 and -1, and asserts rejection. I verified the test fails without the fix and passes with it; the control round-trip still reads all values correctly.Release note
numBytesvalue. Druid now rejects such columns with a descriptiveIllegalArgumentExceptioninstead of returning wrong values.Key changed/added classes in this PR
CompressedVSizeColumnarIntsSupplierCompressedVSizeColumnarIntsSupplierTestThis PR has: