Skip to content

[CICP-5405] Fix ZkBucketDataAccessor data loss when compressed size is an exact multiple of bucket size - #320

Draft
thestreak101 wants to merge 1 commit into
devfrom
copilot/cicp-5405-zkbucket-exact-multiple
Draft

thestreak101 wants to merge 1 commit into
devfrom
copilot/cicp-5405-zkbucket-exact-multiple

Conversation

@thestreak101

Copy link
Copy Markdown
Collaborator

Problem

ZkBucketDataAccessor computed the last bucket's length with a modulo:

compressedRecord.length % _bucketSize

When the compressed payload's size is an exact multiple of the bucket size, that expression is 0, so the final bucket is written empty.

On read, the reconstructed array keeps a zeroed tail, GZipCompressionUtil.uncompress fails, and the accessor throws HelixException("Failed to decompress path").

The write has already destroyed the bytes by then, so this is not recoverable by the reader — the record is permanently unreadable. This affects any bucketized write (WAGED baseline / best-possible assignments) that happens to compress to a multiple of the bucket size.

Fix

1. Write path — derive the last bucket from the remaining bytes instead of a modulo:

Arrays.copyOfRange(compressedRecord, ptr, compressedRecord.length)

2. Read path (a second, latent bug found while fixing the first) — numBuckets was computed from the reader's _bucketSize, while the subsequent arraycopy used the writer's bucketSize read from the metadata node. Any change to the configured bucket size between write and read produced a truncated or over-read payload. Both now use the metadata bucketSize, which is the value the data was actually written with.

3. Unified the copy loop to Math.min(bucketSize, dataSize - copyPtr) so the final short bucket needs no special case.

Tests

Added to TestZkBucketDataAccessor:

Test Covers
testExactMultipleOfBucketSizeRoundTrip the reported failure
testBucketBoundarySizes sizes around N*bucketSize ± 1
testReadWithDifferentBucketSizeThanWrite bug (2) above
testSingleBucketRoundTrip small-payload path

A payload's compressed size is not predictable from its input size, so the helpers search for an input whose compressed size lands exactly on a bucket boundary rather than assuming one.

Verification

  • mvn -pl helix-core -am test-compile passes.
  • The new tests require a live ZK ensemble and were not executed locally — relying on CI.

Risk

Low. The change is confined to bucket-boundary arithmetic; the non-multiple case produces byte-identical output to before.

🤖 Drafted with GitHub Copilot CLI. Filed from an audit of CICP Helix bug tickets — please review carefully before merging.

…s an exact multiple of bucket size

The write path split the compressed payload into buckets by computing the last
bucket's length as `compressedRecord.length % _bucketSize`. When the compressed
size is an exact multiple of the bucket size that expression is 0, so the final
bucket was written EMPTY.

On read, the reconstructed byte array keeps a zeroed tail, GZipCompressionUtil
fails to inflate it, and the accessor throws
`HelixException("Failed to decompress path")`. The write has already destroyed
the bytes at that point, so this is not recoverable by the reader -- the record
is permanently unreadable.

Fixes:

1. Write path: derive the last bucket from the remaining bytes with
   `Arrays.copyOfRange(compressedRecord, ptr, compressedRecord.length)` instead
   of a modulo, so a full-size final bucket is written correctly.

2. Read path: `numBuckets` was computed from the READER's `_bucketSize`, but the
   subsequent `arraycopy` used the WRITER's `bucketSize` from the metadata
   node. Any change to the configured bucket size between write and read
   therefore produced a truncated or over-read payload. Both now use the
   metadata `bucketSize`, which is the value the data was actually written with.

3. Unified the copy loop to `Math.min(bucketSize, dataSize - copyPtr)` so the
   final short bucket is handled without a special case.

Tests added to TestZkBucketDataAccessor:
- testExactMultipleOfBucketSizeRoundTrip -- the reported failure
- testBucketBoundarySizes -- sizes around N*bucketSize +/- 1
- testReadWithDifferentBucketSizeThanWrite -- covers bug (2)
- testSingleBucketRoundTrip -- guards the small-payload path

Because a payload's COMPRESSED size is not predictable from its input size, the
test helpers search for an input whose compressed size lands on a bucket
boundary rather than assuming one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant