[rntuple] Keep and verify page checksums - #128
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #128 +/- ##
==========================================
+ Coverage 92.40% 92.51% +0.11%
==========================================
Files 37 37
Lines 2975 2994 +19
==========================================
+ Hits 2749 2770 +21
+ Misses 226 224 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
6229a65 to
09a6bb0
Compare
A page whose fNElements is negative has an XXH3-64 checksum stored little-endian right after it, over the stored (sealed) bytes. The locator's size does not include it (spec, Page Locations; ROOT's reader adds the eight bytes back itself, RPageStorage.cxx:297). rootfilespec neither read nor checked it. - RPageDescription keeps fNElements as stored (the sign is the flag, and the stored value is what reproduces the bytes) and gains n_elements, has_checksum and stored_size. size stays the locator's size, as the spec defines it. - RPageDescription.page_locator is a new RPageLocator covering the stored bytes (offset, stored_size). Its read_from verifies the checksum and returns an RPage with the raw bytes and the checksum; nothing is decompressed. PageListEnvelope.page_locators returns these, and get_page uses it. Fixes nsmith-#55 Assisted-by: claude-code:claude-opus-5-5
The pre-commit mypy hook installs only pytest, numpy and tomli, so it cannot find xxhash. bootstrap/compression.py already imports it the same way. Assisted-by: claude-code:claude-opus-5-5
Name the module after what it tests, so it does not read as the home of a broad set of tests (review on nsmith-#125). Assisted-by: claude-code:claude-opus-5-5
09a6bb0 to
13224db
Compare
RPageDescription already is the page's locator. Instead of a second locator class next to it, it now fetches and verifies the checksum itself, as suggested on nsmith-#55: - size is the number of bytes to fetch: locator.size, plus the 8-byte checksum when fNElements is negative. The spec's page size, which excludes the checksum (Page Locations), stays in locator.size. - read_from splits off the trailer, verifies XXH3-64 over the stored bytes and returns RPage(page, checksum). get_page goes through it. RPageLocator flattened the RLocator into an in-file (offset, size) and left RPageDescription as a second, unverified way to read a page. Keeping the RLocator leaves room for further locator types (the spec reserves 0x02-0x7f), and the checksum check depends only on the fetched bytes. PageListEnvelope.page_locators is unchanged from main, so this PR no longer changes any API. Assisted-by: claude-code:claude-opus-5-5
|
@nsmith- i reviewed this and think it is ready to merge. this looks a lot cleaner than before |
nsmith-
left a comment
There was a problem hiding this comment.
Ok, and the RPageLocator will follow the Locator protocol (and RPageDescription will gain a page_locator or similar method) once we return to #114 for standard locators. For non-standard, the RPageLocator will have some other shape.
…mith-#128 nsmith-#128's tests index key.fClassName.fString; after nsmith-#68 the class name is plain bytes. Assisted-by: claude-code:claude-opus-5-5
A cluster committed before the model was extended lists only the columns that existed then, so its page list can end before the schema extension's columns do (the spec fixes the columns' order, not their number). ROOT fills those in when reading (AddExtendedColumnRanges, RNTupleDescriptor.cxx:920 at 6.40.04). The strict zip raised on scikit-hep-testdata's test_extension_columns_rntuple_v1-0-0-0.root, which read before this PR: its first cluster lists 2 of 4 columns. get_extended_page_descriptions now gives such trailing columns empty entries, so a position is still the column ID. A cluster listing more columns than the schema, or fewer than the header's, still raises. uncompressedSize uses RPageDescription.n_elements (nsmith-#128). Tests: that file's clusters, a page list missing a header column, and the per-page column/field check on three scikit-hep-testdata files as well as root-io-spec's. Assisted-by: claude-code:claude-opus-5-5
Fixes #55. Its tests use
rntuple/map.rootandrntuple/compressed.root, from #124 (merged).A page whose
fNElementsis negative has an XXH3-64 checksum right after it: 8 bytes, little-endian, over the stored (sealed, possibly compressed) bytes. The locator's size excludes it; ROOT's reader adds the 8 bytes back itself (RPageStorage.cxx:297, root-io-spec NOTES 4).RPageDescriptionstays the page's locator, as onmain, and now covers the checksum:fNElementsstays as stored. Its sign is the checksum flag, and the stored value is what lets bytes be reproduced.n_elements(absolute value) andhas_checksum(fNElements < 0).sizeis the number of bytes to fetch:locator.size, plus 8 when the page has a checksum (as suggested earlier in [rntuple] Store page checksums #55). The spec's page size, which excludes the checksum, stays inlocator.size.read_fromsplits off the trailer, verifies the XXH3-64 and returns anRPage. A buffer of the wrong length raises.get_pagegoes through it, so every existing path verifies.The checksum check depends only on the fetched bytes, and the
RLocatoris kept as is, so nothing here assumes how or from where a page is fetched. That leaves room for the locator types the spec reserves (0x02–0x7f).RPagegainschecksum: int | None. The page bytes are still the stored bytes; nothing is decompressed.PageListEnvelope.page_locatorsis unchanged. It returnsRPageDescriptions, sotest_read.py's walk now verifies every checksum.Tests (
tests/test_page_checksums.py):rntuple/anchor.root, the page at 550:fNElements = -3,locator.size12,size20, checksumde3ce2c4a5a407beat 562, equal to XXH3-64 of bytes 550–562;Page checksum mismatch at StandardLocator(size=12, offset=550); a buffer of the wrong length raises; a page without a checksum reads withchecksum=None;compressed.root's compressed pages andmap.root's shared ranges (page descriptions that name the same bytes, which is common: same-page merging is on by default, root-io-spec NOTES 7).Full suite, on current
main(after #130):471 passed, 61 skipped, 50 xfailed.Behaviour change:
RPageDescription.sizeincludes the 8-byte checksum when there is one, andread_fromexpects those bytes; uselocator.sizefor the page's own size.Assisted-by: claude-code:claude-opus-5-5