Skip to content

[rntuple] Keep and verify page checksums - #128

Merged
sathabbott merged 4 commits into
nsmith-:mainfrom
sathabbott:fix/55-page-checksums
Oct 1, 2026
Merged

sathabbott merged 4 commits into
nsmith-:mainfrom
sathabbott:fix/55-page-checksums

Conversation

@sathabbott

@sathabbott sathabbott commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 AI generated content

Fixes #55. Its tests use rntuple/map.root and rntuple/compressed.root, from #124 (merged).

A page whose fNElements is negative has an XXH3-64 checksum right after it: 8 bytes, little-endian, over the stored (sealed, possibly compressed) bytes. The locator's size excludes it; ROOT's reader adds the 8 bytes back itself (RPageStorage.cxx:297, root-io-spec NOTES 4).

RPageDescription stays the page's locator, as on main, and now covers the checksum:

  • fNElements stays as stored. Its sign is the checksum flag, and the stored value is what lets bytes be reproduced.
  • New: n_elements (absolute value) and has_checksum (fNElements < 0).
  • size is the number of bytes to fetch: locator.size, plus 8 when the page has a checksum (as suggested earlier in [rntuple] Store page checksums #55). The spec's page size, which excludes the checksum, stays in locator.size.
  • read_from splits off the trailer, verifies the XXH3-64 and returns an RPage. A buffer of the wrong length raises. get_page goes through it, so every existing path verifies.

The checksum check depends only on the fetched bytes, and the RLocator is kept as is, so nothing here assumes how or from where a page is fetched. That leaves room for the locator types the spec reserves (0x02–0x7f).

RPage gains checksum: int | None. The page bytes are still the stored bytes; nothing is decompressed.

PageListEnvelope.page_locators is unchanged. It returns RPageDescriptions, so test_read.py's walk now verifies every checksum.

Tests (tests/test_page_checksums.py):

  • rntuple/anchor.root, the page at 550: fNElements = -3, locator.size 12, size 20, checksum de3ce2c4a5a407be at 562, equal to XXH3-64 of bytes 550–562;
  • a flipped byte raises Page checksum mismatch at StandardLocator(size=12, offset=550); a buffer of the wrong length raises; a page without a checksum reads with checksum=None;
  • every page of all 11 root-io-spec RNTuple fixtures verifies, including compressed.root's compressed pages and map.root's shared ranges (page descriptions that name the same bytes, which is common: same-page merging is on by default, root-io-spec NOTES 7).

Full suite, on current main (after #130): 471 passed, 61 skipped, 50 xfailed.

Behaviour change: RPageDescription.size includes the 8-byte checksum when there is one, and read_from expects those bytes; use locator.size for the page's own size.

Assisted-by: claude-code:claude-opus-5-5

@codecov-commenter

codecov-commenter commented Sep 24, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.51%. Comparing base (1f9f492) to head (94c6b00).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #128      +/-   ##
==========================================
+ Coverage   92.40%   92.51%   +0.11%     
==========================================
  Files          37       37              
  Lines        2975     2994      +19     
==========================================
+ Hits         2749     2770      +21     
+ Misses        226      224       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sathabbott
sathabbott force-pushed the fix/55-page-checksums branch from 6229a65 to 09a6bb0 Compare September 30, 2026 14:19
A page whose fNElements is negative has an XXH3-64 checksum stored
little-endian right after it, over the stored (sealed) bytes. The locator's
size does not include it (spec, Page Locations; ROOT's reader adds the eight
bytes back itself, RPageStorage.cxx:297). rootfilespec neither read nor
checked it.

- RPageDescription keeps fNElements as stored (the sign is the flag, and the
  stored value is what reproduces the bytes) and gains n_elements,
  has_checksum and stored_size. size stays the locator's size, as the spec
  defines it.
- RPageDescription.page_locator is a new RPageLocator covering the stored
  bytes (offset, stored_size). Its read_from verifies the checksum and
  returns an RPage with the raw bytes and the checksum; nothing is
  decompressed. PageListEnvelope.page_locators returns these, and get_page
  uses it.

Fixes nsmith-#55

Assisted-by: claude-code:claude-opus-5-5
The pre-commit mypy hook installs only pytest, numpy and tomli, so it cannot
find xxhash. bootstrap/compression.py already imports it the same way.

Assisted-by: claude-code:claude-opus-5-5
Name the module after what it tests, so it does not read as the home of a
broad set of tests (review on nsmith-#125).

Assisted-by: claude-code:claude-opus-5-5
@sathabbott
sathabbott force-pushed the fix/55-page-checksums branch from 09a6bb0 to 13224db Compare September 30, 2026 20:40
RPageDescription already is the page's locator. Instead of a second
locator class next to it, it now fetches and verifies the checksum
itself, as suggested on nsmith-#55:

- size is the number of bytes to fetch: locator.size, plus the 8-byte
  checksum when fNElements is negative. The spec's page size, which
  excludes the checksum (Page Locations), stays in locator.size.
- read_from splits off the trailer, verifies XXH3-64 over the stored
  bytes and returns RPage(page, checksum). get_page goes through it.

RPageLocator flattened the RLocator into an in-file (offset, size) and
left RPageDescription as a second, unverified way to read a page.
Keeping the RLocator leaves room for further locator types (the spec
reserves 0x02-0x7f), and the checksum check depends only on the
fetched bytes. PageListEnvelope.page_locators is unchanged from main,
so this PR no longer changes any API.

Assisted-by: claude-code:claude-opus-5-5
@sathabbott

Copy link
Copy Markdown
Collaborator Author

@nsmith- i reviewed this and think it is ready to merge. this looks a lot cleaner than before

@sathabbott
sathabbott requested a review from nsmith- September 30, 2026 22:51

@nsmith- nsmith- left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, and the RPageLocator will follow the Locator protocol (and RPageDescription will gain a page_locator or similar method) once we return to #114 for standard locators. For non-standard, the RPageLocator will have some other shape.

@sathabbott
sathabbott merged commit 79fd46d into nsmith-:main Oct 1, 2026
9 checks passed
@sathabbott
sathabbott deleted the fix/55-page-checksums branch October 1, 2026 00:18
sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 1, 2026
…mith-#128

nsmith-#128's tests index key.fClassName.fString; after nsmith-#68 the class name is
plain bytes.

Assisted-by: claude-code:claude-opus-5-5
sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 1, 2026
A cluster committed before the model was extended lists only the
columns that existed then, so its page list can end before the schema
extension's columns do (the spec fixes the columns' order, not their
number). ROOT fills those in when reading (AddExtendedColumnRanges,
RNTupleDescriptor.cxx:920 at 6.40.04). The strict zip raised on
scikit-hep-testdata's test_extension_columns_rntuple_v1-0-0-0.root,
which read before this PR: its first cluster lists 2 of 4 columns.

get_extended_page_descriptions now gives such trailing columns empty
entries, so a position is still the column ID. A cluster listing more
columns than the schema, or fewer than the header's, still raises.
uncompressedSize uses RPageDescription.n_elements (nsmith-#128).

Tests: that file's clusters, a page list missing a header column, and
the per-page column/field check on three scikit-hep-testdata files as
well as root-io-spec's.

Assisted-by: claude-code:claude-opus-5-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rntuple] Store page checksums

3 participants