Read every ROOT string type as plain bytes with ROOTString (#68, #20) by sathabbott · Pull Request #127 · nsmith-/rootfilespec

sathabbott · 2026-09-24T17:11:05Z

🤖 AI generated content

Fixes #68 and #20. Every ROOT string type is read as plain bytes, as docs/design.md promises ("Annotated builtin types vs. objects"). The on-disk encoding is set by the annotation, not by the member's Python type.

ROOTString(encoding, framed=False) (rootfilespec.structutil) is a MemberSerDe: Annotated[bytes, ROOTString(...)]. encoding is a Literal with your three values, one per length format (root-io-spec Conventions §5):

`encoding`	on disk	used for
`"RNTuple"`	u32 LE length, then the bytes	RNTuple strings (`rntuple.schema.RNTupleString`)
`"TString"`	counted string: one length byte, or 255 then a u32 (§5.1)	`TString` members, `TKey` strings, `std::string` elements of collections, pointees
`"charstar"`	i32 length, then the bytes; no 255 escape, `n <= 0` reads as `b""` (§5.4)	`char*` members (`kCharStar`, #20), and `TStringLong` (§5.1.1)

framed=True is for a std::string data member, which is the TString encoding inside a byte count and version word; a std::string element of a collection has no frame (§5.3; root-io-spec serialization/collections has both: fStr framed, fWords elements bare). The frame is read with StreamHeader, as StdVector and the other framed members read theirs, and the byte count is checked. STLString used to handle that frame. The frame is a separate option rather than a fourth encoding because it isn't a length format: any encoding could be framed.

ROOTString.read(buffer) is the one string reader: the member reader calls it, and so do TKey and TList for the strings they read by hand (ROOTString("TString").read(buffer)). It lives in structutil beside Fmt, OptionalField and the other MemberSerDes; bootstrap/strings.py holds only the TString / TStringLong / string aliases.

Aliases. bootstrap.TString = Annotated[bytes, ROOTString("TString")], so hand-written classes keep fName: TString. bootstrap.TStringLong and bootstrap.string are aliases too. They stay resolvable by class name because a string can be stored as an object of its own: a key of class TString, TStringLong or string, or a TString*. serializable.read_value(type, buffer) reads a class or an annotated builtin through the same path as a member, and TKey.read_object (typed and untyped: a TypedTKey of a string record holds the alias as its objtype) and read_streamed_item use it for a looked-up type. type_by_name is still annotated as returning a class; that predates this PR (TDatime) and is filed separately as #135.

Design assumption (now in docs/design.md): the bytes don't record which encoding they came from. The annotation, the key's class name or the streamed object's class tag does. So a TString record read from a key round-trips only together with its key's fClassName; writing needs that context, not just the bytes.

Generated classes (dynamic.py) write the annotation out, as you asked:

TStreamerString → Annotated[bytes, ROOTString('TString')]
TStreamerSTLstring → Annotated[bytes, ROOTString('TString', framed=True)]
a kCharStar basic type → Annotated[bytes, ROOTString('charstar')], which was Fmt('charstar') and raised Unimplemented format charstar
in cpptype, string, std::string and TString as container elements or pointees → Annotated[bytes, ROOTString('TString')] (e.g. StdVector[Annotated[bytes, ROOTString('TString')]]), with no class dependency

A by-value member whose class is TStringLong (TStreamerObjectAny) still refers to it by name, like any other class member, which resolves to the alias. Arrays of char* (element types 27 / 47) stay unimplemented; no corpus file uses them.

Side effects on root-io-spec fixtures:

serialization/stringlong.root now reads: its SText has a TStringLong and a TString member, both bytes. Removed from EXPECTED_FAILURES.
serialization/unframed-records.root now reads its TString and TStringLong records. Its TDatime record reads too, as a side effect: read_value handles TDatime = Annotated[int, Fmt(">I")] the same way. That belongs to TDatime and TObject written as records of their own (no byte count) cannot be read #123, so its test goes with TDatime and TObject written as records of their own (no byte count) cannot be read #123, not here. The fixture now stops at the TObject record, which has no byte count (the rest of TDatime and TObject written as records of their own (no byte count) cannot be read #123); its expected failure is updated to that.

Tests (tests/test_root_string.py, renamed from test_counted_string.py):

each encoding at the byte level: TString at lengths 0, 5, 254, 255 and 300 (long form); char* with no 255 escape, empty, null and a negative length; a framed string (framed=True), an empty one (7 bytes), and a missing or wrong byte count;
root-io-spec fixtures, read at the offsets their case.toml pins: serialization/collections fWords = [b"pq", b"rs"] (bare elements) and fStr = b"abc" (framed); serialization/element-types fText = b"hi" and fNull = b"";
the generated code for those members (ROOTString('TString', framed=True), StdVector[Annotated[bytes, ROOTString('TString')]], ROOTString('charstar'));
string records (unframed-records, stringlong), fetched and through TypedTKey, and key/TNamed/streamer-info names are bytes;
RNTuple strings are bytes, as before.

Full suite, rebased on current main (after #128): 492 passed, 59 skipped, 49 xfailed.

Breaking changes (for the release notes):

TString, STLString and RString are no longer classes. Every string member is bytes: key.fClassName instead of key.fClassName.fString, std::string members instead of .value. TString(...), TString.read(...) and isinstance(x, TString) no longer work; use ROOTString("TString").read(buffer).
STLString and RString are removed from rootfilespec.bootstrap. CountedString (new in this PR's first commit) is replaced by ROOTString.

Assisted-by: claude-code:claude-opus-5-5

codecov-commenter · 2026-09-24T17:14:43Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.63866% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.77%. Comparing base (79fd46d) to head (7846ed2).

Files with missing lines	Patch %	Lines
src/rootfilespec/bootstrap/TStreamerInfo.py	90.90%	2 Missing ⚠️
src/rootfilespec/structutil.py	95.65%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #127      +/-   ##
==========================================
+ Coverage   92.51%   92.77%   +0.25%     
==========================================
  Files          37       37              
  Lines        2994     3017      +23     
==========================================
+ Hits         2770     2799      +29     
+ Misses        224      218       -6

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sathabbott · 2026-09-30T14:42:37Z

@nsmith- i reviewed this and think it is ready to merge. if i understand it correctly, it is just converting RString to a regular byte string instead of its own sub class of root serializable.

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott · 2026-09-30T21:22:19Z

@nsmith- i finished reviewing this and i think it makes sense.

…ventions The strings-as-bytes rule and "keep what is on disk" conflict when a string's class (TString vs std::string) is known only from a class tag that plain bytes drop, e.g. a string read through a pointer (nsmith-#127): the value can then not be written back as read. The rules now say the ROOT type must stay recoverable from the annotation or the enclosing record, or the loss must be documented. Setup: CI installs with `uv sync` from uv.lock, and pre-commit is not in the dev group, so `pre-commit install` failed after the documented install. The pip route stays as the fallback. Also: test_spec_cases.py skips without the submodule too; nox runs both checks; the naming rule is for new test modules; new issues are sub-issues of nsmith-#66 or nsmith-#9; nsmith-#67's direction for dynamic.py; the Uninterpreted default for missing classes (nsmith-#74) and checksummed model names; and Assisted-by replaces tools' default attribution trailers. Assisted-by: claude-code:claude-opus-5-5

nsmith- · 2026-09-30T23:31:57Z

🤖 AI generated content

Since this is the omnibus PR for string handling, two fixes for docs/design.md, section "Annotated builtin types vs. objects":

It says the @serializable decorator "takes care of the conversion between str and TString". It should say bytes: strings are never decoded (Represent string-like members (TString, std::string, char*) as plain bytes in deserialized dataclasses, as promised in docs/design.md #68).
Typo: "The read_mupdate_membersembers method" should be "The update_members method".

Related to the round-trip question in this PR: once it's decided whether a string held through a pointer keeps its class (TString vs string), that section is the place to record it as a design assumption. See the AGENTS.md discussion in #134.

Assisted-by: claude-code:claude-opus-5-5

sathabbott · 2026-09-30T23:35:36Z

ok, i reworked this PR a bit now that I understand how to use Claude code. i can iterate much faster now, and taught it some rules about how to behave correctly in this repo. ill let it reply to you.

A std::string data member is the TString encoding inside the usual byte count and version word (root-io-spec Conventions §5.3). The frame is not a fourth string encoding, so: - StringEncoding is back to the three length formats Nick listed on nsmith-#127: "RNTuple", "TString", "charstar". - ROOTString(encoding, framed=False). With framed=True it reads the frame with StreamHeader, as StdVector and the other framed members do, and checks the byte count, instead of parsing the frame itself with a second copy of kByteCountMask. - read_string() is gone. ROOTString.read(buffer) is the one string reader: the member reader calls it, and TKey and TList call it for their hand-read strings. - read_value() takes type[MemberType] | type[ROOTSerializable], as Nick suggested, and goes through _build_read for both kinds of type. - docs/design.md states the design assumption Nick raised: the bytes don't record their encoding; the annotation, key class name or class tag that holds them does, and writing a string back needs it. - test_tdatime_record is dropped: the TDatime record reads as a side effect of read_value, but it belongs to nsmith-#123, not nsmith-#68. Assisted-by: claude-code:claude-opus-5-5

"read_mupdate_membersembers" was a garbled "update_members", as Nick noted on nsmith-#127. Assisted-by: claude-code:claude-opus-5-5

sathabbott · 2026-09-30T23:49:05Z

🤖 AI generated content

Thanks, both are in docs/design.md now:

The str / TString sentence was rewritten earlier in this PR (e264fed). It now says the ROOTString annotation tells @serializable how the bytes are encoded on disk, and that strings stay bytes.
The update_members typo is fixed in 3d177c8.

On the round trip: the same section now states the design assumption (a591ca1). The bytes don't record which encoding they came from; the member's annotation, the key's class name or the streamed object's class tag does. So a TString held through a pointer keeps its class only through that class tag. If that should change, it can be decided alongside #135 and recorded there.

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

The design note claimed a streamed object's class tag keeps a string's type, like an annotation or a key's class name does. It doesn't: read_streamed_item reads the tag to look the type up and then returns bare bytes. Members, container elements and keyed records keep the type; a string read through a pointer can't be written back as read. Tracked in nsmith-#135, as Nick noted on nsmith-#127 and in nsmith-#134's AGENTS.md. Assisted-by: claude-code:claude-opus-5-5

docs/design.md promises that where a builtin Python type captures a ROOT type, the deserialized member is that builtin: bytes for strings. RNTuple strings were RString wrappers instead. Add CountedString, a MemberSerDe for a string stored as its length (in a given struct format) followed by that many bytes, read as plain bytes. An RNTuple string is Annotated[bytes, CountedString("<I")], exported from rntuple.schema as RNTupleString, and the eight RNTuple string members (the header's name, description and library; a field's name, type name, type alias and description; extra type information's type name) use it. RString is removed. This is the RNTuple row of nsmith-#68. The TString, std::string and char* rows stay open there; char* (nsmith-#20) can use CountedString(">i"). Part of nsmith-#68 Assisted-by: claude-code:claude-opus-5-5

Name the module after what it tests, so it does not read as the home of a broad set of tests (review on nsmith-#125). Assisted-by: claude-code:claude-opus-5-5

Assisted-by: claude-code:claude-opus-5-5

…smith-#68, nsmith-#20) Replace CountedString with ROOTString(encoding), one Literal value per on-disk form (root-io-spec Conventions §5), as suggested in review: - "RNTuple": u32 little-endian length, then the bytes - "TString": the counted string, one length byte or 255 then a u32 - "std::string": a std::string data member, the counted string inside a byte count and version word (§5.3); the byte count is checked - "charstar": a char* member, i32 length then the bytes, no 255 escape; a length <= 0 reads as b"" (§5.4, nsmith-#20) TString is now the alias Annotated[bytes, ROOTString("TString")], so every TString member is bytes and .fString is gone. STLString is removed. read_string(buffer, encoding) reads one string by hand (TKey, TList options). Generated classes write the annotation out: TStreamerString emits ROOTString('TString'), TStreamerSTLstring ROOTString('std::string'), a kCharStar basic type ROOTString('charstar'), and cpptype maps string, std::string and TString elements of containers and pointees to ROOTString('TString'). A string stored as an object of its own (a key, or through a pointer) is looked up by class name, so TString, TStringLong and string stay resolvable by name, and serializable.read_value reads either a class or an annotated builtin. TStringLong is the "charstar" encoding (§5.1.1). As a result, root-io-spec's serialization/stringlong.root now reads, and serialization/unframed-records.root reads its TDatime, TString and TStringLong records and stops at the TObject record (nsmith-#123). Assisted-by: claude-code:claude-opus-5-5

A std::string data member is the TString encoding inside the usual byte count and version word (root-io-spec Conventions §5.3). The frame is not a fourth string encoding, so: - StringEncoding is back to the three length formats Nick listed on nsmith-#127: "RNTuple", "TString", "charstar". - ROOTString(encoding, framed=False). With framed=True it reads the frame with StreamHeader, as StdVector and the other framed members do, and checks the byte count, instead of parsing the frame itself with a second copy of kByteCountMask. - read_string() is gone. ROOTString.read(buffer) is the one string reader: the member reader calls it, and TKey and TList call it for their hand-read strings. - read_value() takes type[MemberType] | type[ROOTSerializable], as Nick suggested, and goes through _build_read for both kinds of type. - docs/design.md states the design assumption Nick raised: the bytes don't record their encoding; the annotation, key class name or class tag that holds them does, and writing a string back needs it. - test_tdatime_record is dropped: the TDatime record reads as a side effect of read_value, but it belongs to nsmith-#123, not nsmith-#68. Assisted-by: claude-code:claude-opus-5-5

TypedTKey.read stores the looked-up type as objtype, and after nsmith-#68 a lookup of "TString", "TStringLong" or "string" returns an annotated alias, not a class. read_object called objtype.read() on it and raised AttributeError, so a TypedTKey of a string record could not be read, although fetching the same key untyped returned its bytes. Both branches now read through read_value. Test: the "tstring" record of root-io-spec serialization/unframed-records through TypedTKey reads b"hello". Assisted-by: claude-code:claude-opus-5-5

"read_mupdate_membersembers" was a garbled "update_members", as Nick noted on nsmith-#127. Assisted-by: claude-code:claude-opus-5-5

The design note claimed a streamed object's class tag keeps a string's type, like an annotation or a key's class name does. It doesn't: read_streamed_item reads the tag to look the type up and then returns bare bytes. Members, container elements and keyed records keep the type; a string read through a pointer can't be written back as read. Tracked in nsmith-#135, as Nick noted on nsmith-#127 and in nsmith-#134's AGENTS.md. Assisted-by: claude-code:claude-opus-5-5

…mith-#128 nsmith-#128's tests index key.fClassName.fString; after nsmith-#68 the class name is plain bytes. Assisted-by: claude-code:claude-opus-5-5

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

…#127 nsmith-#127 made key class names plain bytes. Assisted-by: claude-code:claude-opus-5-5

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

…ventions The strings-as-bytes rule and "keep what is on disk" conflict when a string's class (TString vs std::string) is known only from a class tag that plain bytes drop, e.g. a string read through a pointer (nsmith-#127): the value can then not be written back as read. The rules now say the ROOT type must stay recoverable from the annotation or the enclosing record, or the loss must be documented. Setup: CI installs with `uv sync` from uv.lock, and pre-commit is not in the dev group, so `pre-commit install` failed after the documented install. The pip route stays as the fallback. Also: test_spec_cases.py skips without the submodule too; nox runs both checks; the naming rule is for new test modules; new issues are sub-issues of nsmith-#66 or nsmith-#9; nsmith-#67's direction for dynamic.py; the Uninterpreted default for missing classes (nsmith-#74) and checksummed model names; and Assisted-by replaces tools' default attribution trailers. Assisted-by: claude-code:claude-opus-5-5

…ve decoding Rewrite the builtin-types rule after nsmith-#127's review: a value may be an annotated builtin only inside a context that records its ROOT type and encoding (a member's annotation, a container's element type, a TKey's fClassName, or a pointee's Ref, which is to keep the stream header per nsmith-#105). Decoding must be injective on every accepted input, so inputs that would decode to the same value are rejected or kept, not merged. Assisted-by: claude-code:claude-opus-5-5

* Add AGENTS.md: how to work on rootfilespec, for agents and people What the package is and its layout, the design rules from docs/design.md and recent reviews (locators, strings as bytes, keep what is on disk, don't guess), root-io-spec as the format reference, setup with the non-recursive submodule, the checks CI runs (pre-commit's pinned tools, and mypy's minimal environment), how tests are written, the shape of issues, commits and PRs, and the AI-attribution convention. CLAUDE.md imports it, since Claude Code reads CLAUDE.md. Assisted-by: claude-code:claude-opus-5-5 * AGENTS.md: reconcile the string and on-disk rules, fix setup, add conventions The strings-as-bytes rule and "keep what is on disk" conflict when a string's class (TString vs std::string) is known only from a class tag that plain bytes drop, e.g. a string read through a pointer (#127): the value can then not be written back as read. The rules now say the ROOT type must stay recoverable from the annotation or the enclosing record, or the loss must be documented. Setup: CI installs with `uv sync` from uv.lock, and pre-commit is not in the dev group, so `pre-commit install` failed after the documented install. The pip route stays as the fallback. Also: test_spec_cases.py skips without the submodule too; nox runs both checks; the naming rule is for new test modules; new issues are sub-issues of #66 or #9; #67's direction for dynamic.py; the Uninterpreted default for missing classes (#74) and checksummed model names; and Assisted-by replaces tools' default attribution trailers. Assisted-by: claude-code:claude-opus-5-5 * AGENTS.md: match the repository as it is today - Setup: there is no uv.lock (it is gitignored), and CI installs the test group (`uv sync --group test`); dev is that group plus mypy. - The Uninterpreted rule (#74) and checksum-named models (#22, under #67) are open issues: today a missing class raises "Unknown type ...", and a generated class has the bare class name. Say they are planned. - Tests: test_compressed_size.py (from #130) also uses root-io-spec fixtures and fails without the submodule instead of skipping; say to check it out first, and that a test that doesn't skip is a bug. - case.toml files are under gen/cases/ and gen/written/. - "No I/O" is about the parsing code; reader.py does read files. - Sub-issues: only where a tracker fits (the RNTuple issues fit neither #66 nor #9). Assisted-by: claude-code:claude-opus-5-5 * AGENTS.md: state where builtin types are allowed, and require injective decoding Rewrite the builtin-types rule after #127's review: a value may be an annotated builtin only inside a context that records its ROOT type and encoding (a member's annotation, a container's element type, a TKey's fClassName, or a pointee's Ref, which is to keep the stream header per #105). Decoding must be injective on every accepted input, so inputs that would decode to the same value are rejected or kept, not merged. Assisted-by: claude-code:claude-opus-5-5 * Move the design rules into docs/design.md, keep a summary in AGENTS.md AGENTS.md and docs/design.md described the same rules and had already drifted: design.md sent the pointer case to #135, AGENTS.md to #105. docs/design.md now holds each rule in full, with the reasons and the issues still open against it: - "Annotated builtin types vs. objects" states the context rule (member annotation, container element type, TKey, pointee Ref per #105, survey in #139), that the choice of builtin is ergonomic (TDatime, #123), and the injectivity requirement (kBool 0x99, root-io-spec ElementTypes §2.5; #140). - New sections for keeping what is on disk, content the parser does not understand (#74), and generated classes (#67, #22). AGENTS.md keeps one line per rule, linking its design.md section, so an agent still has the rules in context. Assisted-by: claude-code:claude-opus-5-5 --------- Co-authored-by: Nick Smith <nick.smith@cern.ch>

* Verify envelope checksums Every RNTuple envelope ends with an XXH3-64 checksum. rootfilespec read it and compared stored checksums with each other, but never hashed the bytes, so a corrupted header, footer or page list parsed without complaint. REnvelope.read now checks the checksum over [0, length - 8) of the uncompressed envelope: the length includes the checksum, and the checksum covers everything before it (root-io-spec ERRATA 5, checked against ROOT's VerifyXxHash3, RNTupleSerialize.cxx:934). Unknown trailing bytes are covered too, as the spec's compatibility notes require. REnvelopeLocator.read_from also refuses a stored size larger than the uncompressed length. RNTuple decompression branches on equality (root-io-spec NOTES 2, RNTupleZip.hxx:106-113), so that case is an error, where the code used to try to decompress it. Fixes #117 Assisted-by: claude-code:claude-opus-5-5 * Ignore the missing xxhash stubs in the pre-commit mypy environment The pre-commit mypy hook installs only pytest, numpy and tomli, so it cannot find xxhash. bootstrap/compression.py already imports it the same way. Assisted-by: claude-code:claude-opus-5-5 * Rename tests/test_rntuple_envelopes.py to tests/test_envelope_checksums.py Name the module after what it tests, so it does not read as the home of a broad set of tests (review on #125). Assisted-by: claude-code:claude-opus-5-5 * Verify envelope checksums before parsing, and check the fetched size Two fixes from review: - REnvelope.read now verifies the XXH3-64 right after the type and length checks, before the payload is parsed, as ROOT's DeserializeEnvelope does (RNTupleSerialize.cxx:909-939). Before, the payload was parsed first, so 136 of the 224 single-byte corruptions of rntuple/anchor.root's header envelope failed with an unrelated parse error instead of the checksum mismatch. Envelopes shorter than 16 bytes are refused, as in ROOT. - REnvelopeLocator.read_from checks that it was given the locator's size, then applies NOTES 2's raw/compressed rule to the locator's stored size rather than to the buffer's length. A short read (a truncated file) said "Unknown compression algorithm". Tests: every corrupted byte of that envelope reports the checksum mismatch; short reads of 0, 200 and 239 of 240 bytes raise. Assisted-by: claude-code:claude-opus-5-5 * Read the envelope checksum through ReadBuffer, not struct Slice the buffer to its last 8 bytes and unpack there, as the rest of the library does, instead of importing struct into envelope.py. Assisted-by: claude-code:claude-opus-5-5 * Compare key class names as bytes in the envelope tests, after #127 #127 made key class names plain bytes. Assisted-by: claude-code:claude-opus-5-5 * Split the envelope once into the checksummed bytes and the checksum REnvelope.read mixed three views of the envelope: the checksum was read from the buffer after the preamble at length - 16, hashed against a separate envelope_bytes slice at length - 8, and the payload was parsed from the buffer. It now splits the envelope once, into the bytes the checksum covers and the 8-byte checksum, verifies one against the other, and parses the payload from the covered bytes after the preamble, so nothing reads into the checksum (review on #129). The preamble is still read from the whole buffer first, because the split needs its length. Every envelope that read before reads the same. Two errors change: the length mismatch now reports the full envelope length and buffer length (before, both minus the 8-byte preamble), and a frame that runs past the payload raises IndexError ("Cannot get slice") where it raised ValueError ("Cannot consume a negative number of bytes"). Assisted-by: claude-code:claude-opus-5-5 * Test the envelope length checks and the pinned header checksum - rntuple/anchor.root's case.toml pins the header envelope's checksum at 500 and the footer's copy at 854; assert the parsed values equal them. - An envelope whose length is under 16 bytes (the preamble and checksum alone) raises, as in ROOT's DeserializeEnvelope. - anchor.root's header with the preamble's length changed from 240 to 248 raises the length mismatch. Also drop the count of failures before the reordering from a test docstring; the commit that reordered the checks records it. Assisted-by: claude-code:claude-opus-5-5

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

…infos (#132) * Read the extra type information's content ExtraTypeInformation ended at its type name. ROOT's serializer writes one more string, the content (RNTupleSerialize.cxx:389-391; root-io-spec ERRATA 9), so its length and bytes were left in the record frame's _unknown. For content identifier 0 that content is the streamed TList of TStreamerInfo that streamed (role 0x04) fields need. Add fContent, an RNTuple string like the type name. Fixes #118 Assisted-by: claude-code:claude-opus-5-5 * Rename tests/test_rntuple_extra_type_info.py to tests/test_extra_type_info_content.py Name the module after what it tests, so it does not read as the home of a broad set of tests (review on #125). Assisted-by: claude-code:claude-opus-5-5 * Compare key class names as bytes, after the rest of #68 in #127 Assisted-by: claude-code:claude-opus-5-5 * Pin the streamed.root bytes exactly in the extra type info test AGENTS.md asks tests to assert the values a case.toml pins. Compare the content with the file's bytes at offset 1240 (length 438 at 1236), the TStreamerInfo name's length byte at 1319 and the name after it, and the frame size 462, instead of only checking that the name occurs somewhere. Say in fContent's docstring that it has an RNTuple string's layout but holds binary bytes, not UTF-8 text. Assisted-by: claude-code:claude-opus-5-5 * Decode the streamer info content: RNTuple.streamer_infos() The extra type information with content identifier 0 holds the TStreamerInfo of every class that streamed (role 0x04) fields need, as a TList written as if through a pointer (RNTupleSerializer:: SerializeStreamerInfos, ROOT 6.40.04). For an RNTuple read without its TFile, it is the only description of those classes. streamer_infos() reads it with the bootstrap classes, by class name like FileReader.streamerinfos(). It is derived: fContent keeps the bytes. A list item that comes back as a Ref is unwrapped, so the result stays the same when #105 makes every pointee a Ref. Other content identifiers are ignored, as the spec asks; anything else in the content, bytes after the TList, or the same class twice is an error rather than a guess. Assisted-by: claude-code:claude-opus-5-5

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

… field information and global cluster IDs (#131) * Give page descriptions their column, field and global cluster ID RNTuple.get_extended_page_descriptions(): - returns one entry per physical column in every cluster, so a position is the column ID, as the spec guarantees (Page Locations: the outer items follow column order). With the old default includeSuppressed=False, suppressed columns were dropped and positions shifted. The parameter is removed; a suppressed column is an entry with no pages. The zip over columns is strict, as suggested on #49. - gives each page its RNTuple-wide cluster ID. Cluster IDs continue across cluster groups (Page List Envelope); the new RNTuple.firstClusterIDs is the running sum of the footer's fNClusters. Each page list's cluster count is checked against its group. - gives each page its column ID, field ID, FieldDescription and qualified field path. SchemaDescription.field_path(field_id) walks fParentFieldID up to the top-level field (which names itself as parent) and joins the names with "." (forbidden in names, so unambiguous). It returns bytes, like the names, and raises on an out-of-range ID or a cycle. The schema is built once per call; the schemaDescription property itself is unchanged. Fixes #49 Fixes #116 Assisted-by: claude-code:claude-opus-5-5 * Rename tests/test_rntuple_page_descriptions.py to tests/test_extended_page_descriptions.py Name the module after what it tests, so it does not read as the home of a broad set of tests (review on #125). Assisted-by: claude-code:claude-opus-5-5 * Compare key class names as bytes, after the rest of #68 in #127 Assisted-by: claude-code:claude-opus-5-5 * Accept page lists that predate a model extension A cluster committed before the model was extended lists only the columns that existed then, so its page list can end before the schema extension's columns do (the spec fixes the columns' order, not their number). ROOT fills those in when reading (AddExtendedColumnRanges, RNTupleDescriptor.cxx:920 at 6.40.04). The strict zip raised on scikit-hep-testdata's test_extension_columns_rntuple_v1-0-0-0.root, which read before this PR: its first cluster lists 2 of 4 columns. get_extended_page_descriptions now gives such trailing columns empty entries, so a position is still the column ID. A cluster listing more columns than the schema, or fewer than the header's, still raises. uncompressedSize uses RPageDescription.n_elements (#128). Tests: that file's clusters, a page list missing a header column, and the per-page column/field check on three scikit-hep-testdata files as well as root-io-spec's. Assisted-by: claude-code:claude-opus-5-5 * Give each column in a cluster its element range, not just its pages get_extended_page_descriptions() now returns one InterpretableColumn per physical column per cluster, so that the view is enough on its own to read a column. Before, a column was a bare list of pages, and an empty list could mean a suppressed column, a column with no pages, or a column the cluster predates; the element offset was not in the view at all. The ranges are the ones ROOT's reader builds (CommitSuppressedColumnRanges and AddExtendedColumnRanges, tree/ntuple/src/RNTupleDescriptor.cxx:880 and :920 at 6.40.04), which is what the spec asks of a reader: - a suppressed column takes the range of the corresponding column of the field's active representation (spec, Suppressed Columns); - a deferred column covers every cluster from element 0, the elements before its first stored one being zeros with no page (Column Description): nZeroElements counts them; - a column the cluster predates keeps pageLocations=None. New checks, each with a test: a cluster cannot list fewer columns than an earlier one, since only a model extension adds columns; a page list needs a cluster summary per cluster; a suppressed column has no pages; every field has an active representation; a deferred column's pages start where its range says; and a deferred column may not be inside a collection or variant, whose element count the format does not record. The tests now also pin each column's field path and type against root-io-spec's user-class case, and check on every fixture that a column's clusters follow each other with no gap. Assisted-by: claude-code:claude-opus-5-5 * Return clusters, each with its columns, each with its pages RNTuple.clusters() replaces get_extended_page_descriptions(): one InterpretableCluster per cluster, in cluster-ID order, shaped like ROOT's RClusterDescriptor. A cluster carries its ID, its cluster group's position and its summary as stored (first entry, number of entries), so a caller no longer zips the view back with the page lists and the footer. The nesting by page-list envelope goes, and with it RNTuple.firstClusterIDs. Each fact is now stored once. InterpretableColumn keeps the column's and field's descriptions and path; InterpretablePage keeps only the page as stored, its uncompressed size and, new, the column-wide index of its first element (ROOT's RPageInfoExtended). The page's columnType, clusterID, columnID, fieldID, fieldDescription and fieldPath are on its column. SchemaDescription.field_columns() gives every field's physical columns by representation and index, as ROOT numbers them (RNTupleSerialize.cxx:1511 at 6.40.04), and raises when a field's representations differ in size (spec, Suppressed Columns). The cluster view uses it to find corresponding columns. Assisted-by: claude-code:claude-opus-5-5 * Count a page's first element from the start of its cluster InterpretablePage.firstElementIndex, counted from the start of the column, becomes firstElementInCluster, counted from the start of the column's elements in the cluster, as ROOT's RPageInfoExtended counts it. A cluster can be reused unchanged in another RNTuple, because offset columns count from the start of the cluster (spec, Column Description), so this index belongs to the cluster, while the column's firstElementIndex is a position in this RNTuple. InterpretableCluster's docstring now says which values are which. Assisted-by: claude-code:claude-opus-5-5 * Test a column that names a field the schema does not have Assisted-by: claude-code:claude-opus-5-5 * Yield clusters one at a time, and make the columns a step of their own Following Nick's review of #131: - RNTuple.clusters() is now an iterator: each cluster is built when it is reached, so a caller that goes through them holds one cluster's ranges at a time. On the MiniAOD Events RNTuple (10,336 columns, 27 clusters), peak memory while iterating is 11 MB, against 84 MB for the whole list. The schema and the number of page lists and clusters are still checked when it is called; each cluster's page list when it is reached. - RNTuple.columns() is the public form of the per-column work that was private (_Columns): one InterpretableColumn per physical column, the same in every cluster, like ROOT's RColumnDescriptor, with its description, field, path and index within its representation. A caller can store these before going through the clusters. - What was InterpretableColumn, one column in one cluster, is now InterpretableColumnRange (ROOT's RColumnRange with its RPageRange). It refers to its InterpretableColumn instead of repeating the column's description, field and path in every cluster, and InterpretableCluster.columns becomes columnRanges. Assisted-by: claude-code:claude-opus-5-5

… builtin (#147) * FileContext.type_by_name returns object: a lookup can be an annotated builtin type_by_name was annotated -> type[ROOTSerializable], but TString, TStringLong and string resolve to annotated bytes aliases (#68), and a file's context resolves TDatime to an annotated int. mypy therefore let callers call .read() on the result, the TypedTKey bug #127 fixed, and test_read.py's basket reader still did. Option 1 on #135: FileContext.type_by_name and both implementations return object, with the contract in the docstring: read the result with read_value, which now takes object, and narrow with isinstance(t, type) and issubclass(t, ROOTSerializable) where the class is needed. mypy now rejects .read() on a lookup. TKey.read_object (untyped) and TKey.read_from return object as well, since a TString record reads as bytes; so the Locator protocol's type variable is no longer bound to ROOTSerializable. The basket reader in test_read.py uses read_value. Fixes #135 Assisted-by: claude-code:claude-opus-5-5 * tests: drop the object annotations #145 added for string records #145 typed two string-record reads in test_root_string.py as object, because TKey.read_from claimed ROOTSerializable. It is typed object now, so the reads compare directly again. Assisted-by: claude-code:claude-opus-5-5 * Say what Locator's type variable is for The comment on T_co only said why it is unbound. It is the type a locator returns (Locator[RPage], Locator[TTree]), unbound because some locators return a builtin. Wording from Nick's review of #147. Assisted-by: claude-code:claude-opus-5-5

This was referenced Sep 24, 2026

[rntuple] RNTuple.clusters(): every cluster's columns and pages, with field information and global cluster IDs #131

Merged

[rntuple] Read the extra type information's content and its streamer infos #132

Merged

sathabbott requested a review from nsmith- September 30, 2026 14:38

sathabbott force-pushed the fix/68-rntuple-strings-as-bytes branch from f9434b1 to acfd9c6 Compare September 30, 2026 14:53

nsmith- reviewed Sep 30, 2026

View reviewed changes

Comment thread src/rootfilespec/structutil.py Outdated

sathabbott force-pushed the fix/68-rntuple-strings-as-bytes branch from acfd9c6 to f3829c4 Compare September 30, 2026 20:40

sathabbott changed the title ~~[rntuple] Read RNTuple strings as plain bytes (RNTuple part of #68)~~ Read every ROOT string type as plain bytes with ROOTString (#68, #20) Sep 30, 2026

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Sep 30, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

fd9fa0e

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Sep 30, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

8bfdcb3

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Sep 30, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

4bf8f35

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott requested a review from nsmith- September 30, 2026 22:53

nsmith- requested changes Sep 30, 2026

View reviewed changes

Comment thread src/rootfilespec/bootstrap/__init__.py

Comment thread src/rootfilespec/serializable.py Outdated

Comment thread src/rootfilespec/structutil.py Outdated

Comment thread tests/test_tdatime.py Outdated

nsmith- mentioned this pull request Sep 30, 2026

Add AGENTS.md: how to work on rootfilespec #134

Merged

sathabbott mentioned this pull request Sep 30, 2026

FileContext.type_by_name can return an annotated builtin, not only a ROOTSerializable class #135

Closed

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Sep 30, 2026

Fix the update_members typo in docs/design.md

3d177c8

"read_mupdate_membersembers" was a garbled "update_members", as Nick noted on nsmith-#127. Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Sep 30, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

7d0c46d

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Sep 30, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

543b9e1

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott added 4 commits September 30, 2026 19:19

Rename tests/test_rntuple_strings.py to tests/test_counted_string.py

16a3ae8

Name the module after what it tests, so it does not read as the home of a broad set of tests (review on nsmith-#125). Assisted-by: claude-code:claude-opus-5-5

Compare header names with the key names as bytes, after nsmith-#130

215f8a2

Assisted-by: claude-code:claude-opus-5-5

sathabbott added 5 commits September 30, 2026 19:19

Fix the update_members typo in docs/design.md

48c22eb

"read_mupdate_membersembers" was a garbled "update_members", as Nick noted on nsmith-#127. Assisted-by: claude-code:claude-opus-5-5

Compare key class names as bytes in the page checksum tests, after ns…

7846ed2

…mith-#128 nsmith-#128's tests index key.fClassName.fString; after nsmith-#68 the class name is plain bytes. Assisted-by: claude-code:claude-opus-5-5

sathabbott force-pushed the fix/68-rntuple-strings-as-bytes branch from 8598e58 to 7846ed2 Compare October 1, 2026 00:20

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 1, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

128e86c

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 1, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

dff80b1

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott mentioned this pull request Oct 1, 2026

Skip test_compressed_size.py without the root-io-spec submodule #136

Merged

nsmith- approved these changes Oct 1, 2026

View reviewed changes

sathabbott mentioned this pull request Oct 1, 2026

An object slot's class record and the object's own frame are conflated (pointers to non-TObject classes desync) #105

Open

sathabbott merged commit d2758de into nsmith-:main Oct 1, 2026
9 checks passed

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 1, 2026

Compare key class names as bytes in the envelope tests, after nsmith-…

173befd

…#127 nsmith-#127 made key class names plain bytes. Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 1, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

0366e4e

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 1, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

b1f5f03

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott mentioned this pull request Oct 1, 2026

[rntuple] Verify envelope checksums #129

Merged

nsmith- mentioned this pull request Oct 1, 2026

Survey: where each way of writing an object records its ROOT type (wrappers for builtin values) #139

Open

sathabbott mentioned this pull request Oct 1, 2026

Implement ElementType.kCharStar #20

Closed

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 2, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

ebede95

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 2, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

ca98a45

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott added a commit to sathabbott/rootfilespec that referenced this pull request Oct 2, 2026

Compare key class names as bytes, after the rest of nsmith-#68 in nsm…

73d6524

…ith-#127 Assisted-by: claude-code:claude-opus-5-5

sathabbott mentioned this pull request Oct 5, 2026

FileContext.type_by_name returns object: a lookup can be an annotated builtin #147

Merged

Navigation Menu

Repository navigation

Read every ROOT string type as plain bytes with ROOTString (#68, #20) - #127

Read every ROOT string type as plain bytes with ROOTString (#68, #20)#127
sathabbott merged 9 commits into
nsmith-:mainfrom
sathabbott:fix/68-rntuple-strings-as-bytes

sathabbott commented Sep 24, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 24, 2026 •

edited

Loading

Uh oh!

sathabbott commented Sep 30, 2026

Uh oh!

Uh oh!

sathabbott commented Sep 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nsmith- commented Sep 30, 2026

Uh oh!

sathabbott commented Sep 30, 2026

Uh oh!

sathabbott commented Sep 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sathabbott commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sathabbott commented Sep 30, 2026

Uh oh!

Uh oh!

sathabbott commented Sep 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nsmith- commented Sep 30, 2026

Uh oh!

sathabbott commented Sep 30, 2026

Uh oh!

sathabbott commented Sep 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sathabbott commented Sep 24, 2026 •

edited

Loading

codecov-commenter commented Sep 24, 2026 •

edited

Loading