fix(storage): keep the data page limit out of the public API - #59
Merged
Merged
Conversation
`cargo-semver-checks` failed the v0.1.4 release: `ParquetWriteOptions` is a struct callers can build with a literal, so adding a public `data_page_rows` field breaks every exhaustive literal — a major change, and this is a patch. The gate did its job; nothing was published and the tag is deleted. The field was only ever there to make page-index pruning observable, and a 1,000-row data page limit is a test's requirement rather than a lake's. The test builds its own `WriterProperties` with the `parquet` crate instead, matching what `write_parquet` sets in every other respect — PARQUET_2_0, page statistics, uncompressed, one row group for the file. The proof is unchanged: 198,192 of 200,000 rows pruned by the index with no row group pruned. `cargo semver-checks -p oxidelake-storage` now reports no update required. Worth noting for #41: were `ParquetWriteOptions` `#[non_exhaustive]`, adding a field would be routine — which is the point of that issue, and a 0.2.0 change. Refs #43 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v0.1.4 release run failed at
cargo-semver-checksand the publish jobs were skipped — nothing reached crates.io, and the tag is deleted.My error, added while doing #43. Any caller writing
ParquetWriteOptions { .. }without..Default::default()would stop compiling, and 0.1.3 → 0.1.4 is a patch.The cause was process. Before tagging I ran fmt, clippy, the full suite,
cargo deny, crate-metadata and skill-version — and skippedcargo semver-checks, because it is a CI action rather than one of the.github/scriptsI had been working through. It was installed locally the whole time.The fix: a 1,000-row data page limit is a test's requirement, not a lake's, so it does not belong on the public struct. Reverted, and the page-index test builds its own
WriterProperties, matchingwrite_parquetin every other respect. The proof is unchanged — 198,192 of 200,000 rows pruned with no row group pruned.cargo semver-checks -p oxidelake-storage→ no semver update required. Full workspace suite 36/36, clippy, fmt, changelog extracts.For #41: if
ParquetWriteOptionswere#[non_exhaustive]this would have been routine. That is a 0.2.0 change.