[core] Support row-count based data file rolling (write.target-row-num-per-file)#8739
Open
XiaoHongbo-Hope wants to merge 4 commits into
Open
[core] Support row-count based data file rolling (write.target-row-num-per-file)#8739XiaoHongbo-Hope wants to merge 4 commits into
XiaoHongbo-Hope wants to merge 4 commits into
Conversation
XiaoHongbo-Hope
force-pushed
the
de_row_rolling
branch
4 times, most recently
from
July 22, 2026 11:02
db36b3a to
475bb07
Compare
XiaoHongbo-Hope
marked this pull request as ready for review
July 22, 2026 12:05
XiaoHongbo-Hope
marked this pull request as draft
July 22, 2026 12:15
XiaoHongbo-Hope
force-pushed
the
de_row_rolling
branch
from
July 22, 2026 12:51
475bb07 to
b1eb21c
Compare
Files roll by size only today. On data-evolution tables with very wide rows (e.g. vector/embedding columns), a size-bounded file still holds too many rows, so a later add-column / update-by-row-id OOMs materializing that row range. Lance handles this with max_rows_per_file; this adds the same to Paimon. - New option target-file-num-rows (default Long.MAX_VALUE, disabled). Rolls on row count or target-file-size, whichever first; bundle-granular. Validated to be positive at table creation / alter (SchemaValidation). - Applied to the write path only: the append writer (incl. the dedicated BLOB/vector writer) and the PK data + changelog writers. Data-evolution base writes therefore roll into row-bounded files, so a later add-column materializes bounded ranges. Scope: this bounds files at write time. Compaction is left unchanged (size-based) and data-evolution compaction still merges into a single file -- making compaction row-aware (append/bucketed/clustering selection, DE multi-output, DV handling) needs careful, contiguity-preserving work and is left as a follow-up. PyPaimon support is also a follow-up. Tests: RollingFileWriterTest#testRollingByRows/#testRollingByRowsWithBundle, DedicatedFormatRollingFileWriterTest#testRollingByRows, DataEvolutionTableTest#testDataEvolutionWriteRollsByRows/#testDataEvolutionReadWithRolledColumns, SchemaValidationTest#testTargetFileNumRowsMustBePositive.
XiaoHongbo-Hope
force-pushed
the
de_row_rolling
branch
from
July 22, 2026 13:26
b1eb21c to
2b738e8
Compare
Gate target-file-num-rows to the write path only: merge-tree compaction (FileSource.COMPACT) and changelog writers ignore the row cap so compaction stays size-based. Clarify the option description accordingly and regenerate docs. pypaimon has no row-count rolling yet, so its writer now fails fast when the option is set instead of silently ignoring it.
XiaoHongbo-Hope
marked this pull request as ready for review
July 23, 2026 03:53
JingsongLi
reviewed
Jul 23, 2026
XiaoHongbo-Hope
changed the title
XiaoHongbo-Hope
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Files roll by size only. On data-evolution tables with very wide rows (vector/embedding
columns), a size-bounded file still holds too many rows, so a later
add-column/update-by-row-idOOMs materializing that row range. Adds a Lance-stylemax_rows_per_file.write.target-row-num-per-file(defaultLong.MAX_VALUE, disabled). Rolls on rowcount or
target-file-size, whichever is reached first; bundle-granular. Validatedpositive at table create / alter.
the PK data writer roll into row-bounded files, so a later add-column materializes
bounded ranges.
Scope: bounds files at write time only. Compaction stays size-based (PK / changelog /
append all unchanged) and DE compaction still emits a single file. PyPaimon has no
row-count rolling yet, so its writer fails fast rather than silently ignoring the
option. Row-aware compaction is a follow-up.
Tests
New test methods added in this PR:
RollingFileWriterTest#testRollingByRows,#testRollingByRowsWithBundleDedicatedFormatRollingFileWriterTest#testRollingByRowsKeyValueFileReadWriteTest#testMergeTreeCompactionIgnoresRowLimitDataEvolutionTableTest#testDataEvolutionWriteRollsByRows,#testDataEvolutionReadWithRolledColumnsSchemaValidationTest#testWriteTargetRowNumPerFileMustBePositivereader_append_only_test.py::test_write_target_row_num_per_file_fails_fast