Skip to content

Track null value ratio of flushed MemTables#18287

Open
jt2594838 wants to merge 4 commits into
apache:masterfrom
jt2594838:stats_null_ratio_in_memtable
Open

Track null value ratio of flushed MemTables#18287
jt2594838 wants to merge 4 commits into
apache:masterfrom
jt2594838:stats_null_ratio_in_memtable

Conversation

@jt2594838

Copy link
Copy Markdown
Contributor

Description

Null value statistics

  • Count valid and null values for row, aligned-row, tablet, and aligned-tablet writes.
  • Exclude failed measurements and keep the statistics in memory only without WAL serialization.
  • Aggregate the counts across MemTables written to the same TsFile.

Flush logging and monitoring

  • Log the aggregated null value ratio in TsFileProcessor.updateCompressionRatio.
  • Add the flushing_mem_table_status histogram with name=null_value_ratio and the DataRegion dimension.
  • Record ratio times 100, consistent with the existing compression ratio metric.

Tests

  • Extended AbstractMemTablePartialInsertTest to cover all four write paths, including failed measurements and zero-value boundaries.
  • Added scenario, key-input, and expected-result documentation to the test methods.
  • mvn -pl iotdb-core/datanode -DskipTests spotless:check compile
  • AbstractMemTablePartialInsertTest: 14 tests passed

This PR has:

  • been self-reviewed.
  • added comments explaining the intent of the tests.
  • added or updated unit tests to cover the new code paths.
Key changed/added classes
  • AbstractMemTable and IMemTable
  • TsFileProcessor
  • WritingMetrics
  • AbstractMemTablePartialInsertTest

@Caideyipi

Copy link
Copy Markdown
Collaborator

Confirmed the aligned-tablet row-failure issue with a regression test. When a row is marked failed in the TSStatus results array, insertAlignedTablet still counted all of its field values in totalValueCount, while null counting also ignored the row status. The new test failed before the fix with expected totalValueCount 2 but actual 4.\n\nPushed fix ec09574, which excludes failed rows from valid-value/null-value statistics and from pointsInserted.\n\nVerification:\n- Target regression test: 1 test passed\n- AbstractMemTablePartialInsertTest: 15 tests passed\n- datanode spotless:check passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants