Skip to content

[core] Fix query results being scrambled after querying $audit_log ba…#8804

Open
L-Gryps wants to merge 1 commit into
apache:masterfrom
L-Gryps:fix/audit-log-shared-options-leak
Open

[core] Fix query results being scrambled after querying $audit_log ba…#8804
L-Gryps wants to merge 1 commit into
apache:masterfrom
L-Gryps:fix/audit-log-shared-options-leak

Conversation

@L-Gryps

@L-Gryps L-Gryps commented Jul 22, 2026

Copy link
Copy Markdown

Closes #8803

Purpose

When a primary-key table has 'table-read.sequence-number.enabled' = 'true', querying its $audit_log (or $binlog) system table and then querying the base table causes:

  • Column data to be shifted by one position (the internal _SEQUENCE_NUMBER value leaks into the first column, all real columns shift right), or
  • A hard ClassCastException when a base column is non-numeric, e.g. HeapLongVector cannot be cast to BytesColumnVector, surfaced by Flink as Recovery is suppressed by NoRestartBackoffTimeStrategy.

Root cause

AuditLogTable's constructor mutates the wrapped FileStoreTable's options map in place when sequence-number reading is enabled:

this.wrapped.options().put(CoreOptions.KEY_VALUE_SEQUENCE_NUMBER_ENABLED.key(), "true");

FileStoreTable.options() returns schema().options(), which is the same Map<String, String> reference held by the catalog-cached FileStoreTable. After the first SELECT * FROM t$audit_log, the cached base table permanently
carries key-value.sequence_number.enabled=true. Subsequent reads of the base table go through KeyValueTableRead.unwrap, which then makes ValueContentRowDataRecordIterator prepend a long _SEQUENCE_NUMBER column to every row.
The physical row now has one extra column ahead of the logical row type, causing column mis-alignment or vector type mismatch.

Fix

Instead of mutating the shared options map, create a private copy of the wrapped FileStoreTable that carries the internal KEY_VALUE_SEQUENCE_NUMBER_ENABLED=true option only for the audit-log's own reads. The catalog-cached
base table is left untouched.

wrapped = wrapped.copyWithoutTimeTravel(
        Collections.singletonMap(
                CoreOptions.KEY_VALUE_SEQUENCE_NUMBER_ENABLED.key(), "true"));

### Test

Added regression coverage at two layers:

- paimon-core/src/test/java/org/apache/paimon/table/system/AuditLogTableTest.javatestReadBaseTableAfterAuditLogDoesNotIncludeSequenceNumber: constructs an AuditLogTable and asserts the wrapped base table's row type /
  options are not polluted with _SEQUENCE_NUMBER / KEY_VALUE_SEQUENCE_NUMBER_ENABLED.

- paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/BatchFileStoreITCase.javatestReadBaseTableAfterAuditLogWithSequenceNumberEnabled: end-to-end reproduction of the reported user scenario (query
  t$audit_log, then re-query t); base-table result must remain unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant