Skip to content

[arrow] Use the nested field's own nullability for sub writers#8758

Merged
JingsongLi merged 2 commits into
apache:masterfrom
thswlsqls:fix/arrow-nested-field-writer-nullability
Jul 23, 2026
Merged

[arrow] Use the nested field's own nullability for sub writers#8758
JingsongLi merged 2 commits into
apache:masterfrom
thswlsqls:fix/arrow-nested-field-writer-nullability

Conversation

@thswlsqls

Copy link
Copy Markdown
Contributor

Purpose

fix #8757

  • ArrowFieldWriterFactoryVisitor passed the parent field's isNullable to ARRAY/VECTOR/MAP/ROW sub writers, and ArrowFieldWriter.write rejects null with IllegalArgumentException when that flag is false. An ARRAY<INT> NOT NULL column therefore rejected valid null elements, while a NOT NULL child under a nullable parent silently accepted nulls.
  • Each sub writer now uses its own type's isNullable(), matching the child nullability ArrowUtils.toArrowField() declares.
  • Same rule applied to ArrowUtils.createArrowFieldWriters() (test-only caller).
  • Map keys follow the key type; toArrowField() declares the Arrow key field not-null, left as a follow-up.
  • Follow-up of [arrow] Fix bug: field writer contruct in ArrowFormatWriter should use nested field nullable to create sub writers #5538, which fixed only ArrowFormatWriter.

Tests

  • Added four ArrowFormatWriterTest cases covering null children under NOT NULL ARRAY/MAP columns and NOT NULL ARRAY elements / nested ROW fields. All four fail before the change.
  • mvn -pl paimon-arrow -am clean install — 52 tests passed.
  • mvn -pl paimon-core -Dtest=ArrowBatchConverterTest test — 21 tests passed.

@JingsongLi

Copy link
Copy Markdown
Contributor
  • ArrowFieldWriterFactoryVisitor.visit(MapType) changes the key writer’s nullability to mapType.getKeyType().isNullable(), but ArrowUtils.toArrowField() always declares the Arrow map key schema as NOT NULL.
  • Paimon’s MapType explicitly allows null keys; the default INT key is also nullable. Consequently, MAP<INT, INT> NOT NULL can now successfully write null keys, generating data that conflicts with the Arrow schema/spec.
  • The local null-key use case has been reproduced: it is expected to throw an exception, but no exception is actually thrown. It is recommended that the key writer consistently use false; null keys that cannot be expressed should result in a “fail loud” error.

ArrowUtils.toArrowField always declares the Arrow map key field as
NOT NULL (per the Arrow spec, MapVector keys cannot be null). The
previous change made the key writer use the declared key type's own
nullability, so a nullable key type (the default INT key is nullable)
would silently write a null key into a vector whose schema forbids it,
instead of failing loud.

Keep the key writer non-nullable regardless of the key type. The value,
array/vector element, and row-field branches still mirror each nested
type's own nullability, which is what the schema declares for them.

Add testWriteNullKeyInMapColumnFailsLoud covering the null-key case.

Generated-by: claude-code
@thswlsqls

Copy link
Copy Markdown
Contributor Author

Thanks, good catch. You're right — ArrowUtils.toArrowField() forces the map key to notNull(), so the key writer has to stay non-nullable regardless of keyType.isNullable(); otherwise a null key gets written instead of failing loud, which contradicts the schema. Fixed — the key writer is back to non-nullable (the value, array-element, and row-field branches still mirror each nested type's own nullability, which is what the schema declares for them), and I added a null-key test. Just pushed.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit ca8c2ba into apache:master Jul 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Arrow sub field writers inherit the parent field's nullability

2 participants