Core: Read and write geometry and geography values in Avro by huan233usc · Pull Request #17119 · apache/iceberg

huan233usc · 2026-07-07T00:16:23Z

Summary

TypeToSchema mapped every Iceberg primitive to an Avro schema except geometry and geography,
so converting a schema with a geo column threw UnsupportedOperationException: Unsupported type ID: GEOMETRY before any value could be read or written through the Avro object model. This wires up the
value path.

Per the Avro type mapping in the spec, geometry and geography are stored as an Avro bytes field
carrying WKB — the same representation as binary:

Iceberg type	Avro type
`binary`	`bytes`
`geometry`	`bytes` (WKB)
`geography`	`bytes` (WKB)

So TypeToSchema.primitive() now maps both geo types to the existing BINARY_SCHEMA. The value
read/write paths dispatch on the Avro physical type, so once the schema is bytes the existing
byteBuffers reader/writer handle geo unchanged in both the generic and internal object models (geo
values are WKB ByteBuffers, exactly like binary) — no value-path code changes are needed.

This also closes the parquet-avro object model gap: ParquetAvroWriter / ParquetAvroValueReaders
build their Avro schema through AvroSchemaUtil.convert, which routed into the same TypeToSchema
that threw on geo, so geo was unreachable through that path too. Fixing TypeToSchema makes it work
transitively, with no parquet-avro code change. This is the follow-up szehon-ho asked about on #16982
(which added the Parquet WKB value path and deliberately left the Avro object model as a follow-up).

Note on the reverse direction

SchemaToType intentionally still maps Avro bytes → binary. Plain Avro bytes carries no marker to
distinguish geometry/geography from binary, and Iceberg resolves the real column type from the
expected/table schema, never by reverse-inferring from the file's Avro schema — the same way binary
and other bytes-backed types already behave.

Test plan

Object-model coverage via the shared DataTest geo cases and the Avro encoder/writer entry points:

Enable supportsGeospatial() on the Avro object-model tests — TestGenericAvro, TestInternalAvro,
the generic-data TestGenericData, and TestAvroEncoderUtil (so geo also round-trips through the
AvroEncoderUtil.encode/decode entry point, keeping the two generic-model Avro tests in parity).
This round-trips geometry and geography across multiple CRS and edge algorithms with randomly
generated WKB values.
Add the GEOMETRY/GEOGRAPHY → ByteBuffer handling the generic Avro path needs in RandomAvroData
and AvroTestHelpers, mirroring the existing RandomInternalData / InternalTestHelpers handling.
Explicit WKB round-trip tests:
- TestAvroDataWriter.testGeospatialWkbRoundTrip — geometry/geography through the Avro DataWriter
  → PlannedDataReader path, including nulls.
- TestParquet.testGeospatialWkbRoundTrip — through ParquetAvroWriter → ParquetAvroValueReaders,
  covering the parquet-avro object model, including nulls.
./gradlew :iceberg-core:test --tests org.apache.iceberg.avro.TestGenericAvro --tests org.apache.iceberg.avro.TestInternalAvro --tests org.apache.iceberg.avro.TestAvroEncoderUtil --tests org.apache.iceberg.avro.TestAvroDataWriter
./gradlew :iceberg-data:test --tests org.apache.iceberg.data.avro.TestGenericData
./gradlew :iceberg-parquet:test --tests org.apache.iceberg.parquet.TestParquet

TypeToSchema mapped Iceberg types to Avro schema for every primitive except geometry and geography, so converting a schema with a geo column threw UnsupportedOperationException before any value could be read or written. Map both to an Avro bytes field carrying WKB, per the Avro type mapping in the spec (binary, geometry, and geography all use bytes). The value read/write paths dispatch on the Avro physical type, so once the schema is bytes the existing byteBuffers reader/writer handle geo unchanged in both the generic and internal object models (geo values are WKB ByteBuffers, like binary). Enable the shared DataTest geospatial coverage for the Avro object models (TestGenericAvro, TestInternalAvro, generic data TestGenericData) and add the GEOMETRY/GEOGRAPHY -> ByteBuffer handling to RandomAvroData and AvroTestHelpers that the generic Avro path needs, mirroring RandomInternalData/InternalTestHelpers.

…l.wkbPoint TestParquet.testGeospatialWkbRoundTrip wrote arbitrary bytes through the geometry/geography schema, which carries a Parquet geospatial logical type: the writer parses each value with a WKB reader to build geospatial statistics, so the bogus bytes were silently dropped from stats and the round-trip never exercised a real WKB value. Use RandomUtil.wkbPoint for genuine WKB. Also delegate TestAvroDataWriter's local wkbPoint helper to RandomUtil.wkbPoint (already on the test classpath) to keep WKB encoding in one place, matching TestParquetDataWriter.

huan233usc · 2026-07-12T19:46:26Z

Thanks for the careful review @wombatu-kun! Addressed in bcbef59:

testGeospatialWkbRoundTrip now uses real WKB via RandomUtil.wkbPoint(...) so the geospatial statistics builder actually parses the values (the previous arbitrary bytes were silently dropped from stats).
TestAvroDataWriter.wkbPoint delegates to RandomUtil.wkbPoint and drops the ByteOrder import, matching the merged TestParquetDataWriter.
Updated the PR description/test plan to cover TestAvroEncoderUtil and both round-trip tests, and to note this closes the parquet-avro object model gap (the Parquet: Read and write geometry and geography WKB values #16982 follow-up).

szehon-ho

Clean, minimal follow-up to #16982. Mapping geometry/geography to Avro bytes in TypeToSchema is the right fix — value readers/writers already dispatch on the physical BYTES type, so no value-path changes are needed. Test coverage is thorough (shared DataTest geo cases, encoder parity, and explicit WKB round-trips for both Avro and parquet-avro paths). A couple of minor nits inline.

szehon-ho · 2026-07-14T18:50:02Z

+        .isEqualTo(geoRecords);
+  }
+
+  private static ByteBuffer wkbPoint(double xCoord, double yCoord) {


🟢 Nit — The local wkbPoint wrapper is fine (matches TestParquetDataWriter), but you could inline ByteBuffer.wrap(RandomUtil.wkbPoint(...)) the way TestParquet.testGeospatialWkbRoundTrip does and drop the helper. Optional either way.

Leaving the local wkbPoint helper as-is: it delegates to RandomUtil.wkbPoint and matches the merged TestParquetDataWriter.wkbPoint precedent (the same one-place-for-WKB consistency wombatu-kun pointed to above), and it keeps the three call sites in the record list readable. Happy to inline if you feel strongly.

Both map to the Avro bytes schema; sharing the BINARY case block drops a comment that restated the code and matches how RandomInternalData groups these types. Addresses review nit.

szehon-ho · 2026-07-17T01:47:01Z

Merged, thanks @huan233usc , and also @huan233usc @wombatu-kun for additional review !

github-actions Bot added core data labels Jul 7, 2026

wombatu-kun reviewed Jul 7, 2026

View reviewed changes

Comment thread core/src/test/java/org/apache/iceberg/avro/TestGenericAvro.java

huan233usc force-pushed the geo-avro-schema branch from 500ea8d to 7b2cc6d Compare July 7, 2026 03:19

huan233usc force-pushed the geo-avro-schema branch from 7b2cc6d to 67619a7 Compare July 7, 2026 03:28

github-actions Bot added the parquet label Jul 7, 2026

huan233usc requested a review from wombatu-kun July 7, 2026 17:23

nssalian requested review from szehon-ho and removed request for wombatu-kun July 11, 2026 18:57

nssalian approved these changes Jul 11, 2026

View reviewed changes

nssalian requested a review from wombatu-kun July 11, 2026 18:57

wombatu-kun reviewed Jul 12, 2026

View reviewed changes

Comment thread core/src/test/java/org/apache/iceberg/avro/TestAvroDataWriter.java

Comment thread parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java

Comment thread parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java Outdated

huan233usc requested a review from wombatu-kun July 12, 2026 19:48

szehon-ho approved these changes Jul 14, 2026

View reviewed changes

Core: Merge GEOMETRY/GEOGRAPHY into the BINARY case in TypeToSchema

64993fe

Both map to the Avro bytes schema; sharing the BINARY case block drops a comment that restated the code and matches how RandomInternalData groups these types. Addresses review nit.

huan233usc mentioned this pull request Jul 15, 2026

Spark 4.1: [WIP] Geometry / Geography end-to-end support #16650

Closed

szehon-ho approved these changes Jul 17, 2026

View reviewed changes

szehon-ho merged commit 8550723 into apache:main Jul 17, 2026
36 checks passed

nssalian added this to the Iceberg 1.12.0 milestone Jul 20, 2026

Navigation Menu

Uh oh!

Core: Read and write geometry and geography values in Avro - #17119

Core: Read and write geometry and geography values in Avro#17119
szehon-ho merged 3 commits into
apache:mainfrom
huan233usc:geo-avro-schema

huan233usc commented Jul 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huan233usc commented Jul 12, 2026

Uh oh!

szehon-ho left a comment

Uh oh!

Uh oh!

szehon-ho Jul 14, 2026

Uh oh!

huan233usc Jul 14, 2026

Uh oh!

Uh oh!

szehon-ho commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

huan233usc commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Note on the reverse direction

Test plan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huan233usc commented Jul 12, 2026

Uh oh!

szehon-ho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

szehon-ho Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

huan233usc Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

szehon-ho commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

huan233usc commented Jul 7, 2026 •

edited

Loading