Skip to content

Spark 4.1: [WIP] Geometry / Geography end-to-end support - #16650

Closed
huan233usc wants to merge 2 commits into
apache:mainfrom
huan233usc:spark-geo-e2e
Closed

Spark 4.1: [WIP] Geometry / Geography end-to-end support#16650
huan233usc wants to merge 2 commits into
apache:mainfrom
huan233usc:spark-geo-e2e

Conversation

@huan233usc

@huan233usc huan233usc commented Jun 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Note

Closing as done. This was a WIP / tracking PR that wired Iceberg's geometry and
geography types end-to-end on Spark 4.1 to prove the full CREATE / INSERT / SELECT / DELETE path over Parquet. That goal is complete: every piece has since landed on main
through focused, individually reviewed sub-PRs. Nothing here remains to merge.

What landed, and where

The end-to-end change was split into these merged PRs:

Layer Merged PR
API — single-value binary serialization (Conversions) #16607
Parquet — map geometry/geography to Parquet logical types (TypeToMessageType, MessageTypeToType) #16765
Parquet — skip lexicographic footer bounds for geo (ParquetMetrics) #16850
Parquet — read/write WKB values (BaseParquetReaders / BaseParquetWriter) #16982
Core — reland the geo metrics counts-without-bounds test #17147
Spark 4.1 — map geo Spark types (TypeToSparkType, SparkTypeToType, PruneColumnsWithoutReordering) #16851
Spark 4.1 — read/write geo values in Parquet (SparkParquetReaders / SparkParquetWriters) #17073
Spark 4.1 — DML with deletion vectors, nested geo, nulls; vectorized-read fallback #17149

The large in-progress TestSparkGeoTypes here was not merged as-is; its coverage was
reworked into TestSparkGeospatial plus the DML/vectorization-fallback tests in #17149.

Follow-ups (tracked separately)

Thanks to everyone who reviewed the split-out PRs.

@huan233usc

huan233usc commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

cc @szehon-ho

Xin Huang added 2 commits June 8, 2026 11:14
Xin Huang
API: Single-value binary serialization for geometry and geography
a95719c 
Iceberg v3 stores geometry and geography lower/upper bounds as binary using
the x:y:z:m encoding defined in the Bound Serialization section of the spec
(Appendix D). The encoding is already implemented in GeospatialBound, but
Conversions.toByteBuffer / fromByteBuffer have no GEOMETRY / GEOGRAPHY cases
and throw UnsupportedOperationException, blocking ManifestEvaluator and any
metric-based use of geo bounds.

Wire GeospatialBound through Conversions for both geometry and geography,
and add round-trip coverage in TestConversions for the 16-, 24-, and 32-byte
shapes (including the x:y:NaN:m case for XYM bounds), CRS variants, and
nulls.
Xin Huang
Spark 4.1: WIP geometry / geography end-to-end support
201032e 
Build on top of apache#16607 to enable end-to-end CREATE / INSERT / SELECT /
DELETE on Iceberg geometry and geography columns from Spark 4.1.

Parquet schema mapping
- TypeToMessageType: emit Iceberg GEOMETRY / GEOGRAPHY as Parquet BINARY
  with LogicalTypeAnnotation.geometryType / geographyType, propagating
  CRS and (for geography) the EdgeAlgorithm.
- MessageTypeToType: read those annotations back into Iceberg
  Types.GeometryType / Types.GeographyType.
- ParquetMetrics: skip lex min/max bounds for geometry / geography
  (Comparators.forType has no ordering for spatial WKB) and fall back to
  value / null counts only. Spatial bounding boxes (X:Y:Z:M) will be
  plumbed through FieldMetrics in a follow-up.

Generic Parquet readers / writers
- BaseParquetReaders / BaseParquetWriter: dispatch the geometry and
  geography logical type annotations to ParquetValueReaders.byteBuffers
  and ParquetValueWriters.byteBuffers, surfacing WKB as ByteBuffer for
  the engine-agnostic data path.

Spark 4.1 type bridge
- TypeToSparkType / SparkTypeToType: bidirectional mapping between
  Iceberg GEOMETRY / GEOGRAPHY and Spark's GeometryType / GeographyType,
  preserving the CRS string. Geography defaults to the SPHERICAL edge
  algorithm on the Iceberg side.
- PruneColumnsWithoutReordering: register GEOMETRY and GEOGRAPHY in the
  type-id table so column pruning does not reject geo columns.

Spark 4.1 Parquet readers / writers
- SparkParquetReaders: new GeometryReader / GeographyReader that read
  pure WKB from Parquet and prepend the 4-byte little-endian SRID header
  expected by Spark's internal GeometryVal / GeographyVal. The SRID is
  derived from the column's CRS via Spark's CartesianSpatialReference-
  SystemMapper / GeographicSpatialReferenceSystemMapper.
- SparkParquetWriters: new GeometryWriter / GeographyWriter that strip
  Spark's SRID header and write only the WKB body to Parquet, matching
  the Iceberg / Parquet on-disk representation.

Tests
- TestSparkGeoTypes (Spark 4.1): 11 end-to-end SQL tests covering
  - flat geometry / geography round-trip,
  - ST_Srid predicate (validates SRID re-attachment from CRS),
  - DELETE on a v3 merge-on-read table producing a Puffin DV,
  - NULL geometry mixed with non-NULL,
  - mixed geometry + geography in the same table,
  - STRUCT<..., loc: GEOMETRY>, ARRAY<GEOMETRY>, MAP<STRING, GEOMETRY>,
    STRUCT<..., points: ARRAY<GEOMETRY>>,
  - DELETE + DV on a table whose geometry sits inside a struct.
  Vectorized reads are disabled per-table because the vectorized geo
  path is not yet implemented.

Topological predicates such as ST_Intersects are not part of stock
Spark 4.1, so predicate coverage is limited to ST_Srid for now.
huan233usc
huan233usc commented Jun 8, 2026
View reviewed changes
Comment thread spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkGeoTypes.java

insertGeometry(GEOMETRY_TABLE, 1L, point2D(1.0, 2.0));
insertGeometry(GEOMETRY_TABLE, 2L, point2D(3.0, 4.0));

@huan233usc huan233usc Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: look up metadata

This was referenced Jun 11, 2026
Merged
Merged
Merged
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Jul 9, 2026
@nssalian nssalian added this to the Iceberg 1.12.0 milestone Jul 15, 2026
@huan233usc

huan233usc commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Closing this WIP tracking PR — the end-to-end geometry/geography support it prototyped has all landed on main via the split-out PRs listed in the (updated) description (#16607, #16765, #16850, #16982, #17147, #16851, #17073, #17149). Remaining work continues in #17119 (Avro) and #17161 (bounds). Nothing left to merge here.

@huan233usc huan233usc closed this Jul 15, 2026
@nssalian nssalian removed this from the Iceberg 1.12.0 milestone Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

API parquet spark stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@huan233usc @nssalian