Skip to content

Parquet: Compute geometry bounding box metrics - #17161

Open
huan233usc wants to merge 7 commits into
apache:mainfrom
huan233usc:geo-parquet-bbox
Open

Parquet: Compute geometry bounding box metrics#17161
huan233usc wants to merge 7 commits into
apache:mainfrom
huan233usc:geo-parquet-bbox

Conversation

@huan233usc

@huan233usc huan233usc commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Computes file-level 2D bounding-box metrics for geometry columns written to
Parquet. The bounds are stored in lower_bounds/upper_bounds, making spatial
file pruning possible. This PR produces the bounds; expression and scan-planner
integration that consumes them is separate follow-up work.

This is the first slice of the geo bounds work (Phase 2), scoped to the clean,
unambiguous planar case.

Problem

The ordinary Parquet column min/max for WKB is a lexicographic byte bound, not a
spatial bound. Iceberg therefore diverts geometry/geography columns to
counts-only metrics in its current ParquetMetrics path. Although newer Parquet
metadata can represent geospatial statistics separately, Iceberg does not
currently consume those footer bounds. As a result, geometry data files do not
carry spatial bounds in Iceberg metadata.

Approach

Compute the box while values are written, using the existing writer-side
value-scanning metrics channel -- the same path float and double use to track
NaN counts that ordinary footer statistics cannot provide
(ParquetValueWriter.metrics() -> ParquetWriter.metrics()):

  • The generic Parquet GeometryWriter writes byte-identical WKB and folds each
    value's XY coordinates into a running box.
  • The Spark 4.1 GeometryWriter performs the same accumulation after converting
    Spark's typed GeometryVal to the pure WKB stored by Iceberg.
  • Both writers emit FieldMetrics<GeospatialBound> whose lower and upper corners
    serialize through the existing geometry Conversions case into
    lower_bounds/upper_bounds.
  • Value-scanned metrics take precedence over the counts-only footer branch, and
    the optional-field writer reconciles null counts, so the geometry builder sees
    only non-null values.

This does not change ParquetMetrics, ParquetWriter, or Conversions.

WKBBoundingBox (new in api/geospatial, next to GeospatialBound and
BoundingBox) is a pure-Java WKB coordinate scanner with no JTS dependency. It:

  • walks all OGC geometry types (POINT, LINESTRING, POLYGON, multi-geometries,
    and collections);
  • handles both byte orders per geometry, including mixed-endian collections;
  • reads past Z and M ordinates to produce an XY-only box;
  • skips NaN independently per coordinate dimension, as required by the Iceberg
    spec -- for example, POINT (1 NaN) contributes X=1, and another row may supply
    the missing Y bound;
  • emits bounds only after both X and Y have at least one non-NaN value, so
    POINT EMPTY contributes nothing; and
  • validates WKB defensively: truncation, invalid byte order or type, excessive
    nesting, and oversized counts fail with IllegalArgumentException rather than
    reading out of bounds.

Scope

GEOMETRY only, 2D (XY), planar. This PR writes file bounds but does not add a
spatial predicate to the Expression API or wire spatial pruning into scan
planning.

Deliberately left as follow-ups:

  • GEOGRAPHY bounds, including longitude periodicity, edge latitude extrema,
    numerical coverage guarantees, pole handling, and coordinate-range policy;
  • higher-dimensional Z/M bounds;
  • the v4 content_stats geo_lower/geo_upper bridge;
  • ORC and Avro geo bounds (Avro geo value I/O is supported separately, but does
    not produce spatial bounds);
  • CRS validation; and
  • avoiding WKB scanning when the selected MetricsConfig will not retain bounds.

Tests

  • TestWKBBoundingBox covers every geometry type; XY, Z, M, and ZM layouts;
    little-, big-, and mixed-endian inputs; nested and empty collections; empty
    children; outer/interior rings; degenerate and differently oriented polygons;
    per-axis NaN accumulation; infinities; malformed inputs; and nesting limits.
  • TestGeometryFieldMetrics covers cross-value aggregation, counts, and empty or
    no-value results.
  • TestMetrics.testMetricsForGeospatialTypes verifies that generic Parquet writes
    produce the expected geometry bounds while geography remains bounds-less.
  • TestSparkParquetWriter.testGeospatialRoundTrip verifies Spark 4.1 WKB
    round-trip, geometry bounds across multiple rows, null handling, and that
    geography still produces no bounds.

AI Disclosure

  • Model: GPT-5
  • Platform/Tool: Codex
  • Human Oversight: partially reviewed
  • Prompt Summary: Sync the PR with upstream main, resolve conflicts, preserve geo average-size metrics, and complete bbox regression coverage.

@szehon-ho szehon-ho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per-dimension NaN handling looks inconsistent with the spec's bounds rules for geometry.

The spec says null/NaN are skipped per coordinate dimension, and gives the example that POINT (1 NaN) contributes to X but not Y. A bbox is omitted only when a dimension has no valid values after aggregating across the whole file.

addXY currently skips the entire coordinate when either axis is NaN. That matches the single-geometry empty case, but not mixed files — e.g. POINT (1 NaN) + POINT (5 10) should yield bbox (1, 10)–(5, 10), while this implementation produces (5, 10)–(5, 10) (xmin too high). That can violate the manifest invariant that lower bounds must be ≤ all non-null, non-NaN values and lead to incorrect file pruning during scan planning.

Suggested fix: accumulate X and Y independently (update min/max only for non-NaN components), then emit a bbox only when both dimensions have at least one valid value. A test like POINT (1 NaN) + POINT (NaN 20)(1, 20)–(1, 20) would lock this in.

*/
public void addXY(double xCoord, double yCoord) {
if (Double.isNaN(xCoord) || Double.isNaN(yCoord)) {
return;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This skips the whole coordinate when either axis is NaN, but the spec skips per dimension. Consider accumulating X and Y independently so POINT (1 NaN) still contributes X=1 when other rows supply a valid Y.

Xin Huang added 6 commits August 3, 2026 20:06
The Parquet footer's lexicographic min/max over WKB bytes is not meaningful for
geometry, so geometry columns previously wrote counts only and could not be data
skipped. Scan each value's coordinates as it is written and accumulate a 2D (XY)
bounding box, emitting it as a FieldMetrics through the writer-side metrics
channel (the same path float and double use for NaN counts). The box's lower and
upper corners serialize into the existing lower_bounds and upper_bounds maps via
the geometry conversion.

A new pure-Java WKB coordinate scanner (WKBBoundingBox, in api/geospatial, no JTS
dependency) walks all OGC geometry types, skips Z and M, ignores NaN coordinates,
and validates the buffer defensively. Geography, higher dimensions, and the
content_stats geo struct bounds are left as follow-ups.
Skip NaN components independently so valid X and Y values from different coordinates still produce conservative file bounds.

Generated-by: Codex
Exercise empty polygons, empty collections, and interior rings to lock geometry bounding-box behavior across structural edge cases.

Generated-by: Codex
Cover Z, M, and ZM coordinate sequences and collections, empty children, and degenerate or differently oriented polygon rings.

Generated-by: Codex
Collect geometry bounds in the Spark Parquet writer so Spark SQL writes produce the same file metrics as the generic writer.

Generated-by: Codex
- Verify geometry metrics retain average WKB size while producing bounds\n- Exercise per-axis NaN accumulation through Spark SQL round-trip

Generated-by: Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants