WIP: End-to-end spatial file pruning from Spark SQL (ST_INTERSECTS pushdown)

[huan233usc]

Important

Draft / proof-of-concept, stacked on #17161 (geometry bounding-box metrics — the first
commit here is that PR; review it there). This branch demonstrates end-to-end spatial
file pruning from a Spark SQL query. API shapes (predicate, operation, the
st_intersects function signature) are a sketch for discussion, not a final proposal.
Not intended to merge as-is.

What it proves

A Spark SQL query with a spatial filter now prunes files by their geometry bounding box:

SELECT id FROM t WHERE system.st_intersects(geom, minX, minY, maxX, maxY)

TestSpatialFilterPushdown (spark-extensions) writes two geometry files with disjoint
clusters and shows:

• the plan carries the predicate in the Iceberg scan's pushed filters= list;
• a query window over one cluster returns only that cluster's row — the disjoint file is
  skipped during planning.

The full chain (each link was the tricky part)

1. Expression API (api): Operation.ST_INTERSECTS, Unbound/BoundSpatialPredicate
   carrying a query BoundingBox (a spatial window, not a scalar Literal),
   Expressions.stIntersects(...), and dispatch in ExpressionVisitors.
2. File pruning (InclusiveMetricsEvaluator): reads the file's geometry
   lower_bounds/upper_bounds and calls
   GeospatialPredicateEvaluators.intersects(query, fileBbox); disjoint ⇒ ROWS_CANNOT_MATCH.
   (TestSpatialMetricsEvaluator covers this in isolation.)
3. Every visitor: a new predicate kind ripples through every ExpressionVisitor
   subclass — Projections, Binder, RewriteNot, ExpressionUtil sanitizers,
   Spark3Util describe, and the three Parquet row-group filters all need a
   spatialPredicate case (both Bound and Unbound overloads), or they NPE. Enumerated here.
4. Serialization: BoundingBox/GeospatialBound made Serializable so a pushed
   predicate ships to executors.
5. Spark function: system.st_intersects(geom, minX, minY, maxX, maxY) registered as a
   ScalarFunction (with the magic invoke static method and a CLASS_TO_FUNCTIONS
   entry — both required for pushdown).
6. Optimizer rewrite (ReplaceStaticInvoke, spark-extensions): a top-level boolean
   scalar-function filter is left by Spark as a StaticInvoke; the rule now rewrites it to
   an ApplyFunctionExpression that Spark can translate to a pushable DSv2 predicate.
   (Previously it only rewrote functions inside comparisons like bucket(col) = 5.)
7. Pushdown conversion (SparkV2Filters): the boolean UDF arrives wrapped as
   V2Predicate("BOOLEAN_EXPRESSION", [UserDefinedScalarFunc]); unwrap it and build the
   Iceberg ST_INTERSECTS expression.
8. Spark writer: wired the geometry bbox metrics into SparkParquetWriters (in
   addition to the generic-data path from Parquet: Compute geometry bounding box metrics #17161), so Spark-written geo files carry a bbox
   to prune against.

Scope / caveats

• Function signature takes an explicit (minX, minY, maxX, maxY) window (doubles) so the
  pushdown extraction is trivial; a real design would take a constant geometry.
• Row-group-level spatial pruning is not implemented (row-group filters conservatively keep
  the group); file-level pruning does the work. The row-level invoke returns true (a PoC
  stub); correctness for the returned rows relies on file pruning here.
• Requires the Iceberg Spark extensions (for ReplaceStaticInvoke).

Opening as a draft to anchor the spatial-predicate API and the pushdown design.