Shrink the xtable-utilities bundled jar

[slachiewicz]

Feature Request / Improvement

xtable-utilities is 795 lines of Java in two classes, RunSync and RunCatalogSync. It builds a 1.1 GB bundled jar: 3.1 GB uncompressed, 582,980 entries, 483 whitelist entries.

Measured by top-level package:

	MB uncompressed	files
software/amazon/awssdk	1957	379,410
org/apache/hadoop	258	53,612
org/apache/spark	91	12,129
org/apache/paimon	86	18,840
RocksDB natives	159	13
io/grpc/xds	40	4,805
it/unimi/dsi	39	10,665
org/apache/hudi	17	2,890
org/apache/iceberg	12	3,126

The formats XTable exists to translate are about 1% of the jar.

This is the "reduce the size of bundled jar" step from the 2024 [DISCUSS] thread on excluding xtable-utilities from the release, where @vinothchandar suggested moving to the Hudi model of whitelisting what compute engines do not already provide. The whitelist has grown rather than shrunk since.

What can be cut

Four changes, no source touched, all measured:

1. software.amazon.awssdk:bundle is the all-services jar. hadoop-aws 3.4.1's S3A references exactly one service package, services/s3, plus STSClient, and the transfer, http, core, awscore and auth packages. Nothing else. Verified by scanning the constant pools of all 418 org.apache.hadoop.fs.s3a classes. Replacing bundle with s3, sts, netty-nio-client and s3-transfer-manager takes the SDK from 1957 MB to 50 MB.
2. RocksDB ships a native library per platform. Dropping ppc64le, s390x and 32-bit Linux removes 84 MB. x86_64 and aarch64 keep their musl builds; macOS and Windows stay for development.
3. io.grpc:grpc-xds is 40 MB, referenced only from gRPC's own code, and reachable only through xds:// targets.
4. org.codehaus.groovy:groovy-all 2.4.11 arrives via Hive; nothing outside Groovy references it.

Together: 1098 MB to 423 MB, entries 582,980 to 203,041.

What cannot be cut without a design change

Spark's 91 MB is load-bearing today: xtable-core's Delta support uses delta-core, a Spark library, across 12 classes (SparkSession, DeltaLog, StructType). delta-kernel-api and delta-kernel-defaults are already dependencies at 4.0.0. Kernel is the Spark-free Delta API, so migrating that code would let Spark and much of its transitive weight go. That is a larger piece of work and relates to #611.

hadoop-azure and the GCS connector have no static referrer, but they are ServiceLoader-loaded filesystems, so their absence would be a lost feature rather than dead weight.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

This issue was created with AI assistance.