Backport candidates for 1.0.x: triage of every PR merged since branch-1.0 was cut

[andygrove]

Describe the problem

branch-1.0 was cut at af534e0f (2026-08-03). Since then, 192 commits have landed on main and only three
backport batches have gone to branch-1.0 (#5261, #5321, #5480). I went through every one of those 192
commits to work out which are genuine backport candidates for a 1.0.x patch release — pure bug fixes and
performance improvements that neither add features nor change intended behaviour — and which are not, and why.

This issue is the resulting list. It is a triage record, not a decision: the recommendations still need
someone to agree with them and open the backport PRs.

How the list was produced

• Range: git log af534e0f..main, 192 commits.
• 85 carry a fix: or perf: prefix. I also checked all 47 feat: / refactor: / deps: commits for ones
  that close a bug-labelled issue, which is how feat: support unicode case sensitive field names for reading parquet #5602 ended up in tier 1 despite its feat: prefix.
• Prefixes are not trustworthy on their own in either direction, so every candidate was checked against the
  label on the issue it closes. Five fix:-prefixed PRs close enhancement issues and are excluded below.
• Applicability was checked mechanically: each commit was cherry-picked onto a pristine branch-1.0 worktree,
  and every conflicting path was tested for existence on branch-1.0. A conflict in a file that does not
  exist on branch-1.0 means the PR fixes code that only exists on main — not a candidate. A conflict in a
  file that does exist is ordinary context drift and still a candidate.
• branch-1.0 is on DataFusion 54.1.0 / Arrow 58.4.0; main is on 55.0.0 / 59.2.0 since deps: bump DataFusion 55.0 and Arrow/Parquet 59.2 #5262. That rules
  out at least one otherwise-attractive fix.

Applying the recommended set as a stack in merge order, 27 of 37 commits cherry-pick clean and 10 need small
manual resolution. Cherry-picking clean is not the same as compiling — #5461 is a known example — so each
backport PR still needs cargo check --workspace --all-targets plus ./mvnw test-compile before it goes up.

Tier 1 — correctness, crashes and hangs

These change observable results, fail tasks, leak, or hang. I would take all of these.

PR	What it fixes	Why it matters on 1.0.x	Cherry-pick
#5602	Unicode case-insensitive Parquet field matching	Closes #5495 — bug, correctness, priority:critical, and already in the 1.0.1 milestone. Non-ASCII case pairs silently read as NULL	clean
#5623	Codegen dispatcher null short-circuit skipped when a foldable subtree can raise	Closes #5608 — bug, correctness, priority:high. Swallows ANSI errors	clean
#5159	collect_list / collect_set argument coercion made a normalization barrier	Closes #5158 — bug, priority:high. Grouped aggregate fails validating its own output batch	clean
#5757	Dictionary hash fast path kept off nested and reseeded buffers	Dictionary-encoded values hashed differently from identical decoded values — wrong join/aggregate results and wrong shuffle partitioning	clean
#5754	Parent struct's null mask applied before hashing its fields	Wrong answers from hash / xxhash64 on a null struct; the same decision feeds shuffle partitioning	apply after #5757
#5684	Correctly rounded decimal to double/float cast	Last-ulp wrong answers for essentially every DECIMAL(38,18) value above 0.01	clean
#5683	ANSI bound check for float/double to integral casts	Spurious CAST_OVERFLOW for exactly representable Int.MaxValue / Long.MaxValue	clean
#5752	ANSI errors preserved for rejected TIMESTAMP_NTZ casts	Closes #5749 — bug, correctness. Returns NULL where Spark raises CAST_INVALID_INPUT	clean
#5177	Checked TIMESTAMP_MILLIS conversion in Parquet reads	Silently wraps overflowing values; Spark throws regardless of ANSI mode	rebase (parquet_support.rs)
#5740	Same check for nested TIMESTAMP_MILLIS in unfiltered scans	Companion to #5177; take the two together	rebase (parquet_support.rs)
#5680	rpad / lpad return NULL for a NULL length instead of panicking	Enabled by default; a NULL in the length column fails the task with CometNativeException	clean
#5630	Map offsets rebased in mapsort	A sliced map fails the task with "Max offset exceeds length of entries"	rebase (test file only)
#5679	concat_ws with array arguments falls back instead of failing natively	Query fails natively where Spark returns a result	clean
#5357	Spark type names in ANSI abs overflow errors	Reports Int64 overflow where Spark reports long overflow	clean
#5747	copy_array propagates offset-overflow errors instead of panicking	Closes #5706 — bug. Panic instead of a clean error	rebase (copy.rs)
#5539	Arrow vector import rollback on failure	Closes #5534 — bug, correctness, priority:medium. Leaks native buffers, adding allocator pressure while the query is already failing	rebase (test file only)
#5748	Tokio runtime threads attached to the JVM as daemon threads	The JVM hangs forever if an app returns from main without calling spark.stop(). #4734, which is on branch-1.0, does not cover this case	rebase (CI yml only)
#5561	Spark 4.2 Python worker configuration	branch-1.0 ships a spark-4.2 profile and writes workerConf inside writeCommand, so accelerated mapInArrow / mapInPandas fails outright on 4.2	clean
#5759	Iceberg tables partitioned by an unknown transform	Read fails with Invalid Iceberg scan task; this is Iceberg's forward-compatibility contract, and it is what turned Iceberg CI red on main	rebase
#5493	CometDiskBlockWriter spill registry made per-task	The registry is static, so task A force-spills task B's buffered bytes from A's thread and charges them to A	clean
#5494	Task-shared memory pool as a ref-counted RAII guard	A stranded entry holds a JNI global ref to CometTaskMemoryManager, pinning TaskMemoryManager and TaskContext for the executor's lifetime. Nothing prunes the map	rebase (CI yml, jni_api.rs)
#5461	Native shuffle reservation released after spill failure	Reservation, pinned buffers and repeated-spill bytes leak after a failed partition write	clean, but does not compile as-is: its inline test needs metric fields from #5370, so the test needs adapting

Tier 2 — performance, no behaviour change

PR	What it does	Cherry-pick
#4885	spark_base64: 11–21% faster, bit-identical output	clean
#4911	spark_floor: up to 4x faster on decimals	clean
#4939	Vectorized integer-to-decimal cast — the second most frequent cast in TPC-DS	rebase (bench registration in Cargo.toml)
#5233	spark_size list lengths via Arrow's length kernel	clean
#5395	spark_size Map via the offset buffer; follow-up to #5233, take them together	after #5233
#5046	ArrowWriter preallocates fixed-width vectors from the known batch size	clean
#5442	ArrowWriter bulk-copies fixed-width columns; follow-up to #5046, take them together	after #5046
#5440	Arrow cast for the decimal rescale check, replacing duplicated arithmetic	clean
#5735	Arrow cast for date to TIMESTAMP_NTZ	rebase (temporal.rs)
#5737	posexplode array expression evaluated once per batch instead of twice	clean
#5222	Iceberg reflection lookups cached on the planning path — currently O(files) Class.getMethod per scan, and on the Spark 3.4 profile (Iceberg < 1.7) every call constructs a NoSuchMethodException	clean
#5568	Per-partition scratch reused in the shuffle write path	rebase
#5038	Arrow IPC compression context reused across shuffle blocks	clean

Two caveats in this tier. #5038's measured benefit is narrower than it looks: the follow-up measurement in
#5727 shows 2.0–16.8% faster block encoding for 128-row batches and approximately no change for full-size
compressed batches. And #5368 (serialize Python input directly from Comet Arrow vectors) is the largest win
available here, but it is 10 files, it regressed timezone aliases, and the fix for that (#5556) is a separate
PR — take both or neither.

Tier 3 — needs a decision before it can be classified

• fix: match Spark's null short-circuiting in array_join and enable it natively #5558 (array_join null short-circuiting) is a real correctness fix — Spark returns NULL whenever
  nullReplacement is NULL, Comet returned the joined string — but it is bundled with flipping array_join
  from Incompatible to natively enabled. Backporting as-is changes a compatibility flag in a patch release.
  Worth splitting.
• fix: normalize scalar float sort and window rank keys #5469 (normalize scalar float sort and window rank keys) is half applicable. The sort-key normalization
  for signed zeros and NaN is a genuine correctness fix that applies to branch-1.0; the window-rank half
  follows WindowGroupLimitExec (feat: support WindowGroupLimitExec #4870), which is main-only. Needs splitting.
• fix: make CometExplodeExec respect batch size #5362 ("make CometExplodeExec respect batch size") is a real bug — UnnestExec never consults
  datafusion.execution.batch_size, so an 8192-row batch of 100-element arrays comes back as one
  819,200-row batch and peak memory scales with it. But the fix lands a new 1,491-line native ExplodeExec.
  That is feature-sized for a patch release.
• fix: distinguish "nothing spilled" from a spill backend with no local path #5726 (pathless spill backend writes an index claiming bytes that never landed) would be a corrupt
  shuffle file. On branch-1.0 the only spill backend is local and always has a path, so as far as I can
  tell the bug is latent there. Backport only if we want the invariant rather than the fix.
• fix: expand object store option references, uniquify constant metadata names, drop dead parquet JNI #5653 bundles three unrelated core fixes. One of them is genuinely user-visible — Configuration#get
  expands ${...} references and Configuration.Entry#getValue does not, so an object store option composed
  from another property reached native unexpanded. Extract that hunk rather than cherry-picking the PR.
• fix: RAII for memory pool registration  #5464 (RAII for tracing memory pool registration) is a real leak, but only reachable with native tracing
  enabled. Low value for a patch release.
• fix: preserve Arrow Field metadata across C Data exports #5552 (preserve Arrow Field metadata across C Data exports) closes an enhancement ([Variant] Export complete Arrow Fields through the native FFI boundary #5547) and is
  part one of the Variant series. Nothing on branch-1.0 consumes Field metadata, so there is no
  observable effect there.

Excluded, with reasons

Recording these so they do not get re-litigated for the next patch release.

Fixes code that only exists on main. The feature landed after the branch cut, so the fix has nothing to
apply to:

• Celeborn / RSS shuffle (features feat: add RSS partition writer and task-owned JNI callback (1/n) #5473, feat: add partition writer destinations to native shuffle plans (2/n) #5476, feat: add destination-aware native shuffle execution (3/n) #5481, feat: bind task-owned RSS callbacks to native shuffle plans (4/n) #5491, feat: add Celeborn shuffle manager and partition pusher (5/n) #5501, feat: complete Celeborn map-side shuffle push lifecycle (6/n) #5513, feat: add raw Celeborn native shuffle reader (7/n) #5531, feat: enable native-only Celeborn shuffle planning (8/n) #5537): fixes fix: tighten RSS writer visibility and JNI array limits #5475,
  fix: scope Celeborn bootstrap hooks to Comet clients #5627, fix: accept dictionary encodings in remote shuffle #5650, fix(celeborn): reject unsafe native push completion tracking #5665, fix: recover native Celeborn shuffle from oversized rows #5668, perf: cache expected schemas for remote shuffle decoding #5722, perf: validate shuffle IPC context reuse savings #5727
• Native Iceberg V2 writer (feat: implement native Iceberg V2 writer via iceberg-rust #5361) and the split writer/committer work (feat: Optionally split the Iceberg V2 write operator into distinct writer and committer operations #4658, feat: detect Iceberg V2 writes and emit fall-back reasons #5298): fixes fix: ignore reader-side parquet.hadoop.vectored.io.enabled in Iceberg native-write detection #5410, fix: Bump iceberg-rust so native Iceberg writes URL-escape partition paths #5651,
  fix: restore columnar transitions under the native Iceberg write #5696, fix: native Iceberg write panics on an evolved partition spec and on a timestamptz partition path #5729, fix: match iceberg-java's exception for unclustered input to a clustered Iceberg write #5779
• Iceberg system functions (feat: Support Iceberg system functions (bucket, truncate, years/months/days/hours) natively #5638): fix fix: dispatch Iceberg system functions wrapped as ApplyFunctionExpression #5773
• Experimental native in-memory cache (feat: add experimental native support for in-memory cache, disabled by default #5051): fixes fix: let AQE optimize queries over Comet caches #5733, perf: reduce allocations when collecting cache statistics #5734
• Native ExplodeExec (fix: make CometExplodeExec respect batch size #5362): fix perf: slice the child instead of gathering it when unnesting #5667
• CreateArray map literal (feat: support Map for CreateArray literal #5452): fix feat: address remaining issues for CreateArray #5766 — Fix remaining reported issues for arrays of map #5544 only becomes reachable once a folded map literal is
  admitted natively

Blocked on the dependency upgrade. #5041 (enable FIRST/LAST partial merge) depends on the root fix in
DataFusion 55.0.0, which reached main via #5262. branch-1.0 is on 54.1.0.

Spark UI metrics accuracy only — same call as for 1.0.1, these do not change query results or stability:
#5370, #5398, #5408, #5445, #5463, #5497, #5516, #5554, #5628.

fix: prefix but closes an enhancement issue: #5045 (#3389), #5225 (#5190), #5234 (#5096), #5359
(#5074), #5412 (#5258).

Test, benchmark or CI only: #5388, #5439, #5718, #5727, plus every test:, chore:, chore(deps),
docs: and ci: commit in the range.

Previously decided. #5369 (preserve Catalyst nullability and field IDs in native Parquet writes) is a real
bug, but spark.comet.parquet.write.enabled defaults to false and is documented as experimental on
branch-1.0.

Already on branch-1.0 via #5261, #5321 and #5480: #5138, #5162, #5167, #5185, #5192, #5282, #5315,
#5364, #5376, #5392, #5400, #5443.

Additional context

Related open 1.0.1 items with no fix on main to backport: #5366 (to_time / try_to_time still rejects
'T12' and '12:30:45.'; #5364 fixed only the whitespace half) and #4412 (CometHashAggregateExec and
AQEPropagateEmptyRelation).

Optional companion: #5769 refactors #5754 and #5757 into a shared helper. Not required, but it keeps
branch-1.0 closer to main's shape if we want that.