[SPARK-59358][SDP] SCD2 Ignore-null support; Coalesce ignored-null values - #58947

Open

AnishMahto wants to merge 21 commits into

apache:masterfrom

AnishMahto:SPARK-59358-coalesce-ignored-null-columns

Commits on Sep 14, 2026

cleanup and unify helpers to Scd2VersionMap
AnishMahto
committed
df0e827
delete rows in microbatch receive null version maps
AnishMahto
committed
7990d91
document risk for changing ignore-null selection between microbatch retries
AnishMahto
committed
ca3d77b
use withField
AnishMahto
committed
ea8a865
use JSON serialization for version map keys
AnishMahto
committed
f38e3f9
cleanup
AnishMahto
committed
ff2d9b7
fix delete detection
AnishMahto
committed
52158a3
self-review
AnishMahto
committed
4ed70f9
refactor out generic extractLeafPaths
AnishMahto
committed
08674a2
[SPARK-59183][SDP] Handle reductive evolution in version maps
AnishMahto
committed
bd2502a
[SPARK-59183][SDP] Pin version map key casing
AnishMahto
committed
9b003c9

Commits on Sep 15, 2026

PR feedback
AnishMahto
committed
7a96726
use QuotingUtils.quoteNameParts instead of json array for serialization
AnishMahto
committed
db52c5d

Commits on Sep 17, 2026

cleanup documentation
AnishMahto
committed
f61c193
move schema-column filtering to AutoCdcSchemaUtils
AnishMahto
committed
d00936a
linting
AnishMahto
committed
e342846

Commits on Sep 18, 2026

Commits on Sep 21, 2026

add tests
AnishMahto
committed
7b5dff2
explain first upsert row in reconciliation case
AnishMahto
committed
0ff76b9