[SPARK-59678][SQL] Fix pivot failing on array and struct columns with non-nullable nested fields

[jiwen624]

What changes were proposed in this pull request?

PivotTransformer checks and casts each pivot value against pivotColumn.dataType. This PR ignores nullability in that match: the check strips nullability from the value's type before comparing it with the pivot column's type, and values are evaluated against pivotColumn.dataType.asNullable. The If filters for non-PivotFirst aggregates fall back to asNullable only when the exact type can't be cast to, so existing plans are unchanged.

Why are the changes needed?

A pivot value fails analysis when its type allows nulls in a nested position where the pivot column's type does not, with an error that prints the same type on both sides:

 SELECT * FROM (SELECT named_struct('x', id) AS c, id AS v FROM range(2))
 PIVOT (sum(v) FOR c IN (named_struct('x', if(true, 1L, null))))
 -- [PIVOT_VALUE_DATA_TYPE_MISMATCH] Invalid pivot value '...': value data type
 -- struct<x:bigint> does not match pivot column data type struct<x:bigint>

The same happens with explicit DataFrame values, e.g. pivot($"a", Seq(Array(1.0d))) on a non-nullable-element array. Nullability is irrelevant here, as the value is a constant only compared against the column. (pivot() without values is covered by SPARK-59684.)

Note that for a non-nullable nested column, a value that can't be cast now fails with CAST_INVALID_INPUT (ANSI) or becomes null (non-ANSI), the same as the scalar and nullable cases, instead of PIVOT_VALUE_DATA_TYPE_MISMATCH.

Does this PR introduce any user-facing change?

Yes. Queries that failed with PIVOT_VALUE_DATA_TYPE_MISMATCH only because the value's type allows nulls in a nested position where the pivot column's type does not now run. Queries that already passed analysis are unaffected, and other rejected values keep the same error.

How was this patch tested?

Added UT cases.

Was this patch authored or co-authored using generative AI tooling?

Yes

[jiwen624]

Hi @uros-b @cloud-fan since you've reviewed pivot-related fixes recently, could you let me know what do you think about this fix when you get a chance? Thanks! 🙇