[SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output

[srielau]

What changes were proposed in this pull request?

Support first-class CHAR/VARCHAR output from script TRANSFORM when first-class semantics are enabled. Split out from #58584 (Hive inspector conversion + UDF/UDAF/UDTF) per review feedback.

• No-SerDe path: CHAR/VARCHAR scalar, nested, and complex output with padding, overflow (EXCEED_LIMIT_LENGTH), null-token, and malformed-field handling.
• No-SerDe nested maps: Restore JSON string keys to the declared physical key type through a per-row ArrayBasedMapBuilder that validates null and duplicate converted keys.
• SerDe path: Rewrite LazySimpleSerDe CHAR/VARCHAR output types to STRING so Spark applies first-class length checks. Non-LazySimpleSerDe SerDes keep the declared CHAR/VARCHAR schema.
• Legacy: Legacy flag-off behavior remains unchanged.

This PR depends on #58584 for Hive inspector CHAR/VARCHAR conversion. JIRA: https://issues.apache.org/jira/browse/SPARK-59683

Why are the changes needed?

With spark.sql.charVarchar.standardSemantics.enabled=true, script TRANSFORM output was not aware of CHAR/VARCHAR types. Bounded-string values passed through without padding or length enforcement, and nested output containing non-string map keys was not restored from the JSON string representation.

Does this PR introduce any user-facing change?

Yes. When first-class CHAR/VARCHAR semantics are enabled, script TRANSFORM now pads CHAR output, enforces length limits on both CHAR and VARCHAR, and correctly handles nested CHAR/VARCHAR in arrays, maps, and structs. Legacy flag-off behavior remains unchanged.

How was this patch tested?

Added focused coverage for:

• Scalar CHAR/VARCHAR output without SerDe (padding, overflow).
• Nested CHAR/VARCHAR (arrays, maps with non-string keys, structs) without SerDe.
• Malformed JSON and colliding map-key restoration without SerDe.
• Colliding row followed by valid row in the same partition (builder-leak regression).
• Nested collated CHAR/VARCHAR with Hive SerDe.
• Scalar CHAR/VARCHAR overflow with Hive SerDe.
• LazySimpleSerDe-only CHAR/VARCHAR type rewrite verification.

Ran:

sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
  -Dsbt.override.build.repos=true \
  'sql/testOnly org.apache.spark.sql.execution.SparkScriptTransformationSuite -- -z SPARK-59277'

sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
  -Dsbt.override.build.repos=true \
  'hive/testOnly org.apache.spark.sql.hive.execution.HiveScriptTransformationSuite -- -z SPARK-59277'

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor Auto