[SPARK-59699][CONNECT] Make DataFrameReader options case-insensitive

[haoyangeng-db]

What changes were proposed in this pull request?

This PR makes Spark Connect's Python DataFrameReader.option handling case-insensitive.
When a new option matches an existing key ignoring case, it removes the existing entry and stores
the newest spelling and value.

It also adds plan serialization coverage for both named-table and data-source reads.

Why are the changes needed?

Spark reader option keys are case-insensitive and later calls should override earlier values.
Spark Connect currently stores options in a regular Python dictionary, so differently cased forms
of the same logical key can both reach the protobuf map. The server then resolves those duplicates
from an unordered map, which can select a value that does not match the user's call order.

For example, this should deterministically produce one versionAsOf option with value 2:

spark.read.option("versionAsOf", 0).option("versionasof", 1).option("versionAsOf", 2)

The fix preserves the spelling of the most recent key instead of lowercasing keys, so vendor option
names such as oracle.jdbc.timezoneAsRegion remain unchanged.

Does this PR introduce any user-facing change?

Yes. Spark Connect Python reader options now follow case-insensitive, last-call-wins semantics,
matching the classic DataFrameReader behavior. Workloads that supplied the same logical option
using multiple casing variants now deterministically use the most recently supplied value.

How was this patch tested?

• Added regression coverage in SparkConnectPlanTests for mixed-case named-table options and
  repeated data-source options.
• Ran pyspark.sql.tests.connect.test_connect_plan: 83 tests passed before the clean rebase onto
  the latest upstream/master; the rebase did not change the patch.
• Ran Python compilation checks for both modified files after rebasing.
• Ran git diff --check and line-length checks after rebasing.
• A post-rebase suite rerun was blocked because the fresh worktree lacks the optional Connect test
  dependencies, and installing them was blocked by DNS resolution to PyPI.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex 0.154.0

[HyukjinKwon]

Review summary

Reviewed the case-insensitive option dedup in the Connect DataFrameReader and the added plan-serialization test. The dedup correctly reproduces classic DataFrameReader / server-side CaseInsensitiveMap last-call-wins semantics for the supported input domain and preserves the latest key's spelling rather than lowercasing it; options(), load(), table(), and changes() all funnel through option(), so they uniformly inherit the new behavior. The delete-then-assign sequence is safe for every supported value: to_str only wraps str() for str/bool/int/float and returns None for None, none of which raise, so no supported input can leave the reader partially mutated after the deletes. Connect's lack of a None short-circuit (classic's option() has if value is None: return self) predates this PR and is unaffected by it. The added test asserts the exact materialized proto option maps on both the named-table (table()) and data-source (load()) paths, so a wrong winner, a lowercased key, or a surviving duplicate case-variant would fail rather than pass silently. No findings.

Findings

0 total: 0 P0, 0 P1, 0 P2, 0 P3.

No findings.

[HyukjinKwon]

Merge Summary:

• merged into master 97d67a9
• merged into branch-4.x ab46b73

Posted by merge_spark_pr.py