[Data] Fix RCE where attackers can execute arbitrary code with read_lance or nested pickle objects

[bveeramani]

Description

The Ray Data object extension type uses pickle to serialize and deserialize Python objects. When you read Parquet or Lance datasets from untrusted sources, attackers can execute arbitrary code when Ray Data unpickles objects.

I previously landed a PR to mitigate this issue for read_parquet by requiring users to opt-in to unpickling during reads.

In this PR, I'm making two related fixes:

1. Making sure we also check for pickled objects in nested structures like lists
2. Applying the opt-in guard to `read_lance

Related issues

Related to https://github.com/ray-project/ray/security/advisories/GHSA-whgc-7wqm-hx2r and https://github.com/ray-project/ray/security/advisories/GHSA-7h5h-hv6f-qjw9

[gemini-code-assist]

Code Review

This pull request consolidates and strengthens the security guard against reading unsafe ray.data.arrow_pickled_object columns by introducing a centralized, recursive check (raise_on_pickle_object_columns) and applying it to both Parquet and Lance datasources. It also adds tests to ensure nested pickled-object columns are properly rejected. The review feedback highlights a critical security vulnerability in the recursive type-checking logic, where dictionary-encoded columns or custom extension types could bypass the guard, and provides a robust implementation to address this.

[ayushk7102]

Nit: Does it make sense to take these test setup steps (initializing ext_type, storage) out into a common shared helper for test_lance and test_parquet?

[bveeramani]

I think that'd be reasonable.

Since there are only two locations where those setups are used now, I'm going to accept some duplication to avoid indirection in those tests.