[albertlieyingadrian]

What's wrong

With pandas >= 3, the default string dtype is nullable, and anndata >= 0.11 writes such columns, including the obs and var index, as a nullable-string-array group with values and mask datasets:

obs/_index    Group {'encoding-type': 'nullable-string-array'}  ['mask', 'values']
var/gene_name Group {'encoding-type': 'nullable-string-array'}  ['mask', 'values']

PerturbationDataset reads these with self.h5_file[...][:]. On a group that raises:

TypeError: Accessing a group is done with bytes or str, not <class 'slice'>

so get_gene_names() and barcode loading (barcode=True) fail on any h5ad written with current pandas/anndata. pyproject.toml doesn't cap either package and uv.lock is gitignored, so a fresh uv sync --all-extras --dev (as CI does) gets pandas 3.1.1 and anndata 0.13.4. On main, tests/test_dataloader.py::test_barcode_functionality fails this way. Categorical columns are unaffected because anndata still writes their categories as a dataset.

Fix

A small read_h5_column() helper in utils/data_utils.py:

• returns node[:] for a plain dataset (the old layout, unchanged);
• returns values for a nullable-string-array group;
• raises a ValueError that names the encoding for any other group, so it doesn't fall silently into the KeyError fallbacks.

get_gene_names() (var/gene_name, var/gene_name_index, var/_index, var/feature) and _load_cell_barcodes() (obs/_index) now read through it.

Tests

New tests/test_h5_columns.py:

• the helper on a plain dataset and on a hand-built nullable-string-array group;
• an end-to-end case that writes an AnnData with the installed anndata (nullable strings enabled), asserts the file really holds groups, then checks PerturbationDataset.get_gene_names() and _load_cell_barcodes(). It is skipped on anndata < 0.11.

Results:

• pandas 3.1.1 / anndata 0.13.4: test_barcode_functionality fails on main and passes here. The new tests and all of test_dataloader.py pass (17 passed). On main, the same script raises the TypeError above from get_gene_names().
• pandas 2.3 / anndata 0.11: same tests pass.
• ruff check and ruff format --check are clean on the touched lines.

Three test_mapping_strategy.py tests (test_backwards_compatibility_uncached, test_cache_parameter_effectiveness, test_register_split_indices_groups_by_batch_and_cell_type) fail on main for an unrelated reason: the control pool is keyed by cell-type name, while the test expects codes. This PR leaves them as they are.

@abhinadduri, could you take a look when you have a moment?

[gemini-code-assist]

Code Review

This pull request introduces a helper function read_h5_column to support reading 1-D string columns stored as nullable-string-array groups in h5ad files (written by newer versions of anndata and pandas), replacing direct slicing of H5 datasets. Feedback on the changes highlights potential crashes where read_h5_column might raise a ValueError (e.g., for unsupported group encodings or categorical columns) that is not caught by existing except KeyError blocks, preventing proper fallback behavior. Additionally, it is recommended to use read_h5_column in H5MetadataCache.__init__ to prevent type errors when loading non-categorical string columns.

[gemini-code-assist]

If var/gene_name is a group with an unsupported encoding (e.g., if it is a categorical group or some other unsupported group), read_h5_column will raise a ValueError. Since the except block only catches KeyError, this ValueError will propagate and crash the program instead of falling back to var/gene_name_index.

To ensure robust fallback, catch both KeyError and ValueError.

Suggested change

	try:
	raw = self.h5_file["var/gene_name"][:]
	raw = read_h5_column(self.h5_file["var/gene_name"])
	except KeyError:
	raw = self.h5_file["var/gene_name_index"][:]
	raw = read_h5_column(self.h5_file["var/gene_name_index"])
	try:
	raw = read_h5_column(self.h5_file["var/gene_name"])
	except (KeyError, ValueError):
	raw = read_h5_column(self.h5_file["var/gene_name_index"])

[albertlieyingadrian]

Kept except KeyError here on purpose. Widening it to ValueError would also catch encodings we can't read and silently switch to a different column (gene_name_index may hold different IDs). A categorical var/gene_name is already handled by the codes/categories branch just above, and an unknown encoding now fails loudly with the column name. Fixed in 1bab51f.

[gemini-code-assist]

Similarly, if var/_index is a group with an unsupported encoding, read_h5_column will raise a ValueError. This will propagate and crash the program instead of falling back to var/feature because the except block only catches KeyError.

Catching both KeyError and ValueError ensures a robust fallback.

Suggested change

	try:
	fallback = self.h5_file["var/_index"][:]
	fallback = read_h5_column(self.h5_file["var/_index"])
	except KeyError:
	fallback = self.h5_file["var/feature"][:]
	fallback = read_h5_column(self.h5_file["var/feature"])
	try:
	fallback = read_h5_column(self.h5_file["var/_index"])
	except (KeyError, ValueError):
	fallback = read_h5_column(self.h5_file["var/feature"])

[albertlieyingadrian]

Addressed in 1bab51f differently: read_h5_column now expands a categorical group to categories[codes], so a categorical var/_index is read directly instead of falling through to var/feature. Other unknown encodings still raise rather than silently switching columns. Covered by test_read_h5_column_categorical.

[gemini-code-assist]

If obs/_index is a categorical column, it is stored as a group in the H5 file. In this case, read_h5_column will raise a ValueError because its encoding is not nullable-string-array. Since the subsequent except KeyError: block (on line 1522) only catches KeyError, this ValueError will propagate and crash the program instead of falling back to the categorical loading logic.

To fix this, please update the except block on line 1522 to catch both KeyError and ValueError:

        except (KeyError, ValueError):
            # If obs/_index doesn't exist or is categorical, try obs/_index/categories and codes

[albertlieyingadrian]

Good catch, a categorical obs/_index did not reach the fallback (on main it raised TypeError from group[:]). Fixed in 1bab51f by decoding categorical groups inside read_h5_column, instead of widening the except. A wider except ValueError would also swallow UnicodeDecodeError from the decode loop and replace real barcodes. Test: test_categorical_obs_index_loads_barcodes fails before the change and passes after.

[gemini-code-assist]

In H5MetadataCache.__init__ (around line 55), if batch_col is a non-categorical string column, under pandas >= 3 and anndata >= 0.11 it will be written as a nullable-string-array group. The else block currently does raw = batch_ds[:], which will raise a TypeError because batch_ds is a group.

To prevent this crash, we should also use read_h5_column there:

            else:
                self.batch_is_categorical = False
                raw = read_h5_column(batch_ds)

[albertlieyingadrian]

Agreed, fixed in 1bab51f: the non-categorical batch branch now uses read_h5_column(batch_ds). test_metadata_cache_reads_string_batch_column writes a string batch column with convert_strings_to_categoricals=False (a nullable-string-array group). It raised TypeError before and now gives categories ['b1', 'b2'], codes [0, 1, 0, 1]. Full suite: 43 passed. The 3 test_mapping_strategy failures also fail on main.