Repository navigation
Remove load_and_normalize; load, subset, then normalize - #66
Merged
Merged
Conversation
The fused loader and its helpers are replaced by load_packed plus the .normalized() views. Drops the tests and docs that only covered it. Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM
This was referenced Oct 14, 2026
aarmey
added a commit
that referenced
this pull request
Oct 14, 2026
Merging main into this branch kept this branch's _rapid_load.py, which re-added load_and_normalize, _read_ivcsr_group and _validate_min_cells after #66 removed them -- without the numba kernels and imports they call, so ruff flagged 19 undefined names and calling it would raise NameError. Take main's version. load_packed already reads X through the anndata registry, which uses the parallel reader, so the only change this branch needs here is the hdf5plugin comment. Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM
aarmey
added a commit
that referenced
this pull request
Oct 14, 2026
* Replace Blosc2 with zstd-1 and decode chunks in parallel on read Blosc2+LZ4 compresses the IVCSR packed_indices stream by ~1.00x on real files (the delta+varint bytes have no repeats for an LZ codec to find), while zstd's entropy coding gets ~1.15-1.5x. zstd at level 1 is now the default for every numeric array written to .h5ad and zarr. HDF5 runs a dataset's filter pipeline one chunk at a time on one thread, so reads are decode-bound. read_array() looks up every chunk's file offset in one chunk_iter pass, preads the compressed chunks, and decodes them with numcodecs.Zstd on a thread pool sized by numba's thread count. VCS/IVCS arrays read through the anndata registry and the rapid loader both use it. Anything else -- Blosc2 files from earlier versions, uncompressed or contiguous datasets, unwritten chunks, files open for writing or on non-sec2 drivers -- falls back to an ordinary h5py read. read_h5ad now imports hdf5plugin itself: in a fresh process it previously failed on compressed obs/var columns, which are read before X. Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM * Restore main's _rapid_load.py after the merge with #66 Merging main into this branch kept this branch's _rapid_load.py, which re-added load_and_normalize, _read_ivcsr_group and _validate_min_cells after #66 removed them -- without the numba kernels and imports they call, so ruff flagged 19 undefined names and calling it would raise NameError. Take main's version. load_packed already reads X through the anndata registry, which uses the parallel reader, so the only change this branch needs here is the hdf5plugin comment. Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM * Write large arrays in 1 MiB chunks and decode one chunk run per thread h5py's automatic chunking gave the VCS arrays ~64-330 KiB chunks, and the parallel reader pays a fixed, GIL-holding Python cost per chunk, so on lupus (1.26M cells) reading packed_indices took 0.33 s instead of the ~0.08 s the zstd decode itself needs. numeric_only_compression now gives compressed 1-D HDF5 arrays larger than one chunk ~1 MiB chunks (unless the caller picked chunks), and read_array hands each thread one contiguous run of chunks and decodes full chunks straight into the output. On lupus: packed_indices 1.21 s (Blosc2) -> 0.095 s, value_ptr 0.10 s -> 0.031 s, load_packed 3.74 s -> 2.76 s, at the same zstd ratio. Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #40.
As discussed in #40,
load_and_normalizeis not migrated onto a streaming executor. It is removed in favor of the general workflow: load (load_packed), subset, then normalize via the.normalized()views, whichmainnow provides.Changes
load_and_normalizeand every helper only it used (cell totals, selective packed decode, filter/compact, fused normalize kernels, gene-mask code) from_rapid_load.py, which now contains onlyload_packed._filter_and_compactdtype tests intest_index_dtypes.py. Theload_packedtests stay.docs/usage.mdsections to describe load, then.normalized(), and removed the staleload_and_normalizereference in_vcs_norm.py.Dead code
uv run vulture src/vsparse/output is identical before and after this change (only the 60%-confidence public-method reports, which are API surface). No new items were left behind by the removal. Full test suite and ruff pass.https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM