Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 4 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,19 +134,13 @@ va.write_h5ad("archived.h5ad")
va_loaded = vsparse.VCSCAnnData.read_h5ad("archived.h5ad")
```

### Fast Filtering & Depth Normalization (`load_and_normalize`)
### Loading, subsetting, and normalizing

For IVCSR-backed datasets, `vsparse.load_and_normalize` bypasses full array decompression for rapid preprocessing, reproducing the filtering and depth normalization from `parafac2.normalize.prepare_dataset`:

- **Cell filtering without decoding**: Computes cell totals in $O(n_{\text{unique}})$ time directly from group sizes without unpacking varint indices.
- **Fused filter & normalization**: Performs gene filtering, cell/gene depth-scaling, and $\log_{10}(1000x + 1)$ transform in parallel passes over the compact CSR representation.
Loading, subsetting, and normalization are separate steps, so they can be mixed and matched. `vsparse.load_packed` decodes an IVCSR/IVCSC-stored `.h5ad` into a `VCSCAnnData`; normalization is then a view over the stored array (`.normalized("parafac2")` reproduces `parafac2.normalize.prepare_dataset`'s depth normalization and log10 transform) that is never materialized.

```python
adata_norm = vsparse.load_and_normalize(
"archived.h5ad",
min_cell_counts=10.0, # Filter cells with counts <= 10
gene_threshold=0.05, # Filter genes with counts <= 0.05 * n_cells
)
va = vsparse.load_packed("archived.h5ad")
norm = va.X.normalized("parafac2") # lazy view; norm @ B works without densifying
```

### CUDA (`to_gpu`)
Expand Down
1 change: 0 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@

.. autofunction:: vsparse.to_layer

.. autofunction:: vsparse.load_and_normalize

.. autoclass:: vsparse.CudaNormalizedView
:members:
Expand Down
21 changes: 7 additions & 14 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,27 +110,20 @@ views, concatenation, most of the scanpy/anndata ecosystem -- are **not** suppor
directly, so they round-trip through `write_elem`/`read_elem` even when nested inside
a plain `AnnData`'s `.uns`, independent of `VCSCAnnData`.

## Fast IVCSR loading and normalization
## Loading and normalizing IVCSR files

When working with large IVCSR-stored `.h5ad` files, `vsparse.load_and_normalize` provides
a fused loader and depth-normalization kernel reproducing the preprocessing in
`parafac2.normalize.prepare_dataset`:
`vsparse.load_packed` decodes an IVCSR/IVCSC-stored `.h5ad` into a `VCSCAnnData`.
Loading, subsetting, and normalization are separate steps; normalization is a
lazy view (`.normalized("parafac2")`) over the loaded array, so the steps can be
mixed and matched rather than fused into one loader:

```python
import vsparse

adata = vsparse.load_and_normalize(
"archived.ivcsr.h5ad",
min_cell_counts=10.0,
gene_threshold=0.05,
)
va = vsparse.load_packed("archived.ivcsr.h5ad")
norm = va.X.normalized("parafac2")
```

This bypasses generic `AnnData` indexing and decompression overhead:
1. Cell counts are computed in $O(n_{\text{unique}})$ time from unique-value group sizes without touching or decoding the packed `indices` byte stream.
2. The delta+varint indices are unpacked in parallel across CPU cores.
3. Row/gene filtering, compaction, and depth normalization ($(\text{cell\_scale}) \times (\text{gene\_sums})$ followed by $\log_{10}(1000x + 1)$) are executed in fused parallel passes directly into the output CSR representation.

## CUDA

A normalized view can be moved onto an NVIDIA GPU with `.to_gpu()`, which needs
Expand Down
3 changes: 1 addition & 2 deletions src/vsparse/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from vsparse._base import VCSCArray, VCSRArray
from vsparse._cuda import CudaNormalizedView, cuda_is_available
from vsparse._norm_common import RECIPES, Recipe
from vsparse._rapid_load import load_and_normalize, load_packed
from vsparse._rapid_load import load_packed
from vsparse._vcs_norm import VCSCArrayNormalized, VCSRArrayNormalized

__all__ = [
Expand All @@ -25,7 +25,6 @@
"VCSRArrayNormalized",
"cuda_is_available",
"from_anndata",
"load_and_normalize",
"load_packed",
"to_layer",
]
Expand Down
Loading
Loading