dev to main v0.8.0 - #52
Merged
Merged
Conversation
Main to dev
Lets users compare different edge computations (e.g. raw vs. normalized
scoring) on the same tissue without duplicating cell/transcript/boundary
files. Additional edge sets live in a dedicated edges/ subfolder of the
dataset; a single global dropdown (top of the Edge Data panel) switches
between them and applies to all viewer panels.
Backend:
- GET /edges/{dataset}/files now returns {files:[{id,label}], default},
scoped to the legacy top-level edges.parquet + the edges/ subfolder so
it never lists cells/transcripts/boundary parquet.
- _reader() resolves edge_file under the dataset dir with a path-traversal
guard (escape -> 400, missing -> 404). Every endpoint already accepted
edge_file; the reader cache is keyed by (dataset, edge_file).
Frontend:
- New global store.edgeFile; setEdgeFile/setDataset reset edge-scoped
state (lrmCatalogue, hiddenLrms, selectedEdge, edge color range/clamp).
- edge_file threaded through all six edge fetch sites (useEdges x2,
useEdgeColors, EdgeSection schema+catalogue, EdgeCategoricalLegend,
EdgeInfoPanel). Picker shown only when >1 edge file exists.
Tooling/docs:
- make_edges.py gains --out for writing into edges/.
- Two demo edge sets added under sample_data/mouse_ileum_tiny/edges/.
- CLAUDE.md + docs/data_format.md document the convention.
- Version bump to v0.3.1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat: edge-file dropdown to flip between multiple edges.parquet (#46)
Xenium writes its multi-channel stack to morphology_focus/, but list_images only scanned the dataset root, so those channels were unreachable from the image picker even though the files were sitting right there. Both sample datasets ship four of them. list_images now scans the root and one level of subdirectories, returning bare filename stems. _find_source resolves a stem back to a path by searching the same two locations in the same order, so the picker can never list an image the tile builder cannot open. Root-level files are added first and stems are de-duplicated, so a root file wins any name collision with a subdirectory file in both functions. Hidden directories are skipped so .dzi_cache is never scanned. Salvaged from a GitHub Desktop stash (28616a9) whose edge-file changes were superseded by the merged #46 implementation; this part had never landed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d plans; v0.4.0 CLAUDE.md had drifted far enough to mislead. Corrections: - Visium HD was entirely absent despite being a registered reader (second in detection order). Added it, plus a note that VisiumHDReader.transcripts and .cell_boundaries still carry the pre-refactor limit= signature and would raise TypeError if the capability flags did not stop the frontend calling them. - Per-panel rotation (#31) and RenderingStatus.jsx were undocumented. Added a Rotation section covering the two places the angle must be applied, why the fetch bbox is padded, and why viewportActual exists separately. - The cloud deployment story was missing entirely: docs/cloud-deploy.md, Caddyfile, deploy.sh, docker-compose.prod.yml, upload-data.sh. Corrected the claim that there is no auth at all — Caddy basicauth exists but is opt-in and off by default. - Fixed the r/ listing, which named a file deleted in b30bc82, and the App.jsx path. Documented the actual three-script pipeline. - Replaced the stale fracW zoom-skip thresholds with the sampling fractions that actually govern fetch volume now. - Reordered What's Not Built Yet to lead with the real bottleneck: spatial reads pull whole parquet files into pandas on every viewport change instead of pushing the bbox down to DuckDB the way the edge path does. Added the CellInfoPanel supplemental-metadata gap and open issues #35 and #45. README: replaced your-lab placeholder URLs with RaredonLab, added Visium HD to the platform table with an honest note on per-platform completeness, documented multi-channel morphology, multiple edge sets, split-screen and rotation, linked the cloud-deploy runbook, corrected export_for_TissuePlex to export_to_TissuePlex, and added a roadmap pointing at the open issues. vite.config.js defaulted the dev server to port 3000 — the same port docker compose binds — which is why launch.json had to override it. Set it to 5173 so the config matches the documented intent and npm run dev stops colliding with a running container. PLAN.md, PLAN_v2_2026-05-02.md and EDGE_UI_PLAN.md move to OBS/ with a README explaining why each is obsolete. EDGE_UI_PLAN in particular specifies the opposite of what shipped on both edge aggregation and lrm_set coloring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both methods loaded their entire parquet with pq.read_table(...).to_pandas()
and then masked in pandas, so every viewport change materialized the whole
file to keep a viewport-sized slice of it.
Measured on a synthetic 40M-row / 0.78 GB transcripts.parquet, one zoomed-in
viewport query:
pandas full read + mask 2903 MB peak RSS 912 ms
DuckDB streaming 233 MB peak RSS 1369 ms
12x less memory. That is the binding constraint: production runs on a 16 GB
droplet, where a multi-GB transcripts.parquet under the old path exhausts RAM
long before it is merely slow.
DuckDB is somewhat slower here, and the reason is worth recording. Xenium
writes transcripts.parquet in row order, not spatial order, with very large
row groups -- the bundled breast dataset is 1.1M rows in 2 row groups, the
first spanning the entire x-range. Row-group statistics are therefore useless
and DuckDB scans everything anyway, paying predicate-evaluation cost with none
of the pruning payoff. Sorting the file spatially and rewriting it with small
row groups takes the same query from 1010 ms to 29 ms; that is a follow-up,
noted in CLAUDE.md.
Also fixes polygon clipping. Boundaries previously filtered individual
vertices by bbox, so cells straddling the viewport edge came back missing
part of their outline and rendered as torn shapes -- 97 such cells on the
bundled breast dataset. Selection is now per cell: a cell qualifies if any
vertex is in the bbox, and all of its vertices are returned. Sampling likewise
draws whole cells.
Shared query helpers live in readers/duck.py so the other platform readers can
adopt the same path. Verified against pandas ground truth on both bundled
datasets: identical totals for full-extent, bbox-quadrant and gene-filtered
queries, identical pixel-space conversion, and sampling that is deterministic
across repeated calls.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design proposal only -- no reader implemented yet. Records the format research and the decisions taken: target Spatial Genomics GenePS rather than academic seqFISH+ (which has no standard output), one ROI per folder, bare integer cell labels as cell_id, and an edge-metadata/ folder mirroring the existing cell-metadata/ convention. The format claims are verified against the real scverse CI fixture rather than taken from documentation. Two findings changed the plan: A single seqFISH dataset mixes coordinate systems. Measured on the 1000x1000 DAPI at 0.107161 um/px: CellCoordinates center_x spans 1.82-105.66 and TranscriptList x spans 0-107.05, both microns, while Boundaries.geojson vertices span 0-999, pixels. Cells and transcripts need dividing by pixel_size and boundaries must pass through untouched. One global transform in either direction would put cells and their own outlines in different places, which presents as a rendering bug rather than a unit bug. The px-vs-um heuristic (ratio of max coordinate to image width) separates all three cases here with two orders of magnitude of headroom. GeoJSON features carry the cell label in their `id` field -- strings "1".."62" matching CellCoordinates.label exactly. spatialdata-io does not read this and maps polygons to cells positionally, with an open issue about the fragility (scverse/spatialdata-io#249). Joining on id is correct by construction; a silent off-by-one there would draw every outline on the wrong cell. sample_data/.gitignore excludes seqfish*/ (and visium-*/). The test dataset is public for CI by written permission from Spatial Genomics, not under an open licence, so it must not be redistributed from this repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Groundwork for seqFISH support. Two changes, neither altering Xenium behaviour. backend/tests/golden_snapshot.py exercises every reader method against the bundled datasets, digests the results, and diffs them against a recorded baseline -- 62 probes across the two Xenium datasets, covering info, capabilities, gene lists, cells, schema, transcripts (full/bbox/gene-filtered/ sampled), boundaries (full/sampled), per-cell detail and expression, colour values, and every edge endpoint for each edge file. It calls readers directly rather than over HTTP, so no server is needed and failures point at the reader instead of the transport. This repo has no test suite, so cross-platform reader work had nothing to catch a regression. Verified the guard actually bites: flipping duck.SAMPLE_SEED from 42 to 43 fails exactly the six sampled probes and passes again on revert. Two sources of false failure had to be removed first. Record-list digests are now order-independent, because query_grouped uses ORDER BY RANDOM() and SQL GROUP BY promises no ordering -- hashing in returned order failed on every run. And the edge_detail probe picked grouped[0], a different edge each time; it now takes the lexicographic minimum. _load_supplemental_metadata and _read_csv_with_barcodes move from XeniumReader to SpatialDatasetReader, so cell-metadata/ becomes available to every platform rather than Xenium alone. Nothing platform-specific was in them; all Xenium contributes now is the list of its own root CSVs to ignore. That list is the trap this opens, so it is closed here too: MERSCOPE and CosMx both write CSVs at the dataset root that the shared loader would otherwise ingest as user metadata. MERSCOPE gets an exact-name skip set; CosMx prefixes every output with the experiment name, so it needs suffix matching, which _is_platform_csv now supports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a fifth platform reader. Xenium is untouched: all 62 of its golden-snapshot
probes are byte-identical across the change.
"seqFISH" names two unrelated things. The academic Cai-lab method has no
standard output layout; this targets the commercial Spatial Genomics GenePS
platform, which does. Current v2 layout is fully supported -- cells,
transcripts, boundaries, expression, both colour-value modes. Legacy v1 reads
cells and transcripts but declares has_boundaries: False, since v1 ships only a
label mask and polygonising it would mean either a heavy new dependency or
hand-rolled contour tracing; deferred deliberately.
The hard part is coordinates. A single seqFISH dataset mixes units, measured on
the reference dataset (1000x1000 DAPI at 0.107161 um/px):
CellCoordinates center_x 1.82 -> 105.66 microns
TranscriptList x 0.00 -> 107.05 microns
Boundaries vertices 0 -> 999 pixels
Cells and transcripts are divided by pixel_size; boundaries pass through
untouched. One global transform in either direction would put cells and their
own outlines in different places, presenting as a rendering bug rather than a
unit bug. The convention also differs across GenePS software versions, so it
cannot be hard-coded: _units_divisor decides per table by comparing that
table's extent to the image width, and logs its verdict. On real data the
ratios are 0.106 / 0.107 / 0.999 -- two orders of magnitude apart.
Verified geometrically rather than by digest alone: every cell centroid falls
inside its own polygon, 62/62 on the reference dataset and 36/36 on the
synthetic fixture, with zero false positives against a control. Independently,
polygon area recomputed from boundary vertices agrees with the reported
cell_area to within 3%.
Cell identity comes from each GeoJSON feature's id, which equals label.
spatialdata-io maps polygons positionally instead and has an open issue about
the fragility (scverse/spatialdata-io#249); a silent off-by-one there would
draw every outline on the wrong cell.
sample_data/make_seqfish.py generates a committable synthetic v2 ROI (400K) and
deliberately reproduces the mixed units, so a reader that got them wrong would
fail on it. The real reference dataset is public by written permission from
Spatial Genomics rather than under an open licence, so it stays gitignored.
Three bugs fixed along the way, all pre-existing and none seqFISH-specific:
- CellInfoPanel rendered the literal string "undefined um2" for any platform
not reporting a cell area -- the optional chain was not paired with a null
check the way nucleus_area's is.
- Switching datasets kept the previous dataset's activeImage until the image
list resolved, so OSD spent that window requesting the old image from the new
dataset and logging 404s. Only visible now because image names differ across
platforms ("morphology" vs "Roi1_DAPI"). setDataset clears it and OSD init
waits.
- cell_area is kept in um2 to match Xenium, which never converts it, so the
"um2" label in the info panel is true on every platform.
Verified end-to-end in the browser across Xenium -> seqFISH -> Xenium with zero
console errors and no backend errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The edge-side mirror of cell-metadata/: annotate cell pairs -- a call, a
confidence, a review flag -- without regenerating edges.parquet from R.
dataset_dir/
edge-metadata/
annotations.csv key column `edge` = "SendingCell|ReceivingCell"
Rather than write a second loader, the cell-metadata implementation moves to
readers/supplemental.py parameterised on the key column, and both features now
share it. Same file types, same outer-join, same forgiving key resolution
(explicit key column -> "Unnamed: 0", pandas' name for R's unnamed rowname
column -> first column if unique strings), so the R default just works. Moving
it is behaviour-preserving: all Xenium probes stayed identical across the
refactor.
The folder sits beside the dataset, not beside the edge file. _dataset_dir
walks up out of edges/ when the edge file is nested there, so one set of
annotations applies across every edge source in the dataset -- annotations
describe cell pairs, which are a property of the tissue rather than of one
scoring run. Verified against both edges.parquet and edges/*.parquet.
Three integration points in edge_reader.py, and no frontend change was needed
for the main one: schema() merges supplemental columns into the returned map,
and LayerPanel builds the edge colour-by dropdown straight from the schema, so
they appear on their own. edge_color_values() checks the parquet first then the
supplemental frame, skipping the GROUP BY since supplemental data is already
one row per edge. edge_detail() attaches matches under a `metadata` key.
Parquet wins a name collision -- a supplemental column silently shadowing a
real one would be painful to debug.
EdgeInfoPanel renders those generically as an "Annotations" block above the LRM
table, so any column the user adds shows up untouched.
mouse_ileum_tiny gains worked examples of both cell-metadata/ and
edge-metadata/. Neither feature had a sample dataset before, so cell-metadata
had been unexercisable locally since it shipped.
Verified in the browser: the three annotation columns appear in the edge
colour-by dropdown, a categorical one renders its legend and colours the edges,
and clicking an edge shows the annotations alongside its LRM scores --
interaction_class "heterotypic" on a Fibroblast -> Endothelial edge, confirming
the join lands on the right row. No console or backend errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Version bump covering seqFISH platform support, the reader regression guard, and supplemental edge metadata. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
seqFISH functionality; v0.6.0; beta
Viewport queries on a 40M-row transcripts.parquet go from 1180 ms to 44 ms.
Moving these queries to DuckDB earlier fixed their memory use but not their
speed, and this is why: Xenium writes transcripts.parquet in acquisition order
with very large row groups -- the bundled breast dataset is 1.1M rows in two
row groups, the first spanning the entire x-range -- so row-group statistics
exclude nothing and every bbox query scans the whole file.
spatial_cache.sorted_path() rewrites the file sorted by a coarse spatial grid
with 100K-row row groups, built on first access and cached on disk, the same
pattern ensure_pyramid already uses for tiles. Measured, with identical results
in every case:
Xenium transcripts 600 MB / 40M rows 1180 ms -> 44 ms 27x
seqFISH transcripts 229 MB CSV / 8M rows 1595 ms -> 156 ms 10x
Xenium boundaries 40 MB / 3.6M verts 102 ms -> 73 ms 1.4x
Boundaries gain least by design: half that query is a cell_id semi-join to pull
whole polygons, which spatial sorting cannot help. For seqFISH the same pass
also converts CSV to parquet, which is why it helps a format that cannot be
range-scanned at all.
The index alone was not enough, and the reason is worth recording. DuckDB prunes
row groups at plan time by comparing the filter to per-group statistics; with
bound `?` parameters those values are unknown then, so it cannot prune. The
first cached build measured only 1.4x faster until bbox_predicate switched to
inlining the bounds as SQL literals:
sorted file, COUNT literals 6.8 ms vs ? params 155 ms
sorted file, SELECT literals 35 ms vs ? params 321 ms
unsorted file, COUNT literals 186 ms vs ? params 178 ms
On an unsorted file the two are identical, which is why this never mattered
before. Inlining is safe because these are numbers, never user text: every value
goes through float() and non-finite values are rejected, while string filters
such as gene names still bind through in_predicate.
Deliberately conservative in three ways. Files under SPATIAL_CACHE_MIN_BYTES
(64 MB) are left alone, since the build cost is not repaid and it keeps the
bundled datasets uncached so the golden baseline does not depend on whether a
cache happens to exist. A failed build returns None and the query falls back to
the source file, so indexing can never make a dataset unreadable. SPATIAL_CACHE=0
disables it outright.
Cache validity covers the sort columns, not just the source stamp -- testing
turned up that a cache built for one column pair was being served for a query on
another, sorted by the wrong axis and silently so. Not reachable through current
callers, but a trap worth closing.
Two golden probes moved: same population and sample size, different arbitrary
subset, because seeded reservoir sampling draws differently once row order
changes. Totals verified against pandas ground truth, and no returned row falls
outside the requested bbox.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reader was a stub built on guessed paths. Rewritten against a genuine Space Ranger 4.0.1 outs/ tree, now bundled as sample_data/visium_hd_tiny. Three things were broken beyond the known signature bug, and the first two meant no real dataset could ever have worked: - Detection globbed `square_???um` at the dataset root, but Space Ranger nests the bin directories under `binned_outputs/`. Genuine output was never detected at all. - `tissue_positions.parquet` and `scalefactors_json.json` are per bin, under `binned_outputs/square_NNNum/spatial/`. The reader looked for them in the top-level `spatial/`, which holds images only. - Morphology is a PNG. The tile pipeline accepted TIFF only, so even a detected dataset would have shown no image. `_SOURCE_EXTS` and `spatial._TIFF_EXTS` now include .png, and _build_dzi_pillow handles it where libvips is missing, since the tifffile fallback cannot open one. Bins are served as square polygons rather than points. A bin is literally a square of side spot_diameter_fullres, so cell_boundaries() emits four vertices per bin. This is not cosmetic: nothing renders cells() centroids -- the boundary layers are the only path to drawing a unit -- so the previous has_boundaries: False would have left the canvas empty. As squares, fill, outline, colour-by, picking and region selection all work through the existing layers with no frontend change. Also implemented from filtered_feature_bc_matrix.h5, previously all stubs: gene_list, cell_expression and gene-set colour values. pixel_size now comes from microns_per_pixel instead of being hardcoded to 1.0, and transcripts() returns the dict shape every other reader uses instead of a bare list. Verified end to end: 20,830 in-tissue bins at 8 um render as a regular grid registered on the H&E, the UI relabels itself to "Bin Segments" / "Color bins" / "Click a bin to inspect" off unit_label, transcripts hides itself, and clicking a bin shows real per-bin expression. No console or backend errors. The guard grows to 115 probes across 5 datasets, with the four existing datasets byte-identical. Fixture is a 26 MB subset of 10x's 297 MB Tiny Mouse Brain dataset, CC BY 4.0 and so redistributable; PROVENANCE.md records the source, what was dropped and why. It also records the trap this fixture cannot catch -- tissue_hires_scalef is 1.0 and microns_per_pixel is 1.003, both effectively identity, so a missing scale multiply looks correct here and breaks on every real dataset. Separately, docs called the NICHESv2 exporter export_for_TissuePlex; the real name is export_to_TissuePlex, verified against the package. It exists only on the dev branch -- main 404s -- and needs arrow, which is only in Suggests. Both noted in docs/data_format.md, along with the coordinate-unit constraint that matters for driving NICHESv2 from a pixel-space platform like Visium HD. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified docs/data_format.md against R/export_to_TissuePlex.R on the dev branch rather than against memory. Three corrections. The function name was already fixed in 794ffad; confirmed all four occurrences are right and the two that were already correct are untouched. sending_type / receiving_type were described as "optional but recommended". The writer's final.cols always emits all 16 columns; when celltype.col is NULL or the named column is missing from $edge.meta it sets NA_character_ and warns. So the columns are always present and only their values may be NA. They are optional only for a hand-written parquet. The placeholder-row behaviour was undocumented. Every edge in $edge.list is exported, so an edge with no scored signal gets one row with lrm, lrm_id, ligand, receptor, score and score_norm all null. That is deliberate -- it lets the tissue-graph layer show which pairs are neighbours independently of which pairs have signal -- and it is the origin of the "completely null LRM rows" gotcha the backend already guards against. Also noted that validation must use na.rm = TRUE, since checking min(score) or the per-edge score_norm sum without it reports failure whenever placeholders exist. The coordinate table was actively misleading. It said coordinates should be "native (typically µm)" while listing pixel-valued source columns for CosMx and Visium HD. The backend always divides x1..y2 by pixel_size, and no platform has pixel_size 1.0 (Xenium 0.2125, seqFISH 0.107, MERSCOPE 0.108, CosMx 0.18, Visium HD microns_per_pixel), so pixel-valued coordinates land wrong by exactly that factor. The table now states the native unit per platform and the conversion needed, with the seqFISH auto-detection caveat. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One script per data type in r/, plus r/README.md. Each builds a gene x cell
count matrix and a metadata frame, runs NICHESv2 in spatial mode, and exports
edges.parquet. What differs between platforms is the first two steps, which is
why they are separate scripts rather than one with branches:
niches_xenium.R coordinates already um; the simple case to read first
niches_seqfish.R dense cells x genes CSV needing transposition, and
coordinate units that vary by GenePS version
niches_visium_hd.R coordinates in PIXELS, multiplied by microns_per_pixel
before NICHESv2 sees them
niches_common.R shared only: 10x h5 reader, barcode alignment,
LR-coverage check, output validation
The coordinate conversion in the Visium HD script is the point of that script.
export_to_TissuePlex() copies meta.data$x/$y verbatim into x1..y2 and the
backend divides by pixel_size assuming microns, so raw pixels put every edge
offset from its bin by exactly that factor, silently. Verified by joining the
exported edges back to tissue_positions: max |dx| = max |dy| = 0 px across all
3,321 bins.
Run against every demo dataset:
xenium_human_breast_2fov 7,275 cells -> 181,523 edges, 189,063 scored rows,
32 LRMs. Top hits CDH1|CDH1, CXCL12|CXCR4, PTN|SDC4
-- plausible for breast. Written to
edges/niches_rad30.parquet so the existing synthetic
edges.parquet stays the default and both are
selectable from the dropdown.
visium_hd_tiny 3,321 bins -> 66,001 edges, 69 LRMs. Sparse, as the
data is (32k UMIs total), but real.
seqfish_synthetic 36 cells -> 250 edges, 30 LRMs, no placeholders.
mouse_ileum_tiny cannot run, and this is the data not the script: its
matrix is a near-empty placeholder, 467 counts with
7 expressed genes, so no LR pair can score. It stays
on synthetic edges from make_edges.py.
Two changes fell out of running this for real.
make_seqfish.py now generates a panel of 36 real mouse gene symbols forming 20
complete LR pairs in connectomedb2025, instead of Gene00/Gene01/... Invented
symbols match nothing, so the committed seqFISH fixture could not demonstrate
the edge pipeline at all -- NICHESv2 aborted with "No valid LR pairs remain".
check_lr_coverage() counts scorable pairs before the run and stops with an
explanation naming the likely causes. Previously a small targeted panel failed
with that same message thrown from deep inside compute_CellToCell(), which
reads as a bug rather than as a property of the panel. The real 12-gene
seqfish_instrument2 panel is exactly this case.
Validation uses na.rm = TRUE throughout, because unscored edges are exported as
placeholder rows with NA scores; checking min(score) or the per-edge score_norm
sum without it reports failure whenever placeholders exist.
Guard grows to 133 probes across 5 datasets. mouse_ileum_tiny and
seqfish_instrument2 are byte-identical; everything that moved is the
regenerated seqFISH panel or a newly added edge file.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Version bump covering Visium HD support against real Space Ranger output, the per-platform NICHESv2 scripts, and the edges.parquet spec corrections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v0.7.0 — Visium HD + seqFISH support, NICHESv2 pipeline, spatial index
Both readers were written from documentation and had never run against a real dataset. Doing so found five defects, four of which rendered the platform unusable rather than merely degraded. Verified against Vizgen's own VPT test data (Apache-2.0) and the OSTA/OSF CosMx mouse brain set; neither is committed, and sample_data/.gitignore already excluded both patterns. MERSCOPE transcripts() read usecols=["x","y"] from detected_transcripts.csv. Those are FOV-LOCAL PIXEL coordinates; the whole-slide micron pair is global_x/global_y. Measured on the real file, transcripts came back spanning 843-18111 px while the cells they belong to spanned 16-3872 -- a 4.7x mismatch that put the transcript layer nowhere near the tissue. It was invisible because the method was wrapped in a bare `except Exception` returning an empty result. Now reads global_x/global_y, and says so when they are absent instead of silently returning nothing. pixel_size looked for microns_per_pixel in a root manifest.json. The real source is images/micron_to_mosaic_pixel_transform.csv, whose scale term is pixels per micron, so pixel_size = 1/M[0][0]. Reading it gives 0.107962 on the reference data; the old 0.108 fallback was accidentally correct, which is why nothing looked wrong. Boundaries now work. The "HDF5 only" blocker applied to instrument software v231 and earlier; v232+ writes geoparquet, and vizgen-postprocessing writes cell_micron_space.parquet. Both are WKB polygons already in microns. Decoded with ~40 lines of struct unpacking against the OGC spec rather than adding shapely, a C-extension dependency, to a requirements file pinned tightly enough to need cffi<2.0 for pyvips. Validated geometrically: 907 of 918 reported cell centres fall inside their own decoded polygon, median offset 1.4 um. The 11 misses are cells whose volumetric centroid across 7 z-planes sits outside the single z-plane drawn -- expected, not a decode error. CosMx pixel_size was 0.18; Bruker documents 0.12028 (120 nm edge) and Giotto hardcodes the same. 50% too large, which scaled every distance and, since the edge reader divides x1/y1 by pixel_size, would have displaced the whole edge layer. Detection never fired on real data. Some public exports drop the experiment prefix, and `*_tx_file.csv` cannot match a bare `tx_file.csv` -- the glob needs a literal underscore. Several sentinels are now tried, since exports vary in which files ship. _find_file retries without the prefix for the same reason. The same prefix assumption made the reader hang. _ROOT_CSV_SKIP_SUFFIXES only listed prefixed names, so the shared supplemental-metadata loader treated the platform's own exprMat (73 MB), polygons (38 MB) and tx (24 MB) as user metadata and tried to outer-join them. cells() took over 90 s; it now takes 0.2 s. Cell metadata never loaded even so: the rename looked for "x_centroid" but CosMx writes CenterX_global_px. Cells are also keyed on the composite (fov, cell_ID) -- cell_ID restarts at 1 in every field of view, so keying on it alone merges unrelated cells. Newer exports' study-wide `cell` column is preferred when present, with a local+FOV-origin fallback for exports carrying only the per-FOV frame. Boundaries now work from <expt>-polygons.csv, which is a standard AtoMx export and gives vertices already in global pixels. Note the column is `cellID` there and `cell_ID` everywhere else. Validated: 4000/4000 sampled cell centres fall inside their own polygon. has_morphology is now conditional on CellComposite/ rather than always True -- the OSF export has no images. Guard grows to 171 probes across 7 datasets. All five pre-existing datasets are byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Browser verification of CosMx and MERSCOPE. MERSCOPE was correct as committed; CosMx showed a blank canvas, and the cause was architectural rather than CosMx-specific. The viewer derives its entire coordinate space from the tile pyramid: OSD's `open` event is the only thing that ever calls setImageSize, and deck.gl mounts only once imageSize is known. A dataset with no morphology therefore rendered nothing at all -- no cells, no boundaries, no edges -- even though every endpoint served correct data. Any image-less dataset hit this, not just CosMx; it was latent because every dataset until now had an image. Rather than give deck.gl its own navigation path when OSD is absent, which would fork the model that per-panel rotation and match-zoom both depend on, the backend now synthesises a placeholder canvas. list_images offers a reserved "__blank__" entry when a dataset has no image of its own, and the tiles router sizes it from the reader's new data_extent(). No tiles are generated: a blank pyramid is uniform, so one 256x256 tile is written and served for every level/column/row. At CosMx scale that is 8 KB instead of tens of thousands of identical files. CosMx coordinates are now normalised to the dataset's own origin. Its global pixel coordinates are in the slide frame, so the mouse-brain set spans x 128749..175643 and y -9980..15588. Anchoring the canvas at the slide origin put 43% of cells at negative coordinates and off the canvas entirely, with the data occupying 27% of the width. TissuePlex's contract is image pixel space and CosMx ships no image to define it, so with no external frame the dataset's own corner is the only sensible origin. Cells, transcripts and boundaries subtract the same offset and stay registered. Two bugs fell out of testing that. cell_boundaries filtered the bbox against raw slide-frame coordinates while the incoming bbox was already in the shifted frame, so full-extent queries worked and every viewport query returned zero. It also applied _bbox_to_native, which multiplies by pixel_size for micron platforms -- wrong for CosMx, whose coordinates are already pixels. activeImage was only reconciled inside the image-list fetch, whose closure captured it and which only re-runs on dataset change. Whether it got set depended on the order two state updates landed in, and for a dataset whose only image is the placeholder it stayed null, so OSD never opened. Reconciliation is now its own declarative effect keyed on the image list. Also: the Morphology layer row is gated on has_morphology, since the opacity control over a flat placeholder fill is a dead toggle; and the placeholder name is hidden from the viewer header rather than shown as if it were a real image. Verified in the browser. MERSCOPE: 731/731 segments with real WKB-decoded outlines, transcripts sitting inside their cells (the coordinate fix), DAPI pyramid rendering, cell click returning real expression. CosMx: 5k/39k segments drawn on the placeholder canvas, 960 genes, no console or backend errors. Guard holds at 171 probes; only the eight CosMx probes affected by the origin shift moved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CosMx and MERSCOPE fixed against real public data, plus placeholder-canvas rendering for datasets that ship no morphology image. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v0.7.1 MERFISH and CosMX capabilities
Closes #35 Closes #45 Both issues are the same question asked twice — "what kind of thing is this metadata column?" — so they share one module, readers/metadata_filter.py. #35 — treat as categorical Integer cluster IDs from Seurat were routed to a viridis gradient by dtype. color_values() and edge_color_values() now take an explicit `categorical` override, and category labels sort numerically so cluster 10 follows 2. _color_values_meta moves to the base class. All six readers carried a near-identical copy and the copies had drifted: CosMx filled NaN with ""/0 where the others dropped it, and only some sorted with key=str. A reader now supplies only _metadata_frame(). The frontend stops guessing the type from the schema dtype — it could not, since the rule also depends on cardinality. The old guess disagreed for exactly the columns this issue is about: the canvas drew discrete colours while the panel showed a gradient with two sliders that did nothing. Panel 0 now reports the type the backend returned. That also removed the duplicate color-values fetch both legends were making for themselves. #45 — select cells and edges by metadata A MetadataFilter is a categorical allowlist or an inclusive numeric range, resolved server-side and applied BEFORE sampling. That ordering is the point: both queries sample on the server, so a client-side filter would leave a fraction of a subset — a cluster holding 5% of cells at a 10% sample would draw 0.5% of the tissue. cell_boundaries() takes cell_ids in all five readers. query_grouped() takes cell_ids and edge_filter; an edge survives only when both endpoints do, since a half-outside edge runs off to a cell that is not drawn. Large id sets go through a registered DuckDB relation rather than an IN list. An unknown column raises 400 rather than rendering everything under an apparently-active filter. Also fixed, because the filter exposed it: useCellBoundaries picks its auto fraction from the previous fetch's total, which a filter invalidates, and nothing else triggered a refetch — so the layer sat showing a tenth of an already-small subset until the user panned. It now recalibrates once. Guard extended to 191 probes across 7 datasets, covering both features on every platform. The demo fixture gains a 12-level seurat_clusters column so the repo's own data reproduces #35. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.