feat(merxen_stub): stubbed MerXen pipeline for Flow validation (FLOW-700) - #2
Open
mhusbynflow wants to merge 5 commits into
Open
feat(merxen_stub): stubbed MerXen pipeline for Flow validation (FLOW-700)#2mhusbynflow wants to merge 5 commits into
mhusbynflow wants to merge 5 commits into
Conversation
Adds a structural stub of bourdenxlab/MerXen (FLOW-700) so we can iterate on Flow's handling of directory-typed inputs and outputs without running any real compute, GPUs, conda, or reference data. workflows/main.nf, nextflow.config, conf/dwight.config and samplesheet.example.csv are copied verbatim from MerXen. Every module keeps its process signatures, publishDir targets and input/output tuple shapes unchanged; only the script bodies are stubbed to create the declared outputs with placeholder content (.zarr as a directory with a dummy .zattrs, *_out as a directory with a placeholder file). flow/schema/merxen.json is the only Flow-specific addition: it exposes the raw MERSCOPE/Xenium directory inputs and MerXen's hardcoded reference paths as data inputs, and declares every curated output as a directory. Per FLOW-700 the pipeline code is not modified to accommodate Flow. test/ contains a local smoke test (fake inputs, dummy references, and an execution-only override config) that runs the full DAG in two configurations and asserts the expected output-directory tree. Two behaviours surfaced while running the verbatim pipeline and documented in the README: - main.nf only compiles on Nextflow >= 26 (the workflow body exceeds Groovy's 64 KB constant limit on older parsers). - With analysis_segmentation=both, main.nf joins a one-per-platform zarr channel against a two-per-key segmentation gate with .join() (1:1), so only the reseg branch runs downstream and original_seg is silently dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dings Adds flow/flow.config for a Flow docker+slurm environment: runs the pure-shell stub bodies in a stock ubuntu image instead of MerXen's GPU containers/conda envs, and drops the workstation GPU queues, flock GPU locks, and oversized resource requests from conf/dwight.config. It only sets keys the deployment leaves unset, since Flow applies repo config_paths at the lowest precedence. Documents the two Flow-backend behaviours that decide FLOW-700, confirmed from flow-api and by direct Nextflow experiments: - Flow's default Nextflow version (23.04.3) cannot compile main.nf; the pipeline version must allow a 26.x release. - Flow appends a generated publish.config last that forces every process's publishDir to a flat flow-results/<process>/, and a withName publishDir in a later config replaces a module's body publishDir. This flattens MerXen's nested <pair_id>/<platform>/<stage>/ output tree and collides parallel tasks of the same process. Registering the pipeline with "imports samples" yields an empty publish.config (native publishing), which is the path to preserving the nested structure and is the recommended next experiment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
merxen.json uses takes_samples=true, which forces a Flow sample per row. merxen-nosamples.json lets a dummy run pick directory data items directly and type a pair_id, with defaults (stop_stage=enrich, mecr_enabled=false) that need no reference data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stub previously passed merscope_dir/xenium_dir through as opaque value paths and never opened them, so on Flow it would report success even if the selected data path was not mounted into the process container or had the wrong internal structure — a false green for the exact thing FLOW-700 needs to test. BUILD_SPATIALDATA now reproduces the real input resolution from io/builders/pipeline.py: it reads input_path from the build config and fails the task if the path is not an accessible directory, or (for a raw export folder) if it lacks the files the builder reads — MERSCOPE: images/, micron_to_mosaic_pixel_transform.csv, cell_boundaries.parquet, detected_transcripts.parquet|csv; XENIUM: experiment.xenium/specs.json and transcripts.parquet. A reusable .zarr cache path only needs to be a directory. Test input directories are corrected to this real structure (they previously used the wrong filenames). Documented the remaining same-class gap: the value paths consumed by later stages (MECR/clustering/MapMyCells references, cortical-depth/distance GeoJSONs) are not yet access-checked inside the container. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
merxen_stub/, a structural stub of bourdenxlab/MerXen for FLOW-700. It reproduces MerXen's process DAG, directory-typed inputs, and directory-typed outputs while doing zero real computation — so we can iterate on Flow's handling of directory inputs/outputs without GPUs, conda, or reference data.Approach
workflows/main.nf,nextflow.config,conf/dwight.config,samplesheet.example.csv(byte-identical).script:body — replaced withmkdir/echothat create exactly the declared outputs (.zarr→ directory + dummy.zattrs;*_out/→ directory + placeholder). Alltag/publishDir/input/outputsignatures unchanged.flow/schema/merxen.jsonexposes raw MERSCOPE/Xenium directory inputs + MerXen's hardcoded reference paths asdatainputs, and declares all 18 curated outputs as directories ("filetype": ""). Per FLOW-700 the pipeline code is not modified to accommodate Flow.test/: local smoke test with fake inputs, dummy references, and an execution-only override config (disables conda/containers/GPU-lock).Verification
merxen_stub/test/run_smoke.shruns the full DAG in two configs — paired defaults (→ clustering, 27 processes) and all opt-in stages (→ mapmycells, 36 processes) — 21/21 output-directory assertions pass.Findings (documented in README)
main.nfcompiles a >64 KB constant that the legacy Groovy parser rejects (String too long) on 24.x/25.x.analysis_segmentation=both,main.nfuses.join()(1:1) between a one-per-pair_id|platformzarr channel and a two-per-key segmentation gate, sooriginal_segis silently dropped. Reproduced faithfully, not patched — worth confirming against production usage.Not done (deliberate)
outdirpaths (latest/latest_spatialdata.zarr, durablesegmentation/*.npy) are not reproduced — they're written outsidepublishDir, so Flow can't discover them anyway.🤖 Generated with Claude Code