Add automated CMS data stitching workflow - #24
Open
swu4bnl wants to merge 31 commits into
Open
Conversation
- Add auto_stitch.py with _categorize_anchor_failure(), run_auto_stitch_anchor(), and verify_stitch_outputs() tasks, following the linker.py/data_validation.py module pattern - Simplify end_of_run_workflow() signature: replace 8 stitch-specific params with a single stitch_config dict; remove plot/show_colorbar interactive flags from workflow API; remove unconditional dev_probe_test() call - Subprocess/CLI invocation is now isolated inside auto_stitch.py as an explicit adapter boundary - Update prefect.yaml deployment parameters to match simplified interface - Add test/test_auto_stitch.py with 13 tests for _categorize_anchor_failure()
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Rename stitch code into a clearer package structure: stitch_tasks.py plus stitch/runner.py, stitch/core/, and stitch/configs/ - Replace CLI-style stitch invocation with direct Python imports and function calls - Simplify the Prefect entrypoint so it prepares run metadata and calls run_stitch_validation() - Remove redundant path-loader layers, compatibility wrappers, debug helpers, and unused functions - Route stitch outputs through the same proposal/experiment directory logic used by linker.py - Update tests and docs for the new module names and output path behavior
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Add anchor auto-stitch Prefect integration with output verification
…ion for Tiled access failures
…s backup when it fails
swu4bnl
requested review from
Jun Aishima (JunAishima) and
Dan Allan (danielballan)
and
a lite review from Copilot
August 4, 2026 20:47
There was a problem hiding this comment.
Pull request overview
Adds a new “anchor-mode” auto-stitching capability to the CMS end-of-run Prefect flow by introducing a minimal stitch/ runtime package, workflow settings parsing, and Prefect tasks that locate tile groups in Tiled, stitch them, write outputs to proposal folders, and optionally verify outputs.
Changes:
- Added a standalone
stitch/package (core stitcher + Tiled adapter + default config/masks) and a runner entrypoint used by the workflow. - Integrated auto-stitch execution and optional output verification into
end_of_run_workflow, with settings loaded from env/workflow_options. - Updated deployment/configuration and tests (settings parsing, runner selection logic, path traversal prevention, mode/mask behavior).
Reviewed changes
Copilot reviewed 27 out of 50 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| workflow_settings.py | Adds parsing/validation for stitch enablement, verification, and config overrides from env/flow parameters. |
| test/test_workflow_settings.py | Unit tests for stitch settings defaults, env parsing, and workflow_options overrides. |
| test/test_stitch_tasks.py | Unit tests for Prefect-facing stitch tasks (runner invocation, categorization, verification). |
| test/test_stitch_runner.py | Unit tests for anchor-run selection logic in the stitch runner. |
| test/test_stitch_modes.py | Tests for mode registry caching behavior and default mask path resolution. |
| test/test_linker_paths.py | Tests for new linker path traversal checks and proposal-layout directory construction. |
| stitch/runner.py | Implements Tiled fetch + anchor selection + stitching + output writing + index generation. |
| stitch/README.md | Documents purpose and included components of the embedded stitch runtime. |
| stitch/core/serialize.py | Adds JSON + image/NPZ serialization helpers for stitch results. |
| stitch/core/offsets.py | Adds readback-to-pixel offset calculation utilities. |
| stitch/core/modes.py | Adds mode registry and required-label validation based on config. |
| stitch/core/models.py | Defines typed dataclasses for tiles and stitch results. |
| stitch/core/mask.py | Adds mask loading/validation/application helpers. |
| stitch/core/grouping.py | Adds normalization/grouping logic and repeated-acquisition disambiguation. |
| stitch/core/errors.py | Adds domain-specific exceptions for stitching. |
| stitch/core/core.py | Implements the array-based stitcher core (offsets, accumulation, QA flags, optional refinements). |
| stitch/core/config.py | Loads stitch config and resolves relative mask paths to absolute package paths. |
| stitch/core/adapters/tiled_adapter.py | Provides adapter functions to normalize Tiled runs into stitch tile entries. |
| stitch/core/adapters/init.py | Exposes adapter functions via package exports. |
| stitch/core/init.py | Exposes core stitcher APIs and models. |
| stitch/configs/stitching_defaults.json | Provides default stitch configuration (detectors, outputs, tiling configs, masks). |
| stitch/init.py | Declares the stitch package. |
| stitch_tasks.py | Adds Prefect tasks to run anchor-mode stitch and verify output artifacts. |
| prefect.yaml | Updates deployment configuration (repo source + workflow_options parameters). |
| pixi.toml | Adds runtime deps needed for stitching (numpy, pillow). |
| linker.py | Strengthens proposal/experiment path handling and adds proposal-layout helpers. |
| end_of_run_workflow.py | Wires stitch tasks into end-of-run flow and improves Slack notifications. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+26
to
+30
| def _resolve_stitch_path(path_value, default: Path) -> Path: | ||
| """Return an absolute path, interpreting relative overrides under ``stitch/``.""" | ||
| path_value = default if path_value in (None, "") else path_value | ||
| path = Path(path_value).expanduser() | ||
| return path if path.is_absolute() else STITCH_PACKAGE_DIR / path |
Comment on lines
+135
to
+138
| mode = next(iter(mode_set)) | ||
| labels = [t.metadata.stitch_tile_label for t in tiles] | ||
| validate_mode_labels(mode, labels) | ||
|
|
Comment on lines
+38
to
+40
| def proposal_directory(doc): | ||
| """Return the proposal directory for a run start document.""" | ||
| return PROPOSAL_ROOT / doc["cycle"] / doc["data_session"] |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
swu4bnl
marked this pull request as ready for review
August 6, 2026 14:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds an automated data-stitching workflow for CMS beamline data processing. The workflow reads scan metadata, identifies tiled datasets that should be stitched, runs the stitching code, writes processed outputs to the proposal folder, and sends status messages to Slack.
What is new
Review requested
Dan Allan (@danielballan) and Jun Aishima (@JunAishima), could you please review the following areas?
1. Tiled usage
The workflow currently uses key search on
cms/migrationas the primary method to find matching scans for stitching. If that fails or times out, it falls back to looking through ~100–500 recent entries.Please check whether this query strategy is reasonable, and whether Prefect retry/backoff should be used instead of, or together with, the fallback loop.
Relevant code:
2. Data security / permissions
The workflow writes stitched outputs into the user proposal folder. Please check whether the processed data receive the correct proposal-group access and are not exposed outside the intended proposal folder.
Relevant code:
3. Deployment / dependencies
Please check whether the Pixi dependencies and Prefect deployment configuration are appropriate for the CMS Prefect worker environment.
Relevant files:
4. DSSI standards
Please also flag anything else that should be adjusted to better follow DSSI standards before this is run routinely in production.
Testing plan
We have tested this workflow in the
devworkspace with manual triggering. Next, we plan to test in theprodworkspace using dry Bluesky measurement plans at CMS. The Kafka consumer will trigger the workflow at a rate similar to normal user operation.All test data will be generated under proposal
320406(beamline commissioning).We will monitor:
Change Log
stitchpackage for CMS auto-stitching, including core logic, configuration, adapters, default configuration, and required mask files.stitch/end_of_run_workflow. The workflow now loads stitch settings fromworkflow_options, launches the anchor auto-stitch task if enabled, and verifies outputs when configured.end_of_run_workflow.pyprefect.yamlto allow passingworkflow_options, including enabling/disabling anchor auto-stitching, output verification, and custom catalog paths for stitching.prefect.yamlnumpyandpillow.pixi.tomlend_of_run_workflow.pylinker.pyto prevent directory traversal in relative paths and to clean up proposal/experiment directory logic.linker.pyprefect.yamltohttps://github.com/swu4bnl/cms-workflows.git; this will be overwritten after deployment.prefect.yamlAuthor
Faid Faisal (@faidfaisal) (SULI/Stony Brook), swu4bnl