Skip to content
BranchesTags

Repository files navigation

BIOMERO BIOMERO.scripts

Test BIOMERO scripts

🚀 This package is part of BIOMERO BIOMERO 2.0 — For complete deployment and FAIR infrastructure setup, start with the NL-BIOMERO Documentation 📖

These scripts provide a comprehensive OMERO integration for running bioimage analysis workflows on SLURM clusters.

Key Features

  • Multi-format support: TIFF, OME-TIFF, and (OME-)ZARR
  • Automatic data export from OMERO to SLURM clusters
  • Intelligent format conversion with optimization
  • Comprehensive workflow tracking and monitoring
  • Automatic result import back to OMERO
  • Configurable output organization options
  • Optional detached execution that survives browser and requesting-session expiry

These scripts work together with the BIOMERO library to enable seamless bioimage analysis workflows directly from OMERO.

Deploy with NL-BIOMERO

For the easiest deployment and integration with other FAIR infrastructure, consider using the NL-BIOMERO stack:

The NL-BIOMERO stack provides Docker Compose configurations that automatically set up OMERO.web with the OMERO.biomero plugin, databases, and all necessary dependencies.

Overview

In the figure below we show our BIOMERO framework, for BioImage analysis in OMERO.

BIOMERO consists of the Python library BIOMERO and the integrations within OMERO through the scripts in this repository.

OMERO-Figure1_Overview_v5

BIOMERO 2.0 Web Interface

In addition to these command-line scripts, BIOMERO 2.0 introduces a modern web-based user interface through the OMERO.biomero web plugin. This plugin provides:

  • Interactive Workflow Management: Browse and launch workflows with a modern web interface
  • Real-time Progress Tracking: Monitor job progress with live updates
  • Workflow History: View past executions with full tracking and metadata
  • Dashboard Overview: Get an overview of all your workflows at a glance

For new users, we recommend the NL-BIOMERO stack with the web interface for the complete experience. These scripts remain fully supported for advanced users who need custom scripting capabilities.

Script Architecture

Main Workflow Scripts (__workflows/)

  • SLURM_Run_Workflow.py: Primary workflow orchestrator with ZARR support
  • SLURM_Run_Workflow_Batched.py: Batch processing variant for multiple datasets
  • SLURM_CellPose_Segmentation.py: ⚠️ EXAMPLE ONLY - Manual single-workflow script for CellPose. Not installed by default in NL-BIOMERO. Use SLURM_Run_Workflow.py instead.

Data Management Scripts (_data/)

  • _SLURM_Image_Transfer.py: Export data from OMERO to SLURM (with cleanup)
  • _SLURM_File_Transfer.py: Transfer a single OMERO FileAnnotation to a SLURM job's input directory (e.g. model weights, CSV config). Returns the resolved SLURM path for injection as a workflow CLI argument.
  • SLURM_Remote_Conversion.py: Intelligent format conversion on SLURM
  • SLURM_Get_Results.py: Upload workflow results back to OMERO (standard mode)
  • SLURM_Import_Results.py: Import workflow results with full BIOMERO.importer integration — selected automatically when IMPORTER_ENABLED=true
  • SLURM_Get_Update.py: Monitor and update workflow status

Administrative Scripts (admin/)

  • SLURM_Init_environment.py: Initialize SLURM environment
  • SLURM_check_setup.py: Validate BIOMERO configuration
  • SLURM_Cownary.py: Run a fixed, admin-only lolcow cownary to verify SSH, Slurm scheduling, configured shared storage, and Singularity execution end to end. It accepts no command or path input and inherits BIOMERO's default partition, global sbatch_* settings, and configured Apptainer cache, temporary, and bind paths without allowing them to override the fixed cownary job scope.
  • Tail_logs.py: View recent BIOMERO log entries (admin only)
  • Example_Minimal_Slurm_Script.py: ⚠️ Admin/example only — runs ad-hoc SSH commands on the Slurm cluster from OMERO.web. Requires OMERO admin privileges. Note that a compromised admin account can already upload arbitrary scripts and reach the cluster that way, so this adds convenience rather than a new attack surface — but it's still not needed in production. See the note below.

Workflow Process

  1. Export: Selected data transferred from OMERO to SLURM cluster
  2. Convert: Smart format conversion (with ZARR no-op optimization)
  3. Process: Computational workflows executed on SLURM
  4. Monitor: Job progress tracking and status updates (with real-time polling when SlurmClient is available)
  5. Import: Results imported back to OMERO — via SLURM_Import_Results.py (importer-enabled) or SLURM_Get_Results.py (standard), selected automatically based on IMPORTER_ENABLED
  6. Cleanup: Temporary artifacts automatically removed (non-critical cleanup errors are logged but do not fail the workflow)

Optional detached execution

New in BIOMERO.scripts 2.9: BIOMERO_DETACHED_WORKFLOWS is an opt-in feature flag. Installing the updated scripts does not change existing workflow behavior while BIOMERO_DETACHED_WORKFLOWS is absent or false. Existing and custom deployments remain inline until an administrator enables the feature and provides the required background worker supervisor.

Set BIOMERO_DETACHED_WORKFLOWS=true only when the deployment also provides a compatible detached workflow supervisor, such as the biomeroworker in NL-BIOMERO. SLURM_Run_Workflow.py and its batched variant then validate and queue the request before returning. The supervisor performs transfer, conversion, Slurm monitoring, and result import in the background.

Once the script reports that the workflow is queued in the background, the run no longer depends on the browser tab or the OMERO session that submitted it. Administrators do not need seven-day or infinite OMERO sessions, an unusually large OMERO.web cookie age, or an open browser merely to cover the total Slurm runtime. Ordinary timeouts must still cover the initial queue hand-off and each OMERO-side transfer or import subprocess. If detached mode is absent, disabled, or unsupported by the installed BIOMERO library, the scripts retain their established inline behavior and the session must remain active.

See the NL-BIOMERO detached-workflow administrator guide for deployment, recovery, and verification details.

Dynamic Import Script Selection

The import step automatically selects the right script based on your environment:

IMPORTER_ENABLED Script used Dataset import method
false (default) SLURM_Get_Results.py Upload via OMERO API
true SLURM_Import_Results.py In-place import from remote storage via BIOMERO.importer

Set IMPORTER_ENABLED=true in your environment (e.g. docker-compose .env) to enable in-place imports via BIOMERO.importer. The script will raise an error at startup if IMPORTER_ENABLED=true but the BIOMERO.importer module is not installed.

Installation

  1. Change into the scripts location of your OMERO installation

     cd /opt/omero/server/OMERO.server/lib/scripts/

  2. Clone the repository with a unique name (e.g. "biomero")

      git clone https://github.com/NL-BioImaging/biomero-scripts.git biomero

  3. Update your list of installed scripts by examining the list of scripts in OMERO.insight or OMERO.web, or by running the following command

     <path>/<to>/<bin>/omero script list

  4. Install system requirements on the PROCESSOR nodes:

    • If using biomero[full]: Most dependencies are included (ezomero>=1.1.1, tifffile>=2020.9.3, omero-metadata>=0.12.0, omero-cli-zarr>=0.6.1)
    • Manual installation (if using basic biomero without [full] extras):
      • python3 -m pip install biomero ezomero==1.1.1 tifffile==2020.9.3 omero-metadata==0.12.0
    • the OMERO CLI Zarr plugin, e.g. python3 -m pip install omero-cli-zarr==0.6.1 && yum install -y blosc-devel
    • the bioformats2raw-0.11.0, e.g. unzip -d /opt bioformats2raw-0.11.0.zip && export PATH="$PATH:/opt/bioformats2raw-0.11.0/bin"

These examples work on Linux CentOS (i.e. the official OMERO containers); for Windows, or other Linux package managers, check with the original repositories (OMERO CLI ZARR and BioFormats2RAW) for more details on installation.

Requirements

Just to reiterate, you need all these requirements installed to run all these scripts, on the OMERO PROCESSOR node:

  • Python libraries:
    • biomero[full] (latest version, includes ezomero>=1.1.1, tifffile>=2020.9.3, omero-metadata>=0.12.0, omero-cli-zarr>=0.6.1)
    • OR manual installation: biomero (latest version, or at least matching the version number of this repository) + individual packages:
      • ezomero==1.1.1
      • tifffile==2020.9.3
      • omero-metadata==0.12.0
      • omero-cli-zarr>=0.6.1 (see below)
  • the OMERO CLI Zarr plugin, e.g. python3 -m pip install omero-cli-zarr>=0.6.1 && yum install -y blosc-devel
  • the bioformats2raw-0.11.0, e.g. unzip -d /opt bioformats2raw-0.11.0.zip && export PATH="$PATH:/opt/bioformats2raw-0.11.0/bin"

Upgrading

  1. Change into the repository location cloned into during installation

     cd /opt/omero/server/OMERO.server/lib/scripts/<UNIQUE_NAME>

  2. Update the repository to the latest version

     git pull --rebase

  3. Update your list of installed scripts by examining the list of scripts in OMERO.insight or OMERO.web, or by running the following command

     <path>/<to>/<bin>/omero script list

Use the BIOMERO.scripts

This repository provides example OMERO scripts for using BIOMERO. These scripts do not work without installing that client on your OMERO servers/processors that will run these scripts.

Always start with initiating the Slurm environment at least once, for example using admin/SLURM Init environment. With scheduler-native image pulls enabled, this submits one bounded Slurm array for workflow and converter images and returns its array job ID. Run SLURM Check Setup to see exact READY, RUNNING, and FAILED counts plus concise per-image failure reasons; reruns skip valid versioned SIFs.

Shared group folder mappings

SLURM_Import_Results.py supports both the legacy /opt/omero/server/biomero-config.json["group_mappings"] configuration and an optional dedicated /opt/omero/server/group-mappings.json file. Mappings are merged by group key. Entries found only in either source are retained, and the dedicated file wins when the same group is present in both.

Override the default paths with OMERO_BIOMERO_CONFIG_FILE and OMERO_BIOMERO_GROUP_MAPPINGS_FILE. Deployments that do not mount the dedicated file continue using the legacy configuration unchanged.

When BIOMERO_SHALLOW_ZARR=true, Image Transfer also derives each group's managed storage root at runtime as IMPORT_MOUNT_PATH / mapping.folder for canonical Zarr promotion and reuse. There is no separate storage_roots configuration. The processor worker must receive the same read-only mapping files that OMERO.biomero edits and the same shared-storage mount; mappings are read for each script execution, so runtime changes do not require an image rebuild.

Optional shallow Zarr storage

BIOMERO_SHALLOW_ZARR defaults to false. It is effective only together with IMPORTER_ENABLED=true:

  • false: Image Transfer exports normally and Import Results imports normally;
  • true: Image Transfer may promote/reuse a verified canonical Zarr, and Import Results submits a typed biomero.shallow-zarr operation with the exact workflow input snapshot. BIOMERO.importer performs identity comparison, fail-safe normalization, and registration planning in its independent service after the order is committed.

Run Workflow distinguishes complete Zarr inputs from temporary conversion material. A workflow that consumes Zarr receives a reconstructed shallow input containing the canonical original pixels and every managed label. When the selected workflow consumes TIFF, a shallow-backed OMERO Image instead follows the established OMERO CLI Zarr export path: the Image's registered PixelBuffer is exported as a standalone temporary Zarr and then converted to TIFF. This preserves a selected mask Image as mask pixels, avoids transferring unrelated original pixels and labels, and deliberately excludes the temporary export from canonical promotion and returned-Zarr matching. Plates always use the complete Zarr path.

The OMERO script does not hash or mutate returned Zarrs. If the deployed importer does not advertise the lifecycle operation, or no canonical workflow snapshot is available, it uses the established full-import path. Existing legacy label-result controls remain unchanged in that fallback. Identity worker concurrency is configured on BIOMERO.importer, so a web/script timeout cannot stop the expensive Plate comparison after hand-off.

Canonical Plate identities are indexed in OMERO as one compact Plate record plus bounded image- and label-node records. This keeps large Plate metadata below OMERO/PostgreSQL MapAnnotation value limits; existing monolithic records remain readable.

Eligible Image results expose their labels as ordinary OMERO Image projections until label-aware viewers are generally available. Eligible HCS results remain one derived OMERO Plate: its WellSample pixels are served from the canonical source Plate while the in-place shallow collection retains the image-level labels. This avoids flattening a large Plate into thousands of loose mask Images.

Import Plate label preview is an optional result setting, disabled by default. It creates one additional Plate whose WellSample pixels point directly at one common image-level label. Supply Plate label preview name, or leave it empty only when exactly one label name occurs on every Plate image. The preview creates OMERO objects and PixelBuffer links but does not copy label arrays.

Importer-disabled deployments continue to use SLURM_Get_Results.py and do not load BIOMERO.importer Zarr helpers. The worker processor must forward this environment variable to downloaded scripts; current NL-BIOMERO deployments do that dynamically through biomero.constants.slurm_env.

Optional ROI postprocessing

SLURM_Run_Workflow.py can optionally turn imported grayscale label images into ROIs on their exact source images. Enable Create ROIs from label images, import the image results into a Dataset or Screen, and choose Polygon or Mask output. BIOMERO records each imported label-image ID together with the source-image ID it matched and passes those explicit pairs to the OMERO Labels2Rois utility script after import.

Created ROI names use workflow_name__workflow_uuid__label_value, making them filterable by algorithm or by an exact workflow run. By default, the workflow UUID also selects a deterministic color from a curated palette, so separate ROI runs are visually distinct. An optional #RRGGBB override can be supplied by clients such as OMERO.biomero. Labels2Rois applies that color as a translucent Mask fill or as a Polygon fill and outline. The optional Clear existing ROIs on original images setting forwards the native clear behavior; its case-sensitive name filter limits deletion, while an empty filter clears all existing ROIs on each original image. Clearing is disabled by default.

Imported label images are retained in OMERO by default. The optional Delete from OMERO after ROI creation setting forwards the native Labels2Rois cleanup flag, which deletes each imported label image only after its ROI conversion succeeds. This removes only the OMERO image; workflow result files in remote storage (including importer .analyzed storage) are preserved.

If every image output in the selected workflow descriptor has subtype label, all imported images are selected automatically. For mixed or descriptor-less workflows, BIOMERO groups imported results by their matched source image. A sole result is selected directly; with multiple results, label-like names such as mask, label, or segment are selected. Ambiguous groups are skipped without failing import. The lower-level result scripts retain an optional glob such as *_cp_masks.tif as an advanced override, matched before result-image renaming. A missing Labels2Rois script disables this optional step with a warning. Import and workflow completion remain successful, and result images are retained, if selection is ambiguous, the utility is missing, or postprocessing fails.

For example, __workflows/SLURM Run Workflow should provide an easy way to send data to Slurm, run the configured and chosen workflow, poll Slurm until jobs are done (or errors) and retrieve the results when the job is done. This workflow script uses some of the other scripts, like

  • _data/SLURM Image Transfer: to export your selected images / dataset / screen as ZARR files to a Slurm dir.
  • _data/SLURM Get Results: to retrieve your Slurm job results back into OMERO as a zip, dataset or attachment. Datasets are uploaded directly into OMERO via the OMERO API.
  • _data/SLURM Import Results: to retrieve your Slurm job results back into OMERO as a zip, dataset or attachment. Datasets are in-place imported from remote storage via BIOMERO.importer. Selected automatically when IMPORTER_ENABLED=true.

Other example OMERO scripts are:

  • admin/SLURM Cownary: Admin only — submits the fixed docker://godlovedc/lolcow cownary through Slurm, waits for completion, and displays the hostname and cow in OMERO. It uses BIOMERO's configured Slurm destination, default partition, global job parameters, and Apptainer environment while accepting no user-supplied command or path.
  • _data/SLURM Get Update: to run while you are waiting on a job to finish on Slurm; it will try to get a % progress from your job's logfile. Depends on your job/workflow logging a % of course.
  • _data/SLURM File Transfer: transfers a single OMERO FileAnnotation (e.g. model weights, a CSV) to the SLURM job's input directory. The returned SLURM path is injected as the CLI argument for the corresponding workflow parameter by SLURM_Run_Workflow.py.
  • admin/Example Minimal Slurm Script: ⚠️ Admin only — runs ad-hoc SSH commands on the Slurm cluster directly from OMERO. Useful for cluster diagnostics (squeue, sinfo, ls) and as a reference skeleton for building new admin scripts. Dangerous commands are blocked by pattern matching.

Note: Restricted to OMERO admins, who can already upload arbitrary scripts anyway — so this doesn't add a new attack surface. It's just not needed in most production deployments, which is why the NL-BIOMERO Dockerfile removes it during the image build (see the admin/ section above). Install it manually when you need it for diagnostics and remove it when you're done.

  • __workflows/SLURM Run Workflow Batched: This will allow you to run several __workflows/SLURM Run Workflow in parallel, by batching your input images into smaller chunks (e.g. turn 64 images into 2 batches of 32 images each). It will then poll all these jobs.

  • __workflows/SLURM CellPose Segmentation: ⚠️ Example only — a minimal script that runs only the CellPose workflow. You will need to manually transfer data first (with _data/SLURM Image Transfer) and manually retrieve data afterward (with _data/SLURM Get Results). Use as a reference if you want to build your own single-workflow script.

Logging Configuration

BIOMERO.scripts already have comprehensive DEBUG logging enabled by default! All scripts are configured with:

  • DEBUG level logging to rotating log files (biomero.log in /opt/omero/server/OMERO.server/var/log/)
  • INFO level logging to stdout (visible in OMERO.web script output)
  • Rotating log files (500MB max, 9 backups) to prevent disk space issues
  • Pre-silenced verbose libraries (omero.gateway.utils, paramiko.transport, invoke) at WARNING level

Current Logging Setup

Each script automatically configures logging like this:

if __name__ == '__main__':
    # Comprehensive DEBUG logging to rotating biomero.log file
    stream_handler = logging.StreamHandler(sys.stdout)
    stream_handler.setLevel(logging.INFO)  # Only INFO+ to stdout
    logging.basicConfig(level=logging.DEBUG,  # Full DEBUG to file
                        format="%(asctime)s %(levelname)-5.5s [%(name)40s] "
                               "[%(process)d] (%(threadName)-10s) %(message)s",
                        handlers=[
                            stream_handler,
                            logging.handlers.RotatingFileHandler(
                                os.path.join(LOGDIR, 'biomero.log'),
                                maxBytes=500000000, backupCount=9)
                        ])

    # Silence verbose libraries
    logging.getLogger('omero.gateway.utils').setLevel(logging.WARNING)
    logging.getLogger('paramiko.transport').setLevel(logging.WARNING)
    logging.getLogger('invoke').setLevel(logging.WARNING)
    
    runScript()

Reducing Log Verbosity (If Needed)

If the default DEBUG logging is too verbose, you can modify any script to use less logging:

# Change DEBUG to INFO for less verbose logging
logging.basicConfig(level=logging.INFO, ...)

# Or silence additional libraries
logging.getLogger('biomero').setLevel(logging.INFO)
logging.getLogger('fabric').setLevel(logging.WARNING)

Log File Locations

  • Main logs: /opt/omero/server/OMERO.server/var/log/biomero.log*
  • OMERO logs: Standard OMERO logging locations
  • Rotation: Logs rotate when reaching 500MB, keeping 9 backups

Usage Examples

Running a Standard Workflow (TIFF)

  1. Select your images, datasets, or plates in OMERO
  2. Run the SLURM Run Workflow script
  3. Choose your desired workflow (e.g., CellPose, StarDist)
  4. Configure workflow parameters
  5. Select output organization options
  6. Execute - data will be automatically exported, processed, and imported back

Using ZARR Format (New in v2.0.0-alpha.7)

  1. Select your data in OMERO
  2. Run the SLURM Run Workflow script
  3. ✅ Check "Use ZARR Format" for workflows that support native ZARR input
  4. Choose your ZARR-compatible workflow
  5. Configure parameters and output options
  6. Execute - conversion step will be skipped for efficiency

Manual Data Export

For advanced users who need custom processing:

  1. Use SLURM Image Transfer to export data in your preferred format
  2. Use SLURM Remote Conversion if format conversion is needed
  3. Process data using custom workflows on SLURM
  4. Use SLURM Get Results to import results back to OMERO

Monitoring and Debugging

  • SLURM Check Setup: Validate configuration and report per-image READY, RUNNING, or FAILED state
  • SLURM Get Update: Monitor workflow-job progress and retrieve its omero-<job-id>.log; image initialization does not use this path
  • SLURM Init Environment: Initialize or update SLURM environment

Image initialization logs are not combined into the legacy sing.log. Use SLURM Check Setup for structured per-image state and inspect the latest submission under <slurm_script_path>/image-pulls for its individual task logs.

Legal

See LICENSE. Note this is copy-left, as we built on OME's scripts with copy-left license.

About

This section provides machine-readable information about your scripts. It will be used to help generate a landing page and links for your work. Please modify all values on each branch to describe your scripts.

Repository name

BIOMERO.scripts repository

Minimum version

5.6

Maximum version

5.6

Owner(s)

T.T. Luik

Institution

Amsterdam UMC

URL

https://nl-bioimaging.github.io/biomero/

Email

t.t.luik@amsterdamumc.nl

Description

These scripts are to be used with the BIOMERO library.

They show how to use the library to run workflows directly from OMERO on a Slurm cluster.

About

Scripts for use with BIOMERO

nl-bioimaging.github.io/biomero/

Topics

biaflowsbioimagingbiomerocytominefairhigh-content-screeninghigh-performance-computinghigh-throughput-screeningimage-analysisomeroslurm

Resources

Readme
GPL-2.0 license
Activity
Custom properties

Stars

5 stars

Watchers

1 watching

Forks

2 forks
Report repository

Releases

Contributors

Languages

Generated from ome/omero-user-scripts