Skip to content

Latest commit

Β 

History

2,103 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

icon

SciWIn - Scientific Workflow Infrastructure

Rust πŸ¦€ Continuous Integration GitHub License GitHub Release GitHub Downloads

⭐ Star this Repo to say "Thank you!" ⭐

This is the 2.x Branch, for 1.x click this link

Share Share Share Share

πŸ“– Read the Documentation to get started or take a look at some examples! πŸš€

πŸ¦€ Take a look at our latest publications and talks πŸ‘€

or the FAIRagro Blogpost: SciWIn-Client – what is it actually?

πŸ“– Table of Contents

πŸš€ About

Computational workflows, which define complex, multi-step procedures for automated execution, are crucial for ensuring reproducibility, scalability, and efficiency in scientific research. The FAIRagro Scientific Workflow Infrastructure (SciWIn) empowers scientists to create, execute, share, and publish these workflows, promoting collaboration and transparency.

This repository consists of two complementary tools: SciWIn-Client and SciWIn-Studio (currently in testing).

Reproducibility in computational research is essential for effective collaboration, result verification, and maintaining transparency. However, it remains challenging due to complex workflows, inconsistent data management practices, and dependencies on specific software environments. SciWIn-Client is a command-line tool designed to simplify the creation, recording, annotation, and execution of computational workflows. It enables researchers to interactively use intuitive commands to track tasks such as data extraction, cleaning, transformation, analysis, visualization, and computational simulation. By automating and standardizing workflows, SciWIn-Client minimizes error sources and supports transparent, reproducible Open Science practices.

SciWIn-Studio provides a graphical user interface that makes workflow management accessible to researchers who prefer visual tools over command-line interactions, offering an intuitive alternative for designing and executing computational workflows without requiring terminal expertise.

πŸ—οΈ How to Build and Test

This project is being developed using Rust and Cargo. To run SciWIn-Client you can just use cargo run to run from source code, to build use cargo build.

To run the tests use cargo nextest or bacon nextest to run as watcher (implying that bacon and nextest are installed).

# Clone the repository
git clone https://github.com/fairagro/sciwin
cd sciwin

# Build the project
cargo build

# Run the project
cargo run

To run tests (unit and integration)

cargo nextest --workspace           # Run all tests

Crate architecture & sibling repositories

This repository is a Cargo workspace with three crates:

  • crates/sciwin (lib sciwin) β€” the shared core logic for authoring, executing and annotating workflows, used by both the CLI and the GUI.
  • crates/cli (bin s4n) β€” the SciWIn-Client command-line interface, a thin wrapper around sciwin.

The sciwin crate itself builds on three sibling FAIRagro libraries, each maintained in its own repository and published as its own crate:

Crate Repository Purpose
commonwl commonwl Parses CWL documents and provides the execution engine (local, Docker and TES backends) used by s4n execute
rocrate ro-crate-lib Reads, writes, builds and validates RO-Crates, including Workflow RO-Crates and Workflow Run Crates
reana reana-cwl-client Client for REANA, used as an alternative execution backend to local runs

By default these are pulled from crates.io like any other dependency, so a plain cargo build/cargo run works without checking out the other repositories. When developing a change that spans repositories (e.g. a commonwl API change consumed by sciwin), clone commonwl, reana-cwl-client and ro-crate-lib next to this repository and add a [patch.crates-io] section (in a workspace-level .cargo/config.toml one directory above all four checkouts, or directly in this repo's Cargo.toml) pointing each crate at its local path, e.g.:

[patch.crates-io]
commonwl = { path = "../commonwl/crates/commonwl" }
reana = { path = "../reana-cwl-client/crates/reana" }
rocrate = { path = "../ro-crate-lib" }

Building from a directory where such a patch is in scope will then use the local sources instead of the published versions, and all affected repositories should be updated together.

πŸ’š SciWIn-Studio

Note

SciWIn-Studio moved to its own repo: https://github.com/fairagro/sciwin_studio

πŸ’» SciWIn-Client

SciWIn-Client is a command-line tool designed to simplify the creation, recording, annotation, and execution of computational workflows.

🎯 Installation

Detailed installation instructions can be found at the latest release:

GitHub Release

The easiest way is to use the shell or powershell scripts with the provided commands. To install latests binaries you can use the following scripts:

Linux/MacOS:

curl --proto '=https' --tlsv1.2 -LsSf https://fairagro.github.io/sciwin/get_s4n.sh | sh 

If curl responds with code 429 you can try to use wget instead.

wget -qO- https://fairagro.github.io/sciwin/get_s4n.sh | sh 

Windows:

powershell -ExecutionPolicy Bypass -c "irm https://fairagro.github.io/sciwin/get_s4n.ps1 | iex"

πŸ“š How to Use

Take a look at the User documentation. An overview on how to use SciWIn-Client is available below.

Project initialization

Most commands need the context of a Git repo to work. Project initialization can be done using the s4n init command.

s4n init -p <FOLDER/PROJECT NAME>

Creation of CWL CommandLineTools

To create CWL CommandLineTools which can be combined to workflows later a prefix command can be used. s4n create will execute any given command and creates a CWL CommandLineTool accordingly.

s4n create <COMMAND> [ARGUMENTS]

The command comes with a lot of different options on how to handle the CWL creation specifically.

Creates a new CWL File or Workflow

Usage: s4n create [OPTIONS] [COMMAND]...

Arguments:
  [COMMAND]...  Command line call e.g. python script.py [ARGUMENTS]

Options:
  -n, --name <NAME>                        A name to be used for this workflow or tool
  -c, --container-image <CONTAINER_IMAGE>  An image to pull from e.g. docker hub or path to a Dockerfile
  -t, --container-tag <CONTAINER_TAG>      The tag for the container when using a Dockerfile
  -r, --raw                                Outputs the raw CWL contents to terminal
      --no-commit                          Do not commit at the end of tool creation
      --no-run                             Do not run given command
      --clean                              Deletes created outputs after usage
      --no-defaults                        Removes default values from inputs
      --net                                Enables network in container
  -i, --inputs <INPUTS>                    Force values to be considered as an input.
  -o, --outputs <OUTPUTS>                  Force values to be considered as an output.
  -m, --mount <MOUNT>                      Mounts a directory into the working directory
  -f, --force                              Overwrites existing workflow
  -h, --help                               Print help

Creation of CWL Workflows

CWL Workflows can be created semi-automatically using s4n connect commands. Connections to In- or Outputs are added using either @inputs or @outputs as file identifier.

s4n connect <NAME> --from [FILE]/[SLOT] --to [FILE/SLOT]

For example: s4n connect demo --from @inputs/speakers --to calculation/speakers - The Step calculation will be added pointing to workflows/calculation/calculation.cwl, which will use the newly created input speakers as input for its speakers input.

Execution of CWL Files

SciWIn-Client comes with its custom CWL Runner (which does not support all cwltool can do, yet!) to run Workflows and CommandLineTools. s4n execute defaults to its run subcommand, so it can be omitted (s4n execute can also be triggered using s4n ex):

s4n execute workflows/main/main.cwl inputs.yml

By default this runs on the local engine. --engine also accepts docker (every step containerized via Docker directly), tes (submit to a GA4GH TES server), and reana (submit to a REANA server, e.g. s4n execute --engine reana workflows/main/main.cwl inputs.yml) β€” see the execute reference for the full flag set and required environment variables per backend.

πŸͺ‚ Contributors

Made with contrib.rocks.

Measure 4.4
Jens Krumsieck :octocat: @jenskrumsieck ORCID: 0000-0001-6242-5846
Antonia Leidel :octocat: @aleidel ORCID: 0009-0007-1765-0527
Patrick KΓΆnig :octocat: @patrick-koenig ORCID: 0000-0002-8948-6793
Xaver Stiensmeier :octocat: @XaverStiensmeier ORCID: 0009-0005-3274-122X
Harald von Waldow :octocat: @hvwaldow ORCID: 0000-0003-4800-2833

Funded by

dfg

DFG project number 501899475

βš–οΈ License

This work is dual-licensed under Apache 2.0 and MIT . You can choose between one of them if you use this work. SPDX-License-Identifier: Apache-2.0 OR MIT

🏁 Quick Links

πŸ”Ό Back to Top

About

πŸ¦€ SciWIn: Reproducible computational Workflows made easy!

Topics

Resources

Stars

13 stars

Watchers

3 watching

Forks

Releases

Used by

Contributors

Languages