From b2f52fb01d1f1eec6c694a93624ac30ac338b8ea Mon Sep 17 00:00:00 2001 From: Mark Schreiber Date: Fri, 31 Jul 2026 16:50:43 -0700 Subject: [PATCH] adds steering for regional-capabilities, image stagins and requester pays buckets --- aws-healthomics/POWER.md | 2 + aws-healthomics/steering/image-staging.md | 255 ++++++++++++++++++ .../steering/regional-capabilities.md | 4 - .../steering/running-a-workflow.md | 36 +++ 4 files changed, 293 insertions(+), 4 deletions(-) create mode 100644 aws-healthomics/steering/image-staging.md diff --git a/aws-healthomics/POWER.md b/aws-healthomics/POWER.md index 16f66ed..0ae0479 100644 --- a/aws-healthomics/POWER.md +++ b/aws-healthomics/POWER.md @@ -17,6 +17,7 @@ Whenever you are asked to perform a task related to any of the following scenari - Creating a workflow from a remote Git repository URL (GitHub, GitLab, Bitbucket) -> use `./steering/git-integration.md` (takes precedence over workflow-development.md) - Creating a new WDL, Nextflow or CWL workflow from local files -> use `./steering/workflow-development.md` - Running a deployed HealthOmics workflow -> use `./steering/running-a-workflow.md` +- Using S3 Requester Pays buckets with HealthOmics workflow runs -> use `./steering/running-a-workflow.md` - Submitting, monitoring, or managing batch runs (multiple samples/runs at once) -> use `./steering/batch-runs.md` - Onboarding an existing WDL workflow ensuring compatibility with HealthOmics -> use `./steering/migration-guide-for-wdl.md` - Onboarding an existing Nextflow workflow ensuring compatibility with HealthOmics -> use `./steering/migration-guide-for-nextflow.md` @@ -24,6 +25,7 @@ Whenever you are asked to perform a task related to any of the following scenari - Diagnosing workflow creation issues -> use `./steering/troubleshooting.md` - Diagnosing run failures -> use `./steering/troubleshooting.md` - Using public containers with HealthOmics via ECR Pullthrough Caches -> use `./steering/ecr-pull-through-cache.md` +- Using containers from registries not supported by ECR Pull-Through Cache (Seqera Wave, NVIDIA NGC, Google Artifact Registry) via image staging -> use `./steering/image-staging.md` - Setting up VPC infrastructure for HealthOmics workflows (subnets, NAT Gateway, security groups, endpoints) -> use `./steering/vpc-setup.md` - Managing HealthOmics VPC configurations (creating, listing, getting, or deleting configurations) -> use `./steering/healthomics-configuration.md` - Running workflows with VPC connectivity, public internet access, cross-region access, or access to private VPC resources -> use `./steering/vpc-connected-workflow-runs.md` diff --git a/aws-healthomics/steering/image-staging.md b/aws-healthomics/steering/image-staging.md new file mode 100644 index 0000000..8e3ac79 --- /dev/null +++ b/aws-healthomics/steering/image-staging.md @@ -0,0 +1,255 @@ +# SOP: Image Staging for Non-PTC Registries + +## Purpose + +This SOP defines how you, the agent, configure image staging so that HealthOmics workflows can use container images from registries NOT supported by ECR Pull-Through Cache — specifically Seqera Wave, NVIDIA NGC, and Google Artifact Registry. Image staging automatically copies external container images into the customer's ECR at run time and caches them for subsequent runs. + +## Trigger Conditions + +Follow this SOP WHEN: +- User wants to use containers from Seqera Wave (`community.wave.seqera.io`), NVIDIA NGC (`nvcr.io`), or Google Artifact Registry (`-docker.pkg.dev` — e.g., `us-docker.pkg.dev`, `eu-docker.pkg.dev`, `us-west1-docker.pkg.dev` — or the legacy `gcr.io`). +- User references a container registry that is NOT supported by ECR Pull-Through Cache. +- User asks about image staging or registry staging. + +DO NOT follow this SOP WHEN: +- User wants to use containers from Docker Hub, Quay.io, or ECR Public — use the [ECR Pull-Through Cache SOP](./ecr-pull-through-cache.md). +- User already has containers in a private ECR repository — no staging needed. + +## Key Concepts + +- **Image Staging** — HealthOmics pulls container images from an external registry and pushes them into the customer's ECR before the workflow task runs. Staged images are cached and reused on subsequent runs. +- **Registry Staging Mapping** — A configuration entry mapping an upstream registry URL to an ECR repository prefix with credentials. +- **Resolution Order** — At task launch time: ImageMapping → RegistryMapping (PTC) → Registry Staging Mapping. Staging only triggers for non-PTC registries with a mapping configured. +- **Caching** — First run pulls and pushes to ECR (slower). Subsequent runs find the image in ECR and skip the upstream pull. +- **Staleness Detection** — HealthOmics compares the SHA-256 manifest digest of the staged image against the upstream to detect updates. + +## Supported Registries + +Each supported registry uses a slightly different secret shape. All three use the field name `accessToken` for the token — the differences are what other fields (if any) are present and what the `username` value must be. **Watch these carefully — a mismatch causes authentication to fail silently at first-run time:** + +1. **Seqera secrets contain only `accessToken`** — no `username` (any `username` field is ignored by HealthOmics). NVIDIA and Google secrets contain both `username` and `accessToken`. +2. **NVIDIA and Google require literal `username` values** — `$oauthtoken` (NVIDIA), and either `_json_key` or `oauth2accesstoken` (Google, depending on credential type). Do NOT substitute them with a customer-specific value. +3. **Google Artifact Registry supports two credential types** — a long-lived service-account JSON key (`_json_key`) or a short-lived OAuth 2.0 access token (`oauth2accesstoken`). See the Google Artifact Registry subsection below for both formats. + +Under the hood, HealthOmics performs a registry login using Basic Auth `Username:Token`. Each registry's protocol dictates what the username and token need to be; the JSON shape below is what Secrets Manager must store. + +Authentication is **required** for all non-PTC registries. + +### Seqera Wave — `community.wave.seqera.io` + +The secret contains a single field. HealthOmics ignores any `username` field, so do NOT include it — some earlier docs and third-party examples show a `username` alongside the token, but the service reads only `accessToken`. + +| Field | Value | +|-------|-------| +| `accessToken` | A Seqera Tower access token | + +Secret JSON: +```json +{ + "accessToken": "eyJ..." +} +``` + +Where to obtain the token: log in at https://seqera.io/wave/, then create a token at https://cloud.seqera.io/tokens. + +### NVIDIA NGC — `nvcr.io` + +| Field | Value | +|-------|-------| +| `username` | The **literal** string `$oauthtoken` (do not replace with a customer value) | +| `accessToken` | An NGC API key | + +Secret JSON: +```json +{ + "username": "$oauthtoken", + "accessToken": "nvapi-..." +} +``` + +Where to obtain the API key: log in at https://ngc.nvidia.com/, then generate a key at https://org.ngc.nvidia.com/setup/api-key. NGC treats `$oauthtoken` as the well-known username paired with the API key as the password. + +### Google Artifact Registry — `-docker.pkg.dev` (or legacy `gcr.io`) + +Common hosts: `us-docker.pkg.dev`, `eu-docker.pkg.dev`, `asia-docker.pkg.dev` (multi-region), or `us-west1-docker.pkg.dev`, `europe-west4-docker.pkg.dev`, `asia-northeast1-docker.pkg.dev`, etc. (single-region). The legacy Container Registry host `gcr.io` also works. + +Google Artifact Registry supports **two credential formats**. Both use the `accessToken` field name in the secret; the difference is the `username` value and what goes in `accessToken`. Pick one: + +**Option A — Service-account JSON key (long-lived).** Simplest for automation. Google flags this as less secure because the key is long-lived — prefer Option B when the workflow can refresh a short-lived token. + +| Field | Value | +|-------|-------| +| `username` | The **literal** string `_json_key` (do not replace with a customer value) | +| `accessToken` | The GCP service-account JSON key, stored as an escaped JSON string in this field | + +Secret JSON: +```json +{ + "username": "_json_key", + "accessToken": "{\"type\":\"service_account\",\"project_id\":\"...\",\"private_key\":\"...\",\"client_email\":\"...\"}" +} +``` + +Where to obtain: in the GCP project that owns the Artifact Registry, create a service account, grant it the `Artifact Registry Reader` role, then create a JSON key. Paste the entire JSON key object (as an escaped string) into the `accessToken` field. + +**Option B — Short-lived OAuth 2.0 access token.** Google's recommended approach. The token is valid for **60 minutes**, so the secret value MUST be refreshed before expiry — this option only fits workflows where an external process rotates the Secrets Manager value on a cadence shorter than 1 hour. If staging pulls the image after the token expires, authentication fails. + +| Field | Value | +|-------|-------| +| `username` | The **literal** string `oauth2accesstoken` (do not replace with a customer value) | +| `accessToken` | A short-lived OAuth 2.0 access token (starts with `ya29.`) | + +Secret JSON: +```json +{ + "username": "oauth2accesstoken", + "accessToken": "ya29..." +} +``` + +Where to obtain the token: generate it for a service account with the `Artifact Registry Reader` role: + +```bash +gcloud auth print-access-token \ + --impersonate-service-account SERVICE_ACCOUNT_EMAIL +``` + +Then write the returned token into the secret's `accessToken` field via `aws secretsmanager put-secret-value` (or `update-secret`). Automate this rotation before the 60-minute TTL elapses. + +Reference: https://cloud.google.com/artifact-registry/docs/docker/authentication + +## Prerequisites + +1. An AWS account with HealthOmics access. +2. Permission to create ECR repositories in the same Region as the workflow. +3. An IAM run role (see Step 1). +4. Credentials from the upstream registry provider. + +## Procedure + +### Step 1: Create or Verify the IAM Run Role + +The run role must allow HealthOmics to assume it (`omics.amazonaws.com` trust policy) and must have: +- **ECR write**: `ecr:BatchGetImage`, `ecr:GetDownloadUrlForLayer`, `ecr:BatchCheckLayerAvailability`, `ecr:InitiateLayerUpload`, `ecr:UploadLayerPart`, `ecr:CompleteLayerUpload`, `ecr:PutImage`, `ecr:DescribeImages`, `ecr:CreateRepository`, `ecr:GetRepositoryPolicy`, `ecr:SetRepositoryPolicy` on `arn:aws:ecr:*:*:repository/*` +- **ECR auth**: `ecr:GetAuthorizationToken` on `*` +- **Secrets Manager**: `secretsmanager:GetSecretValue` scoped to the secret ARN(s) used as `credentialArn` in the configuration mappings (e.g., `arn:aws:secretsmanager:::secret:-*`). The secret name is chosen by the customer at creation time (Step 2) or by the HealthOmics UI when it auto-creates the secret, so use the actual ARN(s) rather than a fixed prefix. +- **S3**: `s3:GetObject`, `s3:PutObject`, `s3:ListBucket` on `arn:aws:s3:::` and `arn:aws:s3:::/*` +- **CloudWatch Logs**: `logs:CreateLogGroup`, `logs:CreateLogStream`, `logs:DescribeLogStreams`, `logs:PutLogEvents` on `arn:aws:logs:*:*:log-group:/aws/omics/WorkflowLog:*` + +### Step 2: Store Registry Credentials in Secrets Manager + +Store credentials in the SAME region as the workflow. Use the JSON shape from the per-registry subsection above (Seqera Wave / NVIDIA NGC / Google Artifact Registry) — the field names differ between registries and the literal `username` values (`$oauthtoken`, `_json_key`) must be preserved exactly. The secret name is an arbitrary customer choice — use whatever fits your existing naming conventions. When the HealthOmics UI auto-creates the secret for a registry mapping, it may pick its own name; in that case, use the ARN it returns. + +```bash +aws secretsmanager create-secret \ + --name "" \ + --secret-string '' \ + --region +``` + +Note the secret ARN from the output. + +### Step 3: Create the HealthOmics Configuration + +Call `CreateAHOConfiguration` with registry staging mappings. Each mapping takes these fields: + +| Field | Description | +|-------|-------------| +| `upstreamRegistryUrl` | Registry hostname (e.g., `community.wave.seqera.io`) | +| `ecrRepositoryPrefix` | Prefix for staged ECR repositories | +| `upstreamRepositoryPrefix` | Path prefix in the upstream registry (e.g., `library`) | +| `ecrAccountId` *(optional)* | AWS account ID for ECR repositories | +| `credentialArn` | Secrets Manager ARN with registry credentials | + +**Example (Seqera Wave):** +```json +{ + "name": "seqera-wave-image-staging", + "description": "Image staging for Seqera Wave containers", + "workflowConfigurations": { + "stagingConfig": { + "registryStagingConfiguration": { + "registryStagingMappings": [ + { + "upstreamRegistryUrl": "community.wave.seqera.io", + "ecrRepositoryPrefix": "seqera-wave-staged", + "upstreamRepositoryPrefix": "library", + "ecrAccountId": "123456789012", + "credentialArn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:seqera-wave-credentials-AbCdEf" + } + ] + } + } + }, + "runConfigurations": { + "roleArn": "arn:aws:iam::123456789012:role/HealthOmicsStagingRole" + } +} +``` + +For **NVIDIA NGC**, replace: `upstreamRegistryUrl` → `nvcr.io`, `upstreamRepositoryPrefix` → `nvidia/clara`, prefix → `nvidia-ngc-staged`. + +For **Google Artifact Registry**, replace: `upstreamRegistryUrl` → `us-docker.pkg.dev`, `upstreamRepositoryPrefix` → `/`, prefix → `google-ar-staged`. + +Multiple registries can be combined in one configuration by adding entries to the `registryStagingMappings` array. + +### Step 4: Wait for Configuration to Become Active + +Call `GetAHOConfiguration` and wait for status `ACTIVE` before starting runs. + +### Step 5: Start the Run + +Reference the configuration by name. The container image URI can be a workflow parameter or hardcoded in the `runtime` block. + +```bash +aws omics start-run \ + --workflow-id \ + --role-arn arn:aws:iam::123456789012:role/HealthOmicsStagingRole \ + --output-uri s3://my-output-bucket/runs/ \ + --configuration-name seqera-wave-image-staging \ + --parameters '{"runtime_image": "community.wave.seqera.io/library/coreutils_grep_gzip_lbzip2_pruned:838ba80435a629f8"}' \ + --request-id "$(uuidgen)" +``` + +**What happens:** HealthOmics checks ECR for a cached copy → if absent, authenticates with the upstream registry, pulls the image, pushes to `.dkr.ecr..amazonaws.com//:` → runs the task with the staged image. + +### Step 6: Verify Results + +Check ECR — the staged image appears under `ecrRepositoryPrefix`. Example: `123456789012.dkr.ecr.us-east-1.amazonaws.com/seqera-wave-staged/community.wave.seqera.io/library/coreutils_grep_gzip_lbzip2_pruned:838ba80435a629f8` + +## Example WDL Workflow + +Pass the container image as a parameter so staging works without modifying the workflow: + +```wdl +version 1.0 + +workflow TestFlow { + input { + File input_fasta_file + String runtime_image + } + call ProcessTask { input: input_fasta_file = input_fasta_file, runtime_image = runtime_image } + output { File result = ProcessTask.output_file } +} + +task ProcessTask { + input { + File input_fasta_file + String runtime_image + } + runtime { docker: runtime_image } + command { head -n 1000 ~{input_fasta_file} > outfile.fasta } + output { File output_file = "outfile.fasta" } +} +``` + +## Limitations + +- **First run is slower** — waits for image staging. Subsequent runs use the cached ECR copy. +- **Authentication required** — HealthOmics rejects staging mappings without a `credentialArn` for non-PTC registries. +- **Same-region** — Secrets Manager secret and ECR must be in the same Region as the workflow. +- **Large images take time** — NVIDIA Parabricks (~30 GB) may take significant time on first use. +- **PTC registries cannot use staging** — HealthOmics rejects staging mappings for PTC-supported registries. +- **Tags and digests supported** — Both `:latest` / `:1.2.3` and `@sha256:...` references work. +- **Concurrent first runs** — If multiple runs reference the same unstaged image, it's staged only once; others wait. diff --git a/aws-healthomics/steering/regional-capabilities.md b/aws-healthomics/steering/regional-capabilities.md index 5c872b9..4164882 100644 --- a/aws-healthomics/steering/regional-capabilities.md +++ b/aws-healthomics/steering/regional-capabilities.md @@ -4,10 +4,6 @@ This document describes region-specific capabilities and limitations for AWS HealthOmics. Not all HealthOmics features are available in all regions. -## Authoritative Source - -For the most up-to-date information on supported regions, instance types, and features, refer to the [AWS HealthOmics public documentation](https://docs.aws.amazon.com/omics/latest/dev/memory-and-compute-tasks.html) and the [HealthOmics endpoints and quotas page](https://docs.aws.amazon.com/general/latest/gr/healthomics-quotas.html). IF the information in this document appears inconsistent with what public documentation states, defer to the public documentation. - ## When to Reference This Document Reference this document when: diff --git a/aws-healthomics/steering/running-a-workflow.md b/aws-healthomics/steering/running-a-workflow.md index 8134185..745137d 100644 --- a/aws-healthomics/steering/running-a-workflow.md +++ b/aws-healthomics/steering/running-a-workflow.md @@ -55,6 +55,42 @@ Behavior to know (Nextflow profiles): - Explicit run parameters (in `parameters.json`) override profile-defined parameter values. - Recommend pinning `manifest.nextflowVersion` in the workflow when profiles are in use, so profile application is consistent across runs. +### Requester Pays + +If a workflow reads from or writes to S3 Requester Pays buckets, the run must opt in. There are two ways to enable it: + +**Option 1 — Per-run flag:** +```bash +aws omics start-run \ + --workflow-id \ + --role-arn arn:aws:iam:::role/ \ + --output-uri s3:///outputs/ \ + --parameters '{"input_file": "s3:///data/file.txt"}' \ + --requester-pays-enabled +``` + +Use `--no-requester-pays-enabled` to explicitly disable (the default). + +**Option 2 — Configuration-level (reusable across runs):** +```bash +aws omics create-configuration \ + --name requester-pays-config \ + --description "Enables S3 Requester Pays access for runs" \ + --run-configurations '{"requesterPaysEnabled": true}' +``` + +Then reference it when starting the run: +```bash +aws omics start-run \ + --workflow-id \ + --role-arn arn:aws:iam:::role/ \ + --output-uri s3:///outputs/ \ + --configuration-name requester-pays-config \ + --parameters '{"input_file": "s3:///data/file.txt"}' +``` + +Without requester-pays enabled, runs that access Requester Pays buckets will fail with an S3 access error. + ### Handling Failures IF the workflow run fails: