Skip to content

ci(deploy-prod): add the missing step that captures prod's pre-deploy SHA - #2364

Open
sajclarke wants to merge 1 commit into
mainfrom
fix-prod-affected-base
Open

ci(deploy-prod): add the missing step that captures prod's pre-deploy SHA#2364
sajclarke wants to merge 1 commit into
mainfrom
fix-prod-affected-base

Conversation

@sajclarke

Copy link
Copy Markdown
Contributor

The failure

Run 32409904357 — Deploy Production died at Detect affected targets:

❌ Could not read the prod ref, so the affected base is unknown.
   Re-run with deploy_all=true to deploy every target.

Root cause

#2165 replaced nx-set-shas with an affected base derived from the SHA prod was actually serving. It wired setup's PROD_BEFORE to an advance-prod output:

outputs:
  previous-sha: ${{ steps.previous.outputs.sha }}

…but never added a step with id: previous. advance-prod has only the staging guard and the fast-forward. GitHub resolves an unknown steps.<id> expression to the empty string rather than erroring, so previous-sha was always "" and the [ -n "$PROD_BEFORE" ] guard in setup fired on every run.

It stayed hidden because the only two prod deploys since #2165 — 2026-08-04 and 2026-08-18 — both ran with deploy_all=true, which returns early before that guard. 2026-08-20 was the first affected-detection prod deploy and it failed immediately. Latent since #2165, not a regression from anything recent.

Changes

  1. Add the capture step, positioned before the fast-forward — once the ref moves, the SHA prod was serving is unrecoverable.
  2. Gate release-notes on setup. It needs always() so a skipped deploy job (the normal affected-detection outcome) still cuts a release, but with only advance-prod required it tagged a release for a run that deployed nothing — prod-2026-08-20-77e5ae2 exists for the failed run.

The setup guard itself was correct and did its job: it refused to under-deploy rather than silently shipping a partial release, which is exactly the failure mode #2165 was written to prevent. It just never had a producer.

⚠️ Before the next production deploy

Run 32409904357's advance-prod succeeded — the prod ref was fast-forwarded to 77e5ae29 while nothing was built or deployed. Amplify auto-build is off, so no app rebuilt, but the pointer now overstates what shipped.

Prod is still serving the 2026-08-18 deploy (35b86e6). Since the affected base now reads from the ref, a normal run would diff from 77e5ae29 and skip targets that changed between 35b86e6 and 77e5ae29.

The first Deploy Production after this lands should use deploy_all=true to resync the ref with what is actually running. Normal affected deploys are safe from then on.

Verification

Workflow-only change; no application code touched.

  • YAML parses, and advance-prod now has three steps with id: previous present, so steps.previous.outputs.sha resolves.
  • No new uses: pins, so the exact-action-version-comment gate is unaffected.
  • End-to-end proof is the next prod deploy — it can only be exercised on workflow_dispatch against staging.

🤖 Generated with Claude Code

… SHA

Deploy Production has failed at "Detect affected targets" on every run that
didn't set deploy_all=true since #2165:

  ❌ Could not read the prod ref, so the affected base is unknown.

#2165 replaced nx-set-shas with a base derived from the SHA prod was actually
serving, wiring `setup`'s PROD_BEFORE to advance-prod's `previous-sha` output:

  previous-sha: ${{ steps.previous.outputs.sha }}

but never added a step with `id: previous`. GitHub resolves an unknown
`steps.<id>` expression to the empty string rather than erroring, so the output
was always "" and the `[ -n "$PROD_BEFORE" ]` guard in `setup` fired every time.
It went unnoticed because the only two prod deploys since #2165 (2026-08-04,
2026-08-18) both ran with deploy_all=true, which returns before that guard.
Run 32409904357 (2026-08-20) was the first affected-detection prod deploy and
failed.

Add the capture step, before the fast-forward — once the ref moves the previous
SHA is unrecoverable.

Also gate release-notes on `setup`. It needs always() so a SKIPPED deploy job
(the normal affected-detection outcome) still cuts the release, but with only
advance-prod required it tagged a release for a run that deployed nothing:
prod-2026-08-20-77e5ae2 exists for the failed run above.

Note for the next release: run 32409904357's advance-prod succeeded, so the prod
ref was fast-forwarded to 77e5ae2 while nothing shipped. Prod is still serving
the 2026-08-18 deploy (35b86e6), so the first Deploy Production after this lands
should use deploy_all=true to resync the ref with what is actually running.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@amazon-inspector-n-virginia

Copy link
Copy Markdown

⏳ I'm reviewing this pull request for security vulnerabilities and code quality issues. I'll provide an update when I'm done

@sajclarke sajclarke added bug Something isn't working severity:important Significant impact, schedule soon area:infra Infrastructure / CI/CD / containers subsystem:ci GitHub Actions / CI/CD pipelines labels Aug 21, 2026
@github-actions

Copy link
Copy Markdown

Amplify PR previews

App Status URL
forms ⏭ Not affected
landing ⏭ Not affected
chat ⏭ Not affected
form-builder ⏭ Not affected
analytics ⏭ Not affected

Commit: 220f56b · Branches auto-delete when this PR closes.

ℹ️ feature-flagging uses a single shared preview slot at a fixed URL (preview.feature-flagging.sandbox.alpha.gov.bb) — GitHub login works, but only one FF preview exists at a time (the most recent FF PR build wins). It reads/writes sandbox flag data.

@amazon-inspector-n-virginia

Copy link
Copy Markdown

✅ I finished the code review, and didn't find any security or code quality issues.

@sajclarke
sajclarke requested a review from LaronGovT August 21, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:infra Infrastructure / CI/CD / containers bug Something isn't working severity:important Significant impact, schedule soon subsystem:ci GitHub Actions / CI/CD pipelines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant