Automation workflows must validate and format what they generate before committing

[mrbobbytables]

Umbrella issue consolidating four findings about automation workflows committing or publishing generated data without running the validators and formatters that guard it:

• [quality] Import job never runs check:format, so every architecture import PR carries 16 unformatted files #713 — the architecture import job never runs check:format, so every import PR carries unformatted files
• [sec-check] refresh-community-people.yml commits third-party profile data without running validate:community-people #736 — refresh-community-people.yml commits third-party profile data without running validate:community-people
• [quality] automation workflows run test:unit before regenerating the data it guards #738 — automation workflows run test:unit before regenerating the data it guards, so the tests never see the new data
• [sec-check] deploy-gh-pages.yml publishes after running only 4 of 10 data validators #740 — deploy-gh-pages.yml publishes after running only 4 of 10 data validators

Fix direction: a shared "regenerate, then format, then validate everything, then test" ordering in each automation workflow, and running the full validator set in the deploy gate.

[hivecommons-hive]

Exact replacement text for all four workflows

Verified against main@26946ba. This is the complete fix for all four findings consolidated here. It cannot be opened as a pull request by this agent: every change lives under .github/workflows/**, and the token tier this agent runs at does not carry the Workflows permission, so GitHub rejects the push server-side regardless of installation grants. Applying the diff below needs a human or an ISSUES_PRS_MERGE agent. It is given as a patch so applying it is mechanical rather than a rewrite.

The regression guard that locks this in is pushable and is open as #758.

The detail that makes all four findings silent

ci.yml already runs all eleven validate:* gates plus check:format. That is why none of this shows up as a red check: the automation pull requests never reach ci.yml. They are opened by peter-evans/create-pull-request under github.token, and GitHub does not start on: pull_request runs for a token-authored pull request. On the open import PR #681:

$ gh pr view 681 --repo cncf/endusers --json statusCheckRollup,files
files=19   checks=1   ->  DCO   SUCCESS

Nineteen changed files, one check, and it is not ci.yml. So the steps each generating workflow runs itself are the only pre-merge gate its output ever passes, and deploy-gh-pages.yml is the only post-merge one. Both are currently incomplete, which is exactly the four findings above.

This also means #740 is not redundant with ci.yml. The deploy job is a real last line of defence, not a duplicate.

Reproduction of #713

$ git diff --name-only main...automation/import-architectures | tee /tmp/changed.txt | wc -l
19

$ git checkout automation/import-architectures
$ npx prettier --check $(cat /tmp/changed.txt)
[warn] data/architectures/catalog.json
[warn] data/members.json
[warn] docs/architectures/adobe.md
... 13 more
[warn] Code style issues found in 16 files. Run Prettier with --write to fix.

$ git checkout main
$ npx prettier --check $(cat /tmp/changed.txt)
All matched files use Prettier code style!

Clean on main, dirty on the branch: the import job is the source of the drift, and nothing downstream reads it.

The patch

Ordering applied throughout is regenerate → format → validate → test → build. fix:format precedes check:format so the job repairs its own output rather than merely failing on it; check:format then proves the committed tree is clean. test:unit moves last so the suite sees the data that is about to be committed rather than the previous run's.

git apply-able patch for all four workflows

diff --git a/.github/workflows/deploy-gh-pages.yml b/.github/workflows/deploy-gh-pages.yml
index 8bcacb7..14feb29 100644
--- a/.github/workflows/deploy-gh-pages.yml
+++ b/.github/workflows/deploy-gh-pages.yml
@@ -51,6 +51,30 @@ jobs:
       - name: Validate awards data
         run: npm run validate:awards
 
+      # Automation pull requests are authored by github.token, which does not
+      # start ci.yml, so this job is the only gate their data ever passes.
+      # It therefore has to run the full validator set, not a subset.
+      - name: Validate community people data
+        run: npm run validate:community-people
+
+      - name: Validate community group links
+        run: npm run validate:community-groups
+
+      - name: Validate launch metrics data
+        run: npm run validate:launch-metrics
+
+      - name: Validate case studies data
+        run: npm run validate:case-studies
+
+      - name: Validate radar reports data
+        run: npm run validate:radar-reports
+
+      - name: Validate projects-born data
+        run: npm run validate:projects-born
+
+      - name: Validate button contrast
+        run: npm run validate:button-contrast
+
       # Build for the GitHub Pages project URL. When the custom domain in
       # static/CNAME is verified and active, update these values to
       # SITE_URL=https://endusers.cncf.io and BASE_URL=/ so canonical and
diff --git a/.github/workflows/import-architectures.yml b/.github/workflows/import-architectures.yml
index 36ba9a1..6930682 100644
--- a/.github/workflows/import-architectures.yml
+++ b/.github/workflows/import-architectures.yml
@@ -25,15 +25,24 @@ jobs:
           cache: npm
       - run: sudo apt-get update && sudo apt-get install -y librsvg2-bin
       - run: npm ci
-      - run: npm run test:unit
+      # Regenerate first, so everything below inspects the new data.
       - run: npm run collect:metrics
         env:
           GH_TOKEN: ${{ github.token }}
-      - run: npm run validate:metrics
       - run: npm run import:architectures
+      # Format what was just generated, then prove it is formatted. Without
+      # this the pull request carries files `check:format` would reject, and
+      # nothing rechecks it: a token-authored pull request does not start
+      # ci.yml.
+      - run: npm run fix:format
+      - run: npm run check:format
+      - run: npm run validate:metrics
       - run: npm run validate:architectures
       - run: npm run validate:architecture-assets
       - run: npm run validate:awards
+      # Last, so the suite sees the regenerated data rather than the previous
+      # run's.
+      - run: npm run test:unit
       - run: npm run build
       - uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
         with:
diff --git a/.github/workflows/refresh-community-people.yml b/.github/workflows/refresh-community-people.yml
index bd3ed32..b724340 100644
--- a/.github/workflows/refresh-community-people.yml
+++ b/.github/workflows/refresh-community-people.yml
@@ -25,6 +25,12 @@ jobs:
           cache: npm
       - run: npm ci
       - run: npm run fetch:community-people
+      - run: npm run fix:format
+      - run: npm run check:format
+      # This job commits third-party profile data. Its validator is the only
+      # thing that inspects it before it reaches main.
+      - run: npm run validate:community-people
+      - run: npm run test:unit
       - run: npm run build
       - uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
         with:
diff --git a/.github/workflows/refresh-radar-reports.yml b/.github/workflows/refresh-radar-reports.yml
index ca46d6b..a92e4b7 100644
--- a/.github/workflows/refresh-radar-reports.yml
+++ b/.github/workflows/refresh-radar-reports.yml
@@ -24,9 +24,13 @@ jobs:
           node-version: ${{ env.NODE_VERSION }}
           cache: npm
       - run: npm ci
-      - run: npm run test:unit
       - run: npm run collect:radar-reports
+      - run: npm run fix:format
+      - run: npm run check:format
       - run: npm run validate:radar-reports
+      # Last, so the suite sees the regenerated data rather than the previous
+      # run's.
+      - run: npm run test:unit
       - run: npm run build
       - uses: peter-evans/create-pull-request@22a9089034f40e5a961c8808d113e2c98fb63676 # v7.0.11
         with:

Verification performed

All of the following was run on main@26946ba with the patch applied:

• all eleven validators pass offline — so adding the seven missing ones to the deploy gate does not break deploys:
  metrics, architectures, architecture-assets, awards, button-contrast, launch-metrics, case-studies, radar-reports, community-people, community-groups, projects-born → all OK
• npx prettier --check .github/workflows/*.yml → clean
• node --test tests/workflow-scripts.test.mjs tests/ci-supply-chain.test.mjs → 26 pass, 0 fail (the existing workflow and supply-chain contracts still hold, and no action pin is disturbed)
• npm run test:unit:coverage:check → 1344 tests, 0 failures, coverage gate exit 0
• the guard in test: guard regenerate-format-validate-test ordering for data-generating automation jobs #758, run against the patched workflows with its exemption table emptied → all 8 rules green, i.e. the patch resolves all seven recorded violations and nothing else is left outstanding

How the two pieces fit

#758 records today's seven violations in a KNOWN_ORDERING_VIOLATIONS table citing this issue, following the GATES_NOT_RUN_BY_CI convention already used in tests/workflow-scripts.test.mjs. It therefore lands green today, while binding any new automation job to the contract immediately.

A staleness guard fails if a table entry outlives the fault it describes. So as each workflow above is corrected, its entries must be deleted in the same change — the table cannot rot, and a later regression that reintroduces a fixed fault fails CI instead of being excused. Applying the whole patch at once empties the table entirely.

Suggested order: land #758 first (green, no behaviour change), then this patch with the table emptied in the same commit.

---

Filed by quality agent (hold-gated mode).

🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: unknown

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88