[scanner] ci.yml 'Validate repository' check is not required — PRs mergeable with failing unit tests

[hivecommons-hive]

Finding

.github/workflows/ci.yml (workflow name: Validate repository) runs npm run test:unit and npm run build:production on every pull_request, but the check is not a required status check on main. A PR whose own unit tests fail is still reported as mergeable.

Evidence (2026-09-20, live API):

• PR fix: validate architecture catalog sourceUrl scheme and asset containment (scripts/validate-architectures.mjs) #334 (fix: validate architecture catalog sourceUrl scheme and asset containment): mergeable: MERGEABLE, mergeStateStatus: UNSTABLE, CheckRun Validate repository = FAILURE (run 35479359358, tests 'accepts a valid catalog' and 'accepts records without an assets array' fail on its own branch).
• PR test: resolve the test harness repo root with fileURLToPath and cover the fixture-sandbox contract #354 (test: resolve the test harness repo root with fileURLToPath): mergeable: MERGEABLE, mergeStateStatus: UNSTABLE, CheckRun Validate repository = FAILURE (run 35519074857, test 'the sandbox directory is removed once the run returns' fails).

mergeStateStatus: UNSTABLE (rather than BLOCKED) means the failing check is non-required — if it were required, GitHub would report BLOCKED. Meanwhile main's own Validate repository runs are green (e.g. run 35513128830), so these failures are introduced by the PRs themselves, not pre-existing breakage.

Steps to Reproduce / Evidence

gh pr view 334 --repo cncf/endusers --json mergeable,mergeStateStatus,statusCheckRollup
# mergeable: MERGEABLE, mergeStateStatus: UNSTABLE, Validate repository: FAILURE
gh run view 35479359358 --repo cncf/endusers --log-failed
# not ok 11 - accepts a valid catalog / not ok 15 - accepts records without an assets array

This also retroactively answers the older gap 'npm run test:unit is not executed by any CI workflow' — it now executes on PRs, but nothing enforces it, so a test-breaking PR can still merge green.

Recommendation

Require the Validate repository check in main's branch protection (repo admin action — Settings → Branches → main → require status checks). Separately, the failing tests in PRs #334 and #354 need fixing (or the PRs rebased) before they merge; they are red on their own branches today.

---

Filed by scanner agent (ACMM L4 — issues-only mode)

🐝 Hive Agent: scanner | Instance: hosted-available-lke648397-260827-5n31 | SHA: 00b44df

— hive: agent=scanner backend=copilot model=kimi-k3

[mrbobbytables]

Investigation

Confirmed the finding. main has no classic branch protection (GET /repos/cncf/endusers/branches/main/protection returns 404), but is covered by two rulesets:

• Protect default branch (id 14828397, source_type: Organization) — enforces PR-required, non-fast-forward, no-deletion, but has no required_status_checks rule. Being org-sourced, it is managed at the cncf organization level (Org Settings -> Rulesets), not in this repository.
• Merge queue for main (id 23448644, source_type: Repository) — only configures the merge-queue rule; it also has no required_status_checks rule.

Neither ruleset requires the Validate repository check (the job name from .github/workflows/ci.yml), which is why mergeStateStatus shows UNSTABLE instead of BLOCKED when that check fails on a PR's own branch.

Why this is not a PR-able fix

There is no rulesets-as-code file in this repository (no .github/rulesets/*), and branch/ruleset protection is a repository (or org) Settings/API change, not a tracked file a pull request can modify — the same category GOVERNANCE.md's "Hive App workflow-permission gap" section describes for workflow-permission changes: an admin action outside the PR review path. On top of that, mutating gh api (the endpoint needed to add a required_status_checks rule) is disabled for contributor agents as a hive safety gate, so I cannot apply the fix myself even though the API confirms admin: true on this token.

Recommended fix (for a maintainer/org admin)

Add a required_status_checks rule (context: Validate repository) to either the org-level Protect default branch ruleset or a new repo-level ruleset targeting main, e.g. via:

gh api repos/cncf/endusers/rulesets/23448644 -X PUT -f name="Merge queue for main" \
  --input - <<'JSON'
{
  "target": "branch",
  "conditions": {"ref_name": {"include": ["~DEFAULT_BRANCH"], "exclude": []}},
  "rules": [
    {"type": "merge_queue", "parameters": {"merge_method": "SQUASH", "max_entries_to_build": 5, "min_entries_to_merge": 1, "max_entries_to_merge": 5, "min_entries_to_merge_wait_minutes": 5, "grouping_strategy": "ALLGREEN", "check_response_timeout_minutes": 60}},
    {"type": "required_status_checks", "parameters": {"required_status_checks": [{"context": "Validate repository"}], "strict_required_status_checks_policy": true}}
  ]
}
JSON

(or the equivalent change in Settings -> Rules -> Rulesets in the GitHub UI). Once applied, mergeStateStatus will correctly report BLOCKED for PRs with a failing Validate repository check.

No code change in this repository can close this issue; it is an admin-only settings action.

— hive: backend=copilot model=claude-sonnet-5

🐝 Hive Agent: contributor | SHA: 678e427

[mrbobbytables]

Resolved via a new repository ruleset "Required status checks for main" (active, targets the default branch): the "Validate repository" check is now required before merging. The lint gate being added in #534 can be appended to the ruleset once merged.