Upstream sync has been dead since 2026-07-03 — Actions cannot create PRs

[jobordu]

Sync upstream has failed 20+ consecutive daily runs, oldest visible 2026-07-03 (that is the API page size, not the true start). Nothing surfaced it because a scheduled workflow gates nothing — it is a red that nobody sees.

Root cause (VERIFIED — run 29561935235, 2026-07-17)

pull request create failed: GraphQL: GitHub Actions is not permitted to create or
approve pull requests (createPullRequest)

The workflow declares permissions: pull-requests: write, which is necessary but not sufficient — "Allow GitHub Actions to create and approve pull requests" is disabled for this org/repo, and that toggle overrides the workflow-level grant.

repo actions/permissions/workflow -> {"can_approve_pr_reviews": null, "default_workflow_permissions": "read"}
org  actions/permissions/workflow -> {"can_approve_pr_reviews": null, "default_workflow_permissions": "read"}

The cascade this produced

1. Push lands → branch created.
2. gh pr create fails → no PR.
3. set -euo pipefail → step fails → job red.
4. Next day: the branch exists, no open PR exists, so the pre-existing guard says "proceed" → the push is now rejected non-fast-forward → red for a different, more confusing reason that hides the original.

Result: five orphan branches, zero PRs ever created for any of them.

sync/upstream-3.1.0   created 2026-05-23   PRs ever: 0
sync/upstream-3.1.1                        PRs ever: 0
sync/upstream-3.2.0                        PRs ever: 0
sync/upstream-3.2.1   created 2026-06-27   PRs ever: 0
sync/upstream-3.3.0   created 2026-07-17   PRs ever: 0

Upstream drift has therefore been unreviewed since 3.1.0 (2026-05-23) — roughly two months, across four releases.

What PR #10 does and does not do

Does: fixes the $?-after-a-pipe bug that made two of the three outcome paths unreachable (separate and worse — see the PR), and replaces the cryptic ! [rejected] ... (fetch first) with an up-front error naming the cause and the remedy.

Does not: fix this. After #10 the workflow will still fail daily — but legibly, pointing here.

Decisions needed (operator)

1. Enable "Allow GitHub Actions to create and approve pull requests" (Org → Settings → Actions → General), or switch the workflow to a PAT / GitHub App token with pull-requests: write. The org toggle is the smaller change but is org-wide; a scoped App token is tighter. I have not flipped it — loosening an org-wide security setting is not mine to make unilaterally.
2. Triage the five orphan branches. Each holds a real pin bump. Either open a PR per branch:

   gh pr create --head sync/upstream-3.3.0 --title 'sync: bump upstream to 3.3.0' --fill

   or delete the stale ones:

   git push origin --delete sync/upstream-3.1.0   # etc.

   PR chore(ci): pin action refs to verified SHAs #10 deliberately does not automate this — the branches may hold the only copy of a bump someone wants, and the workflow has no mandate to destroy refs.
3. Consider making a dead scheduled workflow visible. This failed ~20 times with no signal. A weekly "is the sync alive?" assertion, or an alert on N consecutive scheduled failures, would have caught it in days rather than three weeks.

Also observed

Run 29561935235's pytest died at collection with ModuleNotFoundError: No module named 'edge_python_sdk._generated' — the patch/generate step had rm -rf'd it and regeneration did not restore it. Whether that is a real breakage or an artifact of the bump is unknown, because the test result was being discarded by the pipe bug. Worth re-checking once #10 makes test failures visible.

[jobordu]

Resolved on the code side (PR #14, merged 8cd9042). Two operator secrets remain — deliberately scoped, not the org-wide flip.

Owner chose the least-privilege path over flipping the org setting: a GitHub App scoped to this repo alone (pull-requests: write + contents: write), rather than granting create-AND-self-approve to every workflow in every org repo.

Done autonomously:

• Wired the workflow to mint a scoped App token (actions/create-github-app-token, SHA-pinned) and use it for checkout + all four PR/push operations.
• Added a Preflight guard that fails fast with a runbook link if the secrets are unset — verified live on run 29988240814: it now fails at step 1 with Sync App not configured … docs/runbooks/sync-upstream-app.md, instead of dying 8 steps later on the create-PR error.
• Deleted the 5 stale orphan sync/upstream-* branches (SHAs recorded for recovery). They predated the fix(sync): two of the three outcome paths were unreachable, and the third lied #12 hardening and would have reverted it if merged; the 3.1.x ones carried the pre-rename edge_provider_sdk package name.

Remaining — one-time, yours (runbook: docs/runbooks/sync-upstream-app.md):

1. Create a GitHub App owned by Digital-Frontier-LDA, permissions Pull requests: RW + Contents: RW, no webhook.
2. Install it on edge-python-sdk only.
3. Set two repo secrets:

   gh secret set SYNC_APP_ID --repo Digital-Frontier-LDA/edge-python-sdk --body '<app id>'
   gh secret set SYNC_APP_PRIVATE_KEY --repo Digital-Frontier-LDA/edge-python-sdk < <the .pem>
   

4. gh workflow run "Sync upstream" to confirm green.

The org-wide policy (this issue's original ask) stays locked — that was the point of the App approach.