Skip to content

feat(approve): opt-in approve.github_review mirrors approved as a GitHub review - #263

Merged
github-actions[bot] merged 11 commits into
cncf:mainfrom
mfahlandt:feat/approve-github-review
Oct 3, 2026
Merged

github-actions[bot] merged 11 commits into
cncf:mainfrom
mfahlandt:feat/approve-github-review

Conversation

@mfahlandt

Copy link
Copy Markdown
Member

The OpenSSF Scorecard Branch-Protection
check scores in tiers and only credits a tier once the previous one is complete:

Tier Points Requirements (non-admin token)
1 3/10 prevent force push, prevent deletion
2 6/10 require at least 1 reviewer for approval (+ admin-only items)
3 8/10 require at least 1 status check
4 9/10 require 2 reviewers, require code owner review
5 10/10 admin-only: dismiss stale reviews, include administrators

A Prow-gated repository (OWNERS + /lgtm + /approve + required prow/lgtm status) has no
required review count, so it stops in tier 2 with partial credit and tier 3 is never counted:
4/10 (observed on seebom-labs/BOMHort:
Warn: branch 'main' does not require approvers). With one required approving review tiers 2
and 3 are complete: 8/10. Today such a repository only gets there by approving twice: a
/approve comment and a GitHub review.

Disclaimer: Parts of this PR and the PR template have been written with GitHub Copilot Model Claude Opus 5.5

This PR adds a opt-in setting of the approve plugin (repositories with OWNERS files only):

While a pull request carries approved, the token keeps an APPROVE review on the PR's current
head commit. When approved goes away, it dismisses that review. Humans keep using /approve and
/lgtm only. A branch protection rule or ruleset requiring 1 approving review is then met by the
Prow decision.

Situation Action
the evaluation ends with approved one APPROVED review by the token on the current head (commit_id); nothing is written when it already exists. Body: Approved via /approve by bob, carol (OWNERS)., one explanatory line, hidden marker <!-- prow-github-actions/approve-review -->. Approvers are named without @: a fresh review is submitted after every push, and a mention would notify the approvers each time
the evaluation ends without approved (/approve cancel, CHANGES_REQUESTED, files no longer covered, ?) every APPROVED review by the token that carries the marker is dismissed, on any commit, with approved removed: <reason>, ex: approved removed: no approver covers sdk/x.go; withdrawn by bob (/approve cancel)
synchronize (new head) while approved stays a fresh review on the new head; the old one is left alone (GitHub may have dismissed it as stale). Reviews without the marker are never touched
PR author is the token's own identity warning cannot submit the approval review: #<n> was opened by the token's own identity (<login>), and GitHub does not let an author approve their own pull request (approve.github_review), no review. Checked up front for a user token; for an installation token GitHub's 422 Can not approve your own pull request is caught
403, or GitHub Actions is not permitted to approve pull requests. (any status) warning, not failure: cannot submit the approval review: enable "Allow GitHub Actions to create and approve pull requests" (Settings ? Actions ? General) or pass a token that can (approve.github_review): <GitHub's message>
any other API error could not submit the approval review: <error> / could not dismiss the approval review <id>: <error> thrown like the plugin's other errors: collected by the handler runner, which still runs tide, then setFailed (error annotation) at the end
the pull request is closed nothing

Invariants

  1. The bot's review never counts. approvalEvents skips any review carrying the marker
    (always, also with the setting off; nobody has such reviews unless the setting was on) and,
    with the setting on, every review by the token's own user. The login comes from GET /user
    (once per client). GITHUB_TOKEN and App installation tokens get a 403 there, which is
    treated as "installation token": their reviews are by a Bot user, which never counted anyway.
    Consequence, documented: with a maintainer's PAT as token, that maintainer's own reviews stop
    counting; use a machine user.
  2. No loops. With GITHUB_TOKEN the review fires no workflow. With a user token it fires
    pull_request_review; approveOnReview returns early when the payload's review carries the
    marker (also for dismissed). Tide still evaluates on that event, which can only help (the
    review may have unblocked mergeable_state).
  3. Order. compute ? label ? notifier ? review ? tide (approve runs before tide in every
    handler registry and before the post-command sweep).
  4. Idempotent. An own APPROVED review whose commit_id is the head means no write.
  5. Off by default. With github_review unset there is no GET /user, no extra
    listReviews (with ignore_review_state: true none at all, as before), no review write.

Not breaking: nothing changes unless the key is set. The legacy path for repositories without
OWNERS files (src/issueComment/approve.ts) is unchanged.

Required repository setting

With GITHUB_TOKEN: Settings ? Actions ? General ? Workflow permissions ? Allow GitHub Actions
to create and approve pull requests
(GitHub docs: this setting configures "whether
GITHUB_TOKEN can create and approve pull requests"; organizations may force it off). A GitHub
App token or a machine user's PAT does not need it. The workflow permissions are unchanged:
pull-requests: write is already required (docs/installing.md lists it for reviews), so
installing.md is untouched.

What is uncertain about GitHub's behaviour

Written down as uncertain in the docs as well:

  • Status of "not permitted". The REST message GitHub Actions is not permitted to approve pull requests.
    is widely reported (GraphQL: ? (addPullRequestReview)); third-party code reports HTTP 422 for
    REST, GitHub does not document the status. The code matches the message regardless of status,
    and treats any 403 the same way (per the issue). A 403 for another reason (ex: missing
    pull-requests: write) therefore also becomes a warning; GitHub's message is appended so the
    operator sees the real cause.
  • mergeable_state right after the review. Tide runs after the review in the same run. The
    existing unknown retries (1 s, 2 s, 4 s) cover an unknown answer; whether GitHub can instead
    return the previous blocked for a moment is not documented. If it does, the PR is skipped as
    not mergeable (blocked) and the next event or the cron merges it. With a user token the review's
    own pull_request_review event re-evaluates the merge.
  • Dismiss stale reviews vs. the fresh review. On a push GitHub dismisses stale approvals; our
    synchronize run submits a new review on the new head. Whether GitHub's asynchronous dismissal
    could land after, and dismiss, the fresh review is not documented. The next evaluation would
    put it back.
  • "Require approval of the most recent reviewable push": satisfied by the fresh review on
    every head, as long as the token's identity is not the pusher (GitHub docs: "approved by
    someone other than the person who pushed it").
  • Rulesets. That a GITHUB_TOKEN approval counts toward branch protection's required count
    is what the old bot review relied on; that a ruleset's "Required approvals" counts it the same
    way is expected but not verified here.
  • Drafts. Not verified whether GitHub accepts an APPROVE review on a draft PR. If it refuses,
    the evaluation fails with could not submit the approval review: ?.
  • Restricted dismissals. With "Restrict who can dismiss pull request reviews", whether the
    token may dismiss its own review is not documented; a refusal fails the run with
    could not dismiss the approval review <id>: ?.
  • CODEOWNERS. A required code owner review is still not satisfied by github-actions[bot].

Docs

  • docs/configuration.md: the key in the approve example and table.
  • docs/commands.md (approve): "No bot review by default" paragraph, new
    "Mirroring approved as a GitHub review" subsection with the situation table and every message verbatim.
  • docs/automatic-merging.md: step 4 of "Upgrading to aggregated approval" gets the third option;
    new section "Required reviews and OpenSSF Scorecard" (what it is and is not, the repository
    setting, dismiss-stale, most-recent-push, mergeability, the Scorecard tiers stated neutrally:
    the check credits a required review count and does not judge who approves).
  • docs/installing.md: checked, pull-requests: write already listed; unchanged.

Tests

  • __tests__/plugins/approveReview.test.ts (vitest + msw), every table row and invariant:
    flag off ? no GET /user, no review write, nothing dismissed; flag off + ignore_review_state
    ? no listReviews; approved ? one review on headsha after label and notifier; idempotent
    re-run; old-head review + synchronize ? new review on the new head, old not dismissed;
    stale-dismissed own review ? re-submitted; marked review by someone else is not ours; self PR
    (user token) ? warning; 422 self-approval ? warning; 403 ? warning with the exact text;
    "not permitted" 422 ? warning; 500 ? error, label/notifier still written, run fails at the end
    while tide still evaluates; closed PR ? nothing; cancel ? every marked own review dismissed with
    the reason; unmarked bot/human reviews and marked reviews by others are never dismissed;
    CHANGES_REQUESTED reason; dismissal 500 ? error; user token that is an OWNERS approver:
    its marked and unmarked reviews never count; GET /user 500 ? error before any write; 404 ?
    installation token, memoized; pull_request_review for the mirrored review (submitted and
    dismissed) evaluates nothing; order label ? notifier ? review ? merge through handlePullReq.
  • Bundle e2e (dist/index.js): /approve ? approved label + one APPROVE review on the head
    before tide's read; /approve cancel ? label removed, only the marked review dismissed.
  • Existing approve/config tests updated for the new key; validation (approve.github_review must be a boolean).
  • npm run build, npm run lint (0 errors, 2 pre-existing warnings), npm run pack,
    npx vitest run --coverage: 70 files, 1486 tests pass; coverage 99.83 % lines, 98.42 % branches,
    100 % functions (thresholds 95/93/98/95); approveReview.ts 100 %. dist/ rebuilt and committed;
    a rebuild leaves git status clean.

…oken's APPROVE review

While a pull request carries approved, the token keeps one APPROVE review on the head commit; when approved goes away the marked reviews are dismissed with the reason. Off by default: no GET /user, no review read or written. The mirrored review and every review by the token's own user never count toward approved, and its own pull_request_review events evaluate nothing. A refused approval (403, Actions not permitted, self-approval) is a warning.

Signed-off-by: Mario Fahlandt <mfahlandt@pixel-haufen.de>
Signed-off-by: Mario Fahlandt <mfahlandt@pixel-haufen.de>
…penSSF Scorecard

Signed-off-by: Mario Fahlandt <mfahlandt@pixel-haufen.de>
Signed-off-by: Mario Fahlandt <mfahlandt@pixel-haufen.de>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Coverage check on this head (npx vitest run --coverage, Node $(node -v), head of feat/approve-github-review):

file stmts branch funcs lines uncovered
src/plugins/approveReview.ts 100 100 100 100 —
src/plugins/approve.ts 100 97.76 100 100 234, 411, 417, 504
src/utils/config.ts 100 99.03 100 100 596, 605
all files 99.83 98.42 100 99.83

Every line this PR adds is exercised — including the new approvalEvents skips (login === tokenLogin, marker in review.body) and the early return in the review handler. The residual uncovered branches in approve.ts/config.ts predate this PR and are already claimed by held PRs #221 and #229, so nothing further is needed here from a test-coverage standpoint.


🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5q9t | SHA: 64eedf6

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

@jeefy jeefy left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks solid: it's off by default, re-runs are safe, only marked bot reviews get dismissed, and dist/ matches a fresh npm run pack. The docs need to be accurate before this merges:

  1. docs/automatic-merging.md (~L453–464): approved stays on a PR across pushes, so the bot re-approves every new head. That makes "Dismiss stale approvals" and "Require approval of the most recent reviewable push" ineffective. Please say so plainly, and say that the real protection after a push is that lgtm is removed on every push, plus the required prow/lgtm status, which must stay required.
  2. docs/automatic-merging.md (~L444) and the github_review row in docs/configuration.md: "Allow GitHub Actions to create and approve pull requests" is a repo-wide setting. Once it's on, any workflow, including one a write-access collaborator adds on their own branch, can approve PRs with GITHUB_TOKEN. Add a warning and suggest a dedicated App or machine-user token as the safer setup.
  3. Scorecard section (~L473): add one sentence saying the required-review credit is nominal with this setup, because the review mirrors the Prow decision rather than being an independent check.

Nits, not blocking:

  • approveReview.ts:201 turns every 403 into the "enable Allow GitHub Actions…" warning, including a workflow that's just missing pull-requests: write. Only give that advice when the response says "not permitted".
  • A refused dismissal (:170) throws on every evaluation until someone dismisses the review by hand. Document that.
  • Check whether GitHub accepts an approval on a draft PR, or skip review sync when pull.draft is true.

jeefy added 7 commits October 3, 2026 14:58
Signed-off-by: Jeffrey Sica <me@jeefy.dev>
…er msw 3

msw 3 ignores onUnhandledRequest and lets unmocked requests through to the network; use utils.failOnUnhandledRequest like the other suites.

Signed-off-by: Jeffrey Sica <me@jeefy.dev>
…says it is off

A plain 403 on the review (for example a workflow without pull-requests: write) now warns about the missing permission instead of advising to enable "Allow GitHub Actions to create and approve pull requests". Both stay warnings.

Signed-off-by: Jeffrey Sica <me@jeefy.dev>
A draft gets its mirrored review once it is ready for review, so approve now re-evaluates on ready_for_review (before tide's merge evaluation in the same run). Dismissals still apply to drafts.

Signed-off-by: Jeffrey Sica <me@jeefy.dev>
…proval rules

Because approved is sticky, the bot re-approves every new head, so "Dismiss stale approvals" and "Require approval of the most recent reviewable push" are neutralized; the post-push protection is lgtm removal plus the required prow/lgtm status. Warn that "Allow GitHub Actions to create and approve pull requests" is repo-wide and recommend an App or machine-user token, note the Scorecard credit is nominal, and document refused dismissals and drafts.

Signed-off-by: Jeffrey Sica <me@jeefy.dev>
…used dismissals

Signed-off-by: Jeffrey Sica <me@jeefy.dev>
Signed-off-by: Jeffrey Sica <me@jeefy.dev>
@jeefy

jeefy commented Oct 3, 2026

Copy link
Copy Markdown
Member

@mfahlandt I pushed a few commits on top of your branch to cover the review. Your commits are unchanged and nothing was force-pushed.

  • Merged main (8341a95). It merged without conflicts, and dist/ is regenerated rather than hand-merged.
  • Stale-approval rules (docs/automatic-merging.md): the docs now say plainly that approved is sticky, so the bot re-approves every new head. That makes "Dismiss stale approvals" and "Require approval of the most recent reviewable push" ineffective. The real protection after a push is lgtm removal plus the prow/lgtm status, which must stay a required check.
  • Repo-wide Actions setting: there's now a warning in the "Repository setting" section and in the github_review row of docs/configuration.md. Once "Allow GitHub Actions to create and approve pull requests" is on, any workflow can approve with GITHUB_TOKEN. The docs recommend a dedicated App or machine-user token instead.
  • Scorecard: added a sentence saying the required-review credit is nominal, because the review mirrors the Prow decision and isn't an independent control.
  • 403 handling (approveReview.ts): you only get the "enable Allow GitHub Actions…" advice when GitHub says "not permitted to approve pull requests". Any other 403 now warns that pull-requests: write is missing. Both are still warnings, and there are tests for each.
  • Refused dismissals: documented that one (e.g. "Restrict who can dismiss") fails every evaluation until someone dismisses the review by hand.
  • Drafts: a draft gets no mirrored review, but dismissals still apply. Approve now also re-evaluates on ready_for_review (it runs before tide in the same run), so the review shows up as soon as the PR leaves draft. Tests and docs are updated (commands.md, events.md).
  • msw 3: approveReview.test.ts now uses utils.failOnUnhandledRequest, because msw 3 ignores onUnhandledRequest and unmocked requests would hit the network.

@jeefy jeefy left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All review asks addressed. Thanks @mfahlandt!

@jeefy

jeefy commented Oct 3, 2026

Copy link
Copy Markdown
Member

/kind cleanup
/lgtm
/approve

@github-actions github-actions Bot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants