Skip to content

A third Sourcery refusal wording is counted as a genuine review; ten PRs merged tonight on it #538

Description

@blooop

The defect

scripts/review_verdict.sh recognises exactly two Sourcery refusal wordings:

body_is_refusal() {
  case $1 in
    *"you have reached your weekly rate limit"*) return 0 ;;
    *"larger than the review limit"*) return 0 ;;
  esac
  return 1
}

Sourcery has a third, currently in effect on this account:

Sorry @blooop, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 6 days and 17 hours by commenting @sourcery-ai review.

It matches neither pattern. is_refusal therefore returns false, the refusal is counted as a review by a second party, and the review check goes green on a pull request that nothing read.

This is the exact failure the guard was built for. From test_review_guard.py: between 2026-08-22 and 2026-08-24, twenty-six consecutive pull requests merged with no review at all, because a quota refusal was posted as a review and a reader looking for one found one.

Audit: it is not hypothetical

Every pull request merged on 2026-08-29, classified by the Sourcery review body actually posted:

PR Sourcery's body
#512, #513, #515 real review
#504, #505, #506, #510 real review
#525, #532 known refusal (size) — correctly caught, review went red
#514, #516, #519, #520, #521, #522, #523, #524, #528, #534 unrecognised refusal, counted as a review

Ten pull requests merged with the review check green for a reason that was not a review.

What covered them was that the practice held where the check did not: most carried an independent agent review posted separately, and several had blocking findings fixed before merge. But that was discipline, not enforcement — the guard was silently not guarding, and on any PR where nobody happened to look, nothing would have said so.

Why enumerating refusals cannot work

The set of refusal wordings belongs to a third party and grows without notice. It has now grown twice: the file's own comments record being wrong about it in both directions already, once matching too loosely and once anchored too tightly.

The fix is to invert the question: enumerate what counts as a review, not what counts as a refusal.

Sourcery's genuine reviews have a small, stable positive form it controls (Hey - I've reviewed your changes and they look great!, Hey - I've found N issues, and the individual-comments shape). Those are a closed set the bot emits deliberately. Refusals are an open set it emits under conditions outside anyone's control.

So a body from a REFUSING_LOGINS author should count as a review only if it matches a recognised review form, with anything unrecognised treated as not-a-review. The failure direction then becomes "a new review format stops counting", which is loud, safe, and fixed by adding a pattern — instead of "a new refusal format starts counting", which is silent and merges unread code.

This is the same lesson as #517 and #529: a check that enumerates the bad cases passes on the case it has not heard of yet. Enumerate the required thing.

Notes for whoever builds it

  • test_review_guard.py already holds the two known refusals verbatim and the two known clean-review forms. The new wording above should join them, and the test should assert that an unrecognised body from a refusing login does not count as a review — that assertion is the actual guard, and it is the one that would have caught this.
  • Keep the REFUSING_LOGINS word-splitting fix and its [bot]-globbing rationale intact.
  • Consider whether state is usable as a discriminator (all of these are COMMENTED), and record the answer either way so the next person does not re-investigate.

Related: #517 (a merge verdict pinned to a stale or partial check set), #529 (external review coverage is inversely correlated with change size), #527. All four are one shape: a check whose green ranges over less than the claim it appears to support.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions