The defect
scripts/review_verdict.sh recognises exactly two Sourcery refusal wordings:
body_is_refusal() {
case $1 in
*"you have reached your weekly rate limit"*) return 0 ;;
*"larger than the review limit"*) return 0 ;;
esac
return 1
}
Sourcery has a third, currently in effect on this account:
Sorry @blooop, you've used your own review budget of 250,000 diff characters for the last 7 days.
You can request another review in 6 days and 17 hours by commenting @sourcery-ai review.
It matches neither pattern. is_refusal therefore returns false, the refusal is counted as a review by a second party, and the review check goes green on a pull request that nothing read.
This is the exact failure the guard was built for. From test_review_guard.py: between 2026-08-22 and 2026-08-24, twenty-six consecutive pull requests merged with no review at all, because a quota refusal was posted as a review and a reader looking for one found one.
Audit: it is not hypothetical
Every pull request merged on 2026-08-29, classified by the Sourcery review body actually posted:
| PR |
Sourcery's body |
| #512, #513, #515 |
real review |
| #504, #505, #506, #510 |
real review |
| #525, #532 |
known refusal (size) — correctly caught, review went red |
| #514, #516, #519, #520, #521, #522, #523, #524, #528, #534 |
unrecognised refusal, counted as a review |
Ten pull requests merged with the review check green for a reason that was not a review.
What covered them was that the practice held where the check did not: most carried an independent agent review posted separately, and several had blocking findings fixed before merge. But that was discipline, not enforcement — the guard was silently not guarding, and on any PR where nobody happened to look, nothing would have said so.
Why enumerating refusals cannot work
The set of refusal wordings belongs to a third party and grows without notice. It has now grown twice: the file's own comments record being wrong about it in both directions already, once matching too loosely and once anchored too tightly.
The fix is to invert the question: enumerate what counts as a review, not what counts as a refusal.
Sourcery's genuine reviews have a small, stable positive form it controls (Hey - I've reviewed your changes and they look great!, Hey - I've found N issues, and the individual-comments shape). Those are a closed set the bot emits deliberately. Refusals are an open set it emits under conditions outside anyone's control.
So a body from a REFUSING_LOGINS author should count as a review only if it matches a recognised review form, with anything unrecognised treated as not-a-review. The failure direction then becomes "a new review format stops counting", which is loud, safe, and fixed by adding a pattern — instead of "a new refusal format starts counting", which is silent and merges unread code.
This is the same lesson as #517 and #529: a check that enumerates the bad cases passes on the case it has not heard of yet. Enumerate the required thing.
Notes for whoever builds it
test_review_guard.py already holds the two known refusals verbatim and the two known clean-review forms. The new wording above should join them, and the test should assert that an unrecognised body from a refusing login does not count as a review — that assertion is the actual guard, and it is the one that would have caught this.
- Keep the
REFUSING_LOGINS word-splitting fix and its [bot]-globbing rationale intact.
- Consider whether
state is usable as a discriminator (all of these are COMMENTED), and record the answer either way so the next person does not re-investigate.
Related: #517 (a merge verdict pinned to a stale or partial check set), #529 (external review coverage is inversely correlated with change size), #527. All four are one shape: a check whose green ranges over less than the claim it appears to support.
The defect
scripts/review_verdict.shrecognises exactly two Sourcery refusal wordings:Sourcery has a third, currently in effect on this account:
It matches neither pattern.
is_refusaltherefore returns false, the refusal is counted as a review by a second party, and thereviewcheck goes green on a pull request that nothing read.This is the exact failure the guard was built for. From
test_review_guard.py: between 2026-08-22 and 2026-08-24, twenty-six consecutive pull requests merged with no review at all, because a quota refusal was posted as a review and a reader looking for one found one.Audit: it is not hypothetical
Every pull request merged on 2026-08-29, classified by the Sourcery review body actually posted:
reviewwent redTen pull requests merged with the
reviewcheck green for a reason that was not a review.What covered them was that the practice held where the check did not: most carried an independent agent review posted separately, and several had blocking findings fixed before merge. But that was discipline, not enforcement — the guard was silently not guarding, and on any PR where nobody happened to look, nothing would have said so.
Why enumerating refusals cannot work
The set of refusal wordings belongs to a third party and grows without notice. It has now grown twice: the file's own comments record being wrong about it in both directions already, once matching too loosely and once anchored too tightly.
The fix is to invert the question: enumerate what counts as a review, not what counts as a refusal.
Sourcery's genuine reviews have a small, stable positive form it controls (
Hey - I've reviewed your changes and they look great!,Hey - I've found N issues, and the individual-comments shape). Those are a closed set the bot emits deliberately. Refusals are an open set it emits under conditions outside anyone's control.So a body from a
REFUSING_LOGINSauthor should count as a review only if it matches a recognised review form, with anything unrecognised treated as not-a-review. The failure direction then becomes "a new review format stops counting", which is loud, safe, and fixed by adding a pattern — instead of "a new refusal format starts counting", which is silent and merges unread code.This is the same lesson as #517 and #529: a check that enumerates the bad cases passes on the case it has not heard of yet. Enumerate the required thing.
Notes for whoever builds it
test_review_guard.pyalready holds the two known refusals verbatim and the two known clean-review forms. The new wording above should join them, and the test should assert that an unrecognised body from a refusing login does not count as a review — that assertion is the actual guard, and it is the one that would have caught this.REFUSING_LOGINSword-splitting fix and its[bot]-globbing rationale intact.stateis usable as a discriminator (all of these areCOMMENTED), and record the answer either way so the next person does not re-investigate.Related: #517 (a merge verdict pinned to a stale or partial check set), #529 (external review coverage is inversely correlated with change size), #527. All four are one shape: a check whose green ranges over less than the claim it appears to support.