Skip to content

External review coverage is inversely correlated with change size, and only the author's own review closes the gap #529

Description

@blooop

The observation

Sourcery refuses to review a pull request over 150,000 diff characters, and posts that refusal as a review:

Sorry @blooop, your pull request is larger than the review limit of 150,000 diff characters

scripts/review_verdict.sh classifies that correctly — it asks the reviewer whether it refused, rather than sniffing the prose, and a refusal is not a review. So the review check goes red and the PR cannot merge unreviewed. The guard is working. This issue is not a bug report against it.

The observation is about what the guard's red means in that case, and about the shape of the coverage underneath it.

Why it is worth recording

The external reviewer withdraws exactly where the change is largest. Two refusal modes, both already named in test_review_guard.py:

  • size — fires on the biggest diffs, which are the ones least safe to merge unread;
  • weekly quota — fires for a week at a time, and did: twenty-six consecutive pull requests merged with nothing reviewing them between 2026-08-22 and 2026-08-24, invisible for a day and a half because the refusal was posted as a review.

So external review coverage is not uniform. It is inversely correlated with the size of the change, and it disappears entirely for a week at a time on a schedule nobody controls.

Live instance, 2026-08-29: #525, 18 files, +5076/-318, the keystone of map #444. Sourcery refused on size. The only thing that reviewed it was a fresh-context agent review, which found two blocking defects — the sweep refusing to collect the gitignored build output the ticket exists to reclaim, and a plan/act divergence removing a nested worktree the plan never named. Neither would have been caught by anything else in the pipeline.

What already answers it

The wf-review route: an author review carrying the provenance line, produced in fresh context by an agent that did not write the code. That is what review_verdict.sh accepts from the author, and it is deliberately not "lgtm".

That route worked on #525. This issue is not asking for a mechanism; it is asking whether the reliance on it should be explicit rather than incidental, given it is load-bearing precisely when the change is biggest.

Questions this is filed to decide

  1. Should a PR over the size limit be required to carry an agent review, rather than that happening because someone noticed? Today it is enforced only in the sense that review stays red until something reviews it — which is arguably already the right enforcement.
  2. Should the size refusal be reported differently from the quota refusal? They mean different things: one says "this change is too big for me", which is information about the change; the other says "I am out of budget", which is information about the reviewer. Today both land as "nothing reviewed this".
  3. Is a diff over 150,000 characters a signal to split the PR, and should the check say so? Rebuild the agent-worktree sweep on the site subtree #525 supersedes Reclaim the agent git worktrees stranded inside live workspace clones #442 and is a rebuild rather than an increment, so it may be irreducible — but the question has never been asked out loud.

Related

All three are the same species: a signal whose green means less than a reader assumes. Here the signal is honest and goes red; what is easy to miss is that the red is routine on large changes and is cleared by a review of a different kind from the one it was asking for.

Found by two sessions merging in parallel and comparing notes, 2026-08-29.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions