fix(server): coalesce same-SHA webhooks and guard terminal CI (pr) status (RIG-1170) - #34
fix(server): coalesce same-SHA webhooks and guard terminal CI (pr) status (RIG-1170)#34rigel-mintaka wants to merge 3 commits into
Conversation
…atus (RIG-1170)
One invariant, three independent mechanisms that violate it: **a non-terminal (`pending`) status POST must never overwrite a terminal status already reported for the same commit+context.** GitHub commit-status is last-write-wins per context, so a late `pending` strands the required `CI (pr)` check forever and blocks the merge.
## Ingest dedup window (RIG-1170, `server/api/hook.go` + `hook_dedup.go`)
A `gt submit` force-push makes GitHub emit **two** `pull_request` deliveries ~1s apart for one head SHA (distinct delivery GUIDs, both `EventPull`). Each spawned a pipeline; the pair then mutually cancelled and raced their status posts, and the losing handler's creation-time `pending` landed last.
`PostHook` now consults a mutex-guarded in-memory TTL map keyed `(repo.ID, refspec, head SHA)` before creating anything. A hit drops the duplicate with HTTP 200 + a debug log; a miss records the key and proceeds. Gated on a new `WOODPECKER_HOOK_DEDUP_WINDOW` duration flag, **default 0 (off)** per the fork-flag convention.
Scoped so it cannot over-dedup: pull-request pipelines only, so the push path and `cancelPreviousPipelines` are untouched; the SHA in the key means two distinct head SHAs still create and supersede as before; the refspec means two PRs sharing a head SHA never coalesce.
**Reopen carve-out — the design record's mechanism does not exist in current code.** The record says to read the action off the parsed pipeline's `EventReason`. Verified against the GitHub driver: `opened`/`reopened`/`synchronize` all collapse to `EventPull` with an **empty** `EventReason` (only metadata actions populate a reason), so a reopen is byte-identical to an ordinary push delivery at this seam — the same indistinguishability that makes the wedge reproduce. The carve-out therefore keys off the delivery that necessarily *precedes* a reopen: a PR can only be reopened if it was closed, and the close arrives as a separate `EventPullClosed` on the same refspec, which purges the window. Needs no parser change, works on every forge, and fails safe (a stray purge costs one non-coalesced duplicate, never a missing pipeline).
## Stale-pending guard (RIG-1170, `server/pipeline/helper.go` + `server/model/const.go`)
Defense in depth for when a duplicate slips the window. The shared poster `updatePipelineStatus` now re-reads the pipeline by ID when the about-to-post status is non-terminal, and skips the shared-context (aggregate + meta) POST if the stored status has since gone terminal. One site covers `start.go`, `cancel.go`, `decline.go` and `restart.go`.
Deliberately narrow: skip-only (never invents or upgrades a status, never gates a terminal post, a genuinely-pending pipeline still posts); per-workflow statuses are not gated (workflow-scoped contexts are not the wedged check); fail-open on a store error, because dropping a report is worse than a redundant one. A re-read narrows but cannot close the TOCTOU window — which is why the dedup window is the load-bearing half.
New `StatusValue.IsTerminal()` predicate in `server/model`, partitioned to match `convertStatus` exactly (`convertStatus` itself untouched). Note `StatusSkipped` is **non**-terminal here: it reports as GitHub `pending`, so calling it terminal would let a pending-mapped write bypass the guard.
## Terminal aggregate on the `Done` guard-hit (RIG-1129, `server/rpc/rpc.go`)
An agent-fault double-finish hit `checkWorkflowState` and returned before the only forge report on the `Done` path, so a terminal pipeline never resolved its check. The guard-hit now loads the workflow tree and posts the terminal aggregate via the same async poster the clean path uses (`s.reportForgeStatusAsync`, aggregate-only), then still returns the rejection — the fix adds a missing *report*, never a state *mutation*.
The record's cited call (`updatePipelineStatus(ctx, forge, currentPipeline, repo, user)`) is stale: that helper now takes a `_store`, and neither `_forge` nor `user` is in scope at the guard-hit. Matched the current clean-path mechanism instead.
Terminality is established by **two** signals, both required. `IsThereRunningStage` alone is insufficient: it asks whether any workflow is pending-or-running, and a **blocked** workflow — the other state this same guard rejects — is neither, so a pipeline merely awaiting approval would look terminal and get a premature verdict posted. Checking the pipeline's own stored status first also means the blocked and still-running paths do no store read at all.
## Red-green evidence
Every new test was run against the pre-fix code and observed failing:
- `TestPostHookCoalescesDoubleSameCommitDelivery` — `expected: 1 / actual: 2` ("two same-commit pull_request deliveries are one push and must create exactly ONE pipeline").
- `TestPostHookNeverCoalescesReopen` — with `purgeOnClose` removed: `expected: 2 / actual: 1`.
- `TestUpdatePipelineStatusSkipsStalePendingOverTerminal` — `Should be zero, but was 1` for both the aggregate and the meta post.
- `TestRPCDoneGuardHitPostsTerminalAggregate` — `expected: 1 / actual: 0` against the bare `return err`.
- `TestRPCDoneGuardHitOnBlockedWorkflowPostsNothing` — reddens when the `IsTerminal` precondition is dropped.
- `TestStatusValueIsTerminal` — build failure (method undefined) before the predicate existed.
Over-suppression tripwires (green in both directions by design, red under mutation) cover the inverse risks: a genuinely-pending pipeline still posting, distinct SHAs and distinct PRs never coalescing, the flag defaulting off, push events out of scope, and the clean `Done` path still posting exactly one aggregate.
`go test -tags test ./server/... ./cmd/...` green, plus `-race` on the touched packages; `go vet` and `gofmt` clean. One unrelated failure, `server/web` `Test_custom_file_...`, is **pre-existing** — reproduced identically in a scratch workspace at the untouched `main@origin` (`0b881dd3`).
Spec-impact: docs/specs/tools/woodpecker.md — fork webhook-ingest same-SHA dedup window + flag, and the terminal-wins status-post invariant at updatePipelineStatus (driver lands the spec delta orion-side). Refs RIG-1170. Co-authored-by: Matt Wilkinson <matt@rigel.build>
Resolves the round-1 review of the same-SHA webhook dedup + terminal CI (pr) status-wedge fix. Covers every finding at or above the high+medium block floor, plus two cheap low findings. - rpc.go guard-hit: post the terminal aggregate on the pipeline's own terminal status alone, dropping the second `!IsThereRunningStage` signal and its tree-load. A cascade-cancel leaves running sibling rows untouched, so a terminal pipeline routinely still carries a running row; gating on it re-suppressed the very POST this exists to add, stranding the required check pending. The async poster self-loads the tree and reconcileTerminalStatus makes the terminal verdict win over a stale running row. Adds a stale-running-sibling regression test. - hook dedup tests: add a reset seam so the process-global deduper starts cold per case; the stable keys otherwise survive a -count>1 / parallel / CI-retry rerun and dedupe the first delivery. - isDedupableHook: an empty head SHA is non-dedupable (no dedup signal), so SHA-less deliveries fail open to create instead of collapsing onto one key. - const_test.go: drop the tautological completeness guard (it measured the test's own literals) for an honest note. - helper_test.go: add the missing ID==0 fail-open regression for the stale-pending guard. - docs: document WOODPECKER_HOOK_DEDUP_WINDOW; comment the deliberately-unconditional purgeOnClose ordering. Spec-impact: none. Refs RIG-1170 Co-authored-by: Matt Wilkinson <matt@rigel.build>
…G-1170) Clears the fork's `test-checks` golangci-lint and `static` cspell gates on the review-fix commit; no behavior change. - godot: reword three test doc-comments so no sentence begins with a lowercase identifier. - unparam: drop the always-`true` `aggregate` parameter from `setStatusFlags` (forced on in the helper) and the always-`"abc"` `commit` parameter from `testKey`, updating every caller. - cspell: add `deduper`, `dedupable`, `refspecs`, `terminality`, `TOCTOU` to the dictionary; reword two coined words (`re-Doneing`, `indistinguishability`) in production comments. Spec-impact: none. Refs RIG-1170 Co-authored-by: Matt Wilkinson <matt@rigel.build>
|
CI disposition: every owned workflow is green — This is tracked by RIG-2476 (fork upstream re-sync), and per the standing ruling there, fork PRs are admin-merged past the securityscan red until that re-sync lands. Review loop is all-clear (round-1: 2 high + 3 medium + 3 low, all resolved in the review-fix and lint/spell-hygiene commits on this branch). |
One invariant, three independent mechanisms that violate it: a non-terminal (
pending) status POST must never overwrite a terminal status already reported for the same commit+context. GitHub commit-status is last-write-wins per context, so a latependingstrands the requiredCI (pr)check forever and blocks the merge.Ingest dedup window (RIG-1170,
server/api/hook.go+hook_dedup.go)A
gt submitforce-push makes GitHub emit twopull_requestdeliveries ~1s apart for one head SHA (distinct delivery GUIDs, bothEventPull). Each spawned a pipeline; the pair then mutually cancelled and raced their status posts, and the losing handler's creation-timependinglanded last.PostHooknow consults a mutex-guarded in-memory TTL map keyed(repo.ID, refspec, head SHA)before creating anything. A hit drops the duplicate with HTTP 200 + a debug log; a miss records the key and proceeds. Gated on a newWOODPECKER_HOOK_DEDUP_WINDOWduration flag, default 0 (off) per the fork-flag convention.Scoped so it cannot over-dedup: pull-request pipelines only, so the push path and
cancelPreviousPipelinesare untouched; the SHA in the key means two distinct head SHAs still create and supersede as before; the refspec means two PRs sharing a head SHA never coalesce.Reopen carve-out — the design record's mechanism does not exist in current code. The record says to read the action off the parsed pipeline's
EventReason. Verified against the GitHub driver:opened/reopened/synchronizeall collapse toEventPullwith an emptyEventReason(only metadata actions populate a reason), so a reopen is byte-identical to an ordinary push delivery at this seam — the same indistinguishability that makes the wedge reproduce. The carve-out therefore keys off the delivery that necessarily precedes a reopen: a PR can only be reopened if it was closed, and the close arrives as a separateEventPullClosedon the same refspec, which purges the window. Needs no parser change, works on every forge, and fails safe (a stray purge costs one non-coalesced duplicate, never a missing pipeline).Stale-pending guard (RIG-1170,
server/pipeline/helper.go+server/model/const.go)Defense in depth for when a duplicate slips the window. The shared poster
updatePipelineStatusnow re-reads the pipeline by ID when the about-to-post status is non-terminal, and skips the shared-context (aggregate + meta) POST if the stored status has since gone terminal. One site coversstart.go,cancel.go,decline.goandrestart.go.Deliberately narrow: skip-only (never invents or upgrades a status, never gates a terminal post, a genuinely-pending pipeline still posts); per-workflow statuses are not gated (workflow-scoped contexts are not the wedged check); fail-open on a store error, because dropping a report is worse than a redundant one. A re-read narrows but cannot close the TOCTOU window — which is why the dedup window is the load-bearing half.
New
StatusValue.IsTerminal()predicate inserver/model, partitioned to matchconvertStatusexactly (convertStatusitself untouched). NoteStatusSkippedis non-terminal here: it reports as GitHubpending, so calling it terminal would let a pending-mapped write bypass the guard.Terminal aggregate on the
Doneguard-hit (RIG-1129,server/rpc/rpc.go)An agent-fault double-finish hit
checkWorkflowStateand returned before the only forge report on theDonepath, so a terminal pipeline never resolved its check. The guard-hit now loads the workflow tree and posts the terminal aggregate via the same async poster the clean path uses (s.reportForgeStatusAsync, aggregate-only), then still returns the rejection — the fix adds a missing report, never a state mutation.The record's cited call (
updatePipelineStatus(ctx, forge, currentPipeline, repo, user)) is stale: that helper now takes a_store, and neither_forgenoruseris in scope at the guard-hit. Matched the current clean-path mechanism instead.Terminality is established by two signals, both required.
IsThereRunningStagealone is insufficient: it asks whether any workflow is pending-or-running, and a blocked workflow — the other state this same guard rejects — is neither, so a pipeline merely awaiting approval would look terminal and get a premature verdict posted. Checking the pipeline's own stored status first also means the blocked and still-running paths do no store read at all.Red-green evidence
Every new test was run against the pre-fix code and observed failing:
TestPostHookCoalescesDoubleSameCommitDelivery—expected: 1 / actual: 2("two same-commit pull_request deliveries are one push and must create exactly ONE pipeline").TestPostHookNeverCoalescesReopen— withpurgeOnCloseremoved:expected: 2 / actual: 1.TestUpdatePipelineStatusSkipsStalePendingOverTerminal—Should be zero, but was 1for both the aggregate and the meta post.TestRPCDoneGuardHitPostsTerminalAggregate—expected: 1 / actual: 0against the barereturn err.TestRPCDoneGuardHitOnBlockedWorkflowPostsNothing— reddens when theIsTerminalprecondition is dropped.TestStatusValueIsTerminal— build failure (method undefined) before the predicate existed.Over-suppression tripwires (green in both directions by design, red under mutation) cover the inverse risks: a genuinely-pending pipeline still posting, distinct SHAs and distinct PRs never coalescing, the flag defaulting off, push events out of scope, and the clean
Donepath still posting exactly one aggregate.go test -tags test ./server/... ./cmd/...green, plus-raceon the touched packages;go vetandgofmtclean. One unrelated failure,server/webTest_custom_file_..., is pre-existing — reproduced identically in a scratch workspace at the untouchedmain@origin(0b881dd3).Spec-impact: docs/specs/tools/woodpecker.md — fork webhook-ingest same-SHA dedup window + flag, and the terminal-wins status-post invariant at updatePipelineStatus (driver lands the spec delta orion-side). Refs RIG-1170. Co-authored-by: Matt Wilkinson matt@rigel.build