Skip to content

fix(server): coalesce same-SHA webhooks and guard terminal CI (pr) status (RIG-1170) - #34

Open
rigel-mintaka wants to merge 3 commits into
mainfrom
pipelines/rig-1170-status-wedge
Open

fix(server): coalesce same-SHA webhooks and guard terminal CI (pr) status (RIG-1170)#34
rigel-mintaka wants to merge 3 commits into
mainfrom
pipelines/rig-1170-status-wedge

Conversation

@rigel-mintaka

Copy link
Copy Markdown

One invariant, three independent mechanisms that violate it: a non-terminal (pending) status POST must never overwrite a terminal status already reported for the same commit+context. GitHub commit-status is last-write-wins per context, so a late pending strands the required CI (pr) check forever and blocks the merge.

Ingest dedup window (RIG-1170, server/api/hook.go + hook_dedup.go)

A gt submit force-push makes GitHub emit two pull_request deliveries ~1s apart for one head SHA (distinct delivery GUIDs, both EventPull). Each spawned a pipeline; the pair then mutually cancelled and raced their status posts, and the losing handler's creation-time pending landed last.

PostHook now consults a mutex-guarded in-memory TTL map keyed (repo.ID, refspec, head SHA) before creating anything. A hit drops the duplicate with HTTP 200 + a debug log; a miss records the key and proceeds. Gated on a new WOODPECKER_HOOK_DEDUP_WINDOW duration flag, default 0 (off) per the fork-flag convention.

Scoped so it cannot over-dedup: pull-request pipelines only, so the push path and cancelPreviousPipelines are untouched; the SHA in the key means two distinct head SHAs still create and supersede as before; the refspec means two PRs sharing a head SHA never coalesce.

Reopen carve-out — the design record's mechanism does not exist in current code. The record says to read the action off the parsed pipeline's EventReason. Verified against the GitHub driver: opened/reopened/synchronize all collapse to EventPull with an empty EventReason (only metadata actions populate a reason), so a reopen is byte-identical to an ordinary push delivery at this seam — the same indistinguishability that makes the wedge reproduce. The carve-out therefore keys off the delivery that necessarily precedes a reopen: a PR can only be reopened if it was closed, and the close arrives as a separate EventPullClosed on the same refspec, which purges the window. Needs no parser change, works on every forge, and fails safe (a stray purge costs one non-coalesced duplicate, never a missing pipeline).

Stale-pending guard (RIG-1170, server/pipeline/helper.go + server/model/const.go)

Defense in depth for when a duplicate slips the window. The shared poster updatePipelineStatus now re-reads the pipeline by ID when the about-to-post status is non-terminal, and skips the shared-context (aggregate + meta) POST if the stored status has since gone terminal. One site covers start.go, cancel.go, decline.go and restart.go.

Deliberately narrow: skip-only (never invents or upgrades a status, never gates a terminal post, a genuinely-pending pipeline still posts); per-workflow statuses are not gated (workflow-scoped contexts are not the wedged check); fail-open on a store error, because dropping a report is worse than a redundant one. A re-read narrows but cannot close the TOCTOU window — which is why the dedup window is the load-bearing half.

New StatusValue.IsTerminal() predicate in server/model, partitioned to match convertStatus exactly (convertStatus itself untouched). Note StatusSkipped is non-terminal here: it reports as GitHub pending, so calling it terminal would let a pending-mapped write bypass the guard.

Terminal aggregate on the Done guard-hit (RIG-1129, server/rpc/rpc.go)

An agent-fault double-finish hit checkWorkflowState and returned before the only forge report on the Done path, so a terminal pipeline never resolved its check. The guard-hit now loads the workflow tree and posts the terminal aggregate via the same async poster the clean path uses (s.reportForgeStatusAsync, aggregate-only), then still returns the rejection — the fix adds a missing report, never a state mutation.

The record's cited call (updatePipelineStatus(ctx, forge, currentPipeline, repo, user)) is stale: that helper now takes a _store, and neither _forge nor user is in scope at the guard-hit. Matched the current clean-path mechanism instead.

Terminality is established by two signals, both required. IsThereRunningStage alone is insufficient: it asks whether any workflow is pending-or-running, and a blocked workflow — the other state this same guard rejects — is neither, so a pipeline merely awaiting approval would look terminal and get a premature verdict posted. Checking the pipeline's own stored status first also means the blocked and still-running paths do no store read at all.

Red-green evidence

Every new test was run against the pre-fix code and observed failing:

  • TestPostHookCoalescesDoubleSameCommitDeliveryexpected: 1 / actual: 2 ("two same-commit pull_request deliveries are one push and must create exactly ONE pipeline").
  • TestPostHookNeverCoalescesReopen — with purgeOnClose removed: expected: 2 / actual: 1.
  • TestUpdatePipelineStatusSkipsStalePendingOverTerminalShould be zero, but was 1 for both the aggregate and the meta post.
  • TestRPCDoneGuardHitPostsTerminalAggregateexpected: 1 / actual: 0 against the bare return err.
  • TestRPCDoneGuardHitOnBlockedWorkflowPostsNothing — reddens when the IsTerminal precondition is dropped.
  • TestStatusValueIsTerminal — build failure (method undefined) before the predicate existed.

Over-suppression tripwires (green in both directions by design, red under mutation) cover the inverse risks: a genuinely-pending pipeline still posting, distinct SHAs and distinct PRs never coalescing, the flag defaulting off, push events out of scope, and the clean Done path still posting exactly one aggregate.

go test -tags test ./server/... ./cmd/... green, plus -race on the touched packages; go vet and gofmt clean. One unrelated failure, server/web Test_custom_file_..., is pre-existing — reproduced identically in a scratch workspace at the untouched main@origin (0b881dd3).

Spec-impact: docs/specs/tools/woodpecker.md — fork webhook-ingest same-SHA dedup window + flag, and the terminal-wins status-post invariant at updatePipelineStatus (driver lands the spec delta orion-side). Refs RIG-1170. Co-authored-by: Matt Wilkinson matt@rigel.build

…atus (RIG-1170)

One invariant, three independent mechanisms that violate it: **a non-terminal (`pending`) status POST must never overwrite a terminal status already reported for the same commit+context.** GitHub commit-status is last-write-wins per context, so a late `pending` strands the required `CI (pr)` check forever and blocks the merge.

## Ingest dedup window (RIG-1170, `server/api/hook.go` + `hook_dedup.go`)

A `gt submit` force-push makes GitHub emit **two** `pull_request` deliveries ~1s apart for one head SHA (distinct delivery GUIDs, both `EventPull`). Each spawned a pipeline; the pair then mutually cancelled and raced their status posts, and the losing handler's creation-time `pending` landed last.

`PostHook` now consults a mutex-guarded in-memory TTL map keyed `(repo.ID, refspec, head SHA)` before creating anything. A hit drops the duplicate with HTTP 200 + a debug log; a miss records the key and proceeds. Gated on a new `WOODPECKER_HOOK_DEDUP_WINDOW` duration flag, **default 0 (off)** per the fork-flag convention.

Scoped so it cannot over-dedup: pull-request pipelines only, so the push path and `cancelPreviousPipelines` are untouched; the SHA in the key means two distinct head SHAs still create and supersede as before; the refspec means two PRs sharing a head SHA never coalesce.

**Reopen carve-out — the design record's mechanism does not exist in current code.** The record says to read the action off the parsed pipeline's `EventReason`. Verified against the GitHub driver: `opened`/`reopened`/`synchronize` all collapse to `EventPull` with an **empty** `EventReason` (only metadata actions populate a reason), so a reopen is byte-identical to an ordinary push delivery at this seam — the same indistinguishability that makes the wedge reproduce. The carve-out therefore keys off the delivery that necessarily *precedes* a reopen: a PR can only be reopened if it was closed, and the close arrives as a separate `EventPullClosed` on the same refspec, which purges the window. Needs no parser change, works on every forge, and fails safe (a stray purge costs one non-coalesced duplicate, never a missing pipeline).

## Stale-pending guard (RIG-1170, `server/pipeline/helper.go` + `server/model/const.go`)

Defense in depth for when a duplicate slips the window. The shared poster `updatePipelineStatus` now re-reads the pipeline by ID when the about-to-post status is non-terminal, and skips the shared-context (aggregate + meta) POST if the stored status has since gone terminal. One site covers `start.go`, `cancel.go`, `decline.go` and `restart.go`.

Deliberately narrow: skip-only (never invents or upgrades a status, never gates a terminal post, a genuinely-pending pipeline still posts); per-workflow statuses are not gated (workflow-scoped contexts are not the wedged check); fail-open on a store error, because dropping a report is worse than a redundant one. A re-read narrows but cannot close the TOCTOU window — which is why the dedup window is the load-bearing half.

New `StatusValue.IsTerminal()` predicate in `server/model`, partitioned to match `convertStatus` exactly (`convertStatus` itself untouched). Note `StatusSkipped` is **non**-terminal here: it reports as GitHub `pending`, so calling it terminal would let a pending-mapped write bypass the guard.

## Terminal aggregate on the `Done` guard-hit (RIG-1129, `server/rpc/rpc.go`)

An agent-fault double-finish hit `checkWorkflowState` and returned before the only forge report on the `Done` path, so a terminal pipeline never resolved its check. The guard-hit now loads the workflow tree and posts the terminal aggregate via the same async poster the clean path uses (`s.reportForgeStatusAsync`, aggregate-only), then still returns the rejection — the fix adds a missing *report*, never a state *mutation*.

The record's cited call (`updatePipelineStatus(ctx, forge, currentPipeline, repo, user)`) is stale: that helper now takes a `_store`, and neither `_forge` nor `user` is in scope at the guard-hit. Matched the current clean-path mechanism instead.

Terminality is established by **two** signals, both required. `IsThereRunningStage` alone is insufficient: it asks whether any workflow is pending-or-running, and a **blocked** workflow — the other state this same guard rejects — is neither, so a pipeline merely awaiting approval would look terminal and get a premature verdict posted. Checking the pipeline's own stored status first also means the blocked and still-running paths do no store read at all.

## Red-green evidence

Every new test was run against the pre-fix code and observed failing:

- `TestPostHookCoalescesDoubleSameCommitDelivery` — `expected: 1 / actual: 2` ("two same-commit pull_request deliveries are one push and must create exactly ONE pipeline").
- `TestPostHookNeverCoalescesReopen` — with `purgeOnClose` removed: `expected: 2 / actual: 1`.
- `TestUpdatePipelineStatusSkipsStalePendingOverTerminal` — `Should be zero, but was 1` for both the aggregate and the meta post.
- `TestRPCDoneGuardHitPostsTerminalAggregate` — `expected: 1 / actual: 0` against the bare `return err`.
- `TestRPCDoneGuardHitOnBlockedWorkflowPostsNothing` — reddens when the `IsTerminal` precondition is dropped.
- `TestStatusValueIsTerminal` — build failure (method undefined) before the predicate existed.

Over-suppression tripwires (green in both directions by design, red under mutation) cover the inverse risks: a genuinely-pending pipeline still posting, distinct SHAs and distinct PRs never coalescing, the flag defaulting off, push events out of scope, and the clean `Done` path still posting exactly one aggregate.

`go test -tags test ./server/... ./cmd/...` green, plus `-race` on the touched packages; `go vet` and `gofmt` clean. One unrelated failure, `server/web` `Test_custom_file_...`, is **pre-existing** — reproduced identically in a scratch workspace at the untouched `main@origin` (`0b881dd3`).

Spec-impact: docs/specs/tools/woodpecker.md — fork webhook-ingest same-SHA dedup window + flag, and the terminal-wins status-post invariant at updatePipelineStatus (driver lands the spec delta orion-side). Refs RIG-1170. Co-authored-by: Matt Wilkinson <matt@rigel.build>
@linear-code

linear-code Bot commented Sep 4, 2026

Copy link
Copy Markdown

RIG-1170

rigel-mintaka and others added 2 commits September 4, 2026 14:30
Resolves the round-1 review of the same-SHA webhook dedup + terminal CI (pr) status-wedge fix. Covers every finding at or above the high+medium block floor, plus two cheap low findings.

- rpc.go guard-hit: post the terminal aggregate on the pipeline's own terminal status alone, dropping the second `!IsThereRunningStage` signal and its tree-load. A cascade-cancel leaves running sibling rows untouched, so a terminal pipeline routinely still carries a running row; gating on it re-suppressed the very POST this exists to add, stranding the required check pending. The async poster self-loads the tree and reconcileTerminalStatus makes the terminal verdict win over a stale running row. Adds a stale-running-sibling regression test.
- hook dedup tests: add a reset seam so the process-global deduper starts cold per case; the stable keys otherwise survive a -count>1 / parallel / CI-retry rerun and dedupe the first delivery.
- isDedupableHook: an empty head SHA is non-dedupable (no dedup signal), so SHA-less deliveries fail open to create instead of collapsing onto one key.
- const_test.go: drop the tautological completeness guard (it measured the test's own literals) for an honest note.
- helper_test.go: add the missing ID==0 fail-open regression for the stale-pending guard.
- docs: document WOODPECKER_HOOK_DEDUP_WINDOW; comment the deliberately-unconditional purgeOnClose ordering.

Spec-impact: none. Refs RIG-1170

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…G-1170)

Clears the fork's `test-checks` golangci-lint and `static` cspell gates on the review-fix commit; no behavior change.

- godot: reword three test doc-comments so no sentence begins with a lowercase identifier.
- unparam: drop the always-`true` `aggregate` parameter from `setStatusFlags` (forced on in the helper) and the always-`"abc"` `commit` parameter from `testKey`, updating every caller.
- cspell: add `deduper`, `dedupable`, `refspecs`, `terminality`, `TOCTOU` to the dictionary; reword two coined words (`re-Doneing`, `indistinguishability`) in production comments.

Spec-impact: none. Refs RIG-1170

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@rigel-mintaka
rigel-mintaka marked this pull request as ready for review September 4, 2026 19:12
@rigel-mintaka

Copy link
Copy Markdown
Author

CI disposition: every owned workflow is green — test, test-checks (golangci-lint + openapi + license-header), static (spellcheck), web, test-datastore (postgres + mysql). The single red leg is securityscan (Trivy), failing on stale upstream dependency CVEs (golang.org/x/crypto on the backend; fast-uri/qs under docs/) that this PR does not touch — the diff changes zero dependency manifests. Fork main was securityscan-green on the 2026-08-26/27 push pipelines; the CVEs post-date those runs after a Trivy DB refresh.

This is tracked by RIG-2476 (fork upstream re-sync), and per the standing ruling there, fork PRs are admin-merged past the securityscan red until that re-sync lands. Review loop is all-clear (round-1: 2 high + 3 medium + 3 low, all resolved in the review-fix and lint/spell-hygiene commits on this branch).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant