Skip to content

fix(workflows): make tracked-event dispatch durable - #467

Draft
vladbisceanu wants to merge 6 commits into
useplunk:nextfrom
vladbisceanu:upstream/138-durable-track
Draft

fix(workflows): make tracked-event dispatch durable#467
vladbisceanu wants to merge 6 commits into
useplunk:nextfrom
vladbisceanu:upstream/138-durable-track

Conversation

@vladbisceanu

Copy link
Copy Markdown

Problem

/v1/track stores an event before it dispatches matching workflows. A process exit or workflow-start failure in that gap can leave an accepted event with no workflow execution. The previous path logged dispatch failures and moved on, so there was no bounded recovery path.

What changed

  • Treat the event row as the workflow outbox. A minutely maintenance sweep reclaims pending events after a five-minute grace period.
  • Fence dispatch with a renewable database lease. Failed dispatches use exponential backoff from one minute to one hour and dead-letter after eight attempts.
  • Make workflow enrollment idempotent per workflow and source event.
  • Persist wait-resume ownership, payload, and merged execution context so a retry can finish the continuation without replaying an acknowledged downstream step.
  • Add online-safe migrations and document the accepted-response contract.

The recovery invariant is: an accepted event remains recoverable until every matching workflow target has a durable enrollment or resume claim, or the bounded retry policy dead-letters it for inspection.

Stack

This PR is stacked because the durability path builds on three open PRs that all target next:

The P6-specific commits are 27f4332 and 8b2e421. The dependency diff will disappear when those PRs merge and this branch is rebased.

Known follow-up

Final local review found the event-versus-timeout race already assigned to setup-works#139: the timeout path still needs the compare-and-swap state transition planned for P7. This PR does not claim to close that race. P7 will be a separate upstream PR before deployment.

Verification

  • API and dependency build passed on Node 24.
  • Focused durability, SNS, workflow execution, and workflow service tests: 114 passed.
  • Full API suite: 36 files, 938 tests passed.
  • Fresh PostgreSQL schema: all 26 migrations applied successfully.
  • Final local review: one finding, deferred to the existing P7 issue above.

Tracks setup-works#138.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant