Skip to content

[workers] Worker has no pause, and a clean stop consumes delivery attempts #2635

Closed

Description

@rickylabs

Summary

A worker cannot stop taking new jobs without also aborting the jobs it is running, and every job sent back during a clean stop uses up one delivery attempt. A few restarts in a row dead-letter healthy jobs.

Evidence (main at b0eb079, same on v0.0.9)

  • plugins/workers/worker/worker.ts:277-327: drain() aborts every in-flight job at drain start (execution.abortController.abort(shutdownReason)), through the same step that stops the listeners. There is no pause() / resume().
  • plugins/workers/worker/job-dispatcher.ts:220-228: an aborted job is requeued with queueContext.nack({ requeue: true }).
  • packages/queue/adapters/_redis-lease-state.ts: the delivery count is incremented at claim time, and maxDeliveries defaults to 5. packages/queue/adapters/_redis-lease.ts dead-letters once the count reaches it. A requeue caused by shutdown is counted like a failure.

Expected behavior

  • pause() stops intake and leaves in-flight jobs running; resume() restarts intake.
  • A drain stops intake, waits for in-flight jobs up to a budget, and only then aborts what is left.
  • A job handed back because of shutdown does not consume a delivery attempt.

Acceptance

  • Worker exposes pause() and resume(); a paused worker claims nothing and its running jobs finish.
  • A drain waits for in-flight jobs up to a configurable budget before aborting, proven by a test where a job that finishes inside the budget completes and is acknowledged.
  • A requeue caused by shutdown leaves the delivery count unchanged, on every queue adapter that tracks one, proven by a test that restarts a worker more than maxDeliveries times without dead-lettering the job.
  • A requeue caused by a handler failure still counts.
  • The workers documentation describes pause, drain and the delivery-count rule.

Acceptance evidence: PR #2645 implementation and local validation. Each checked criterion maps to its fenced acceptance-evidence entry in the PR body. Full supported runtime and independent evaluation remain PR readiness gates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions