Skip to content

fix: bind durable DLQ consumer so retry drains on workqueue stream - #220

Open
ecv wants to merge 2 commits into
mainfrom
fix/dlq-retry-durable-consumer
Open

fix: bind durable DLQ consumer so retry drains on workqueue stream#220
ecv wants to merge 2 commits into
mainfrom
fix/dlq-retry-durable-consumer

Conversation

@ecv

@ecv ecv commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Problem

The periodic DLQ retry run never drains anything (totalSucceeded: 0 on every run) and the consumer frequently can't even attach:

"Failed to create DLQ consumer","subject":"activity.dlq.>","err":"nats: filtered consumer not unique on workqueue stream"
"Failed to fetch DLQ messages","subject":"activity.dlq.>","err":"nats: no responders available for request"

ACTIVITY_DEAD_LETTER is a workqueue-retention stream, which permits only one consumer per non-overlapping subject filter. But processRetryBatch created an ephemeral pull consumer on every batch (PullSubscribe(subject, "", BindStream)):

  • Two processor replicas each run periodicRetry on their own ticker, so two consumers filter activity.dlq.> and the second is rejected.
  • RetryForPolicy used a narrower filter activity.dlq.*.<apiGroup>.<kind> that overlaps activity.dlq.>, so it collides even within one pod.
  • A consumer that briefly attaches then hits defer Unsubscribe leaves Fetch with no responders available.

Net: the consumer usually can't attach, so nothing drains. In prod the backlog has been flat at 177 for 42h with zero drain.

Secondary: when a consumer did attach, backoff-ineligible events were msg.Nak()'d (immediate redelivery), so the same events were re-fetched every batch, processed == 0 was never reached, and the run spun to the 2-minute cap. That is the totalProcessed: 107000, totalSucceeded: 0 signature.

Fix

  1. Provision a durable DLQ consumer. config/components/nats-streams/dlq-retry-consumer.yaml (durable pull consumer, filter activity.dlq.>), wired into the nats-streams kustomization, following the same declarative pattern as every other stream's consumer.
  2. Bind the shared durable in processRetryBatch instead of creating an ephemeral one, and drop the overlapping per-policy server-side filter. Per-policy narrowing already happens client-side in the batch loop. Both replicas compete for messages on the one consumer; no second overlapping consumer is ever created, and the durable is never unsubscribed or deleted.
  3. NakWithDelay backed-off events instead of an immediate Nak, so a periodic run reaches processed == 0 and terminates instead of re-fetching the same events until the deadline.

Tests

New internal/activityprocessor/dlq_retry_integration_test.go runs an in-process JetStream server and reproduces all three failure modes:

  • TestDLQRetryDrainsWorkqueueStream publishes 5 events to a workqueue DLQ stream, runs the controller, asserts the DLQ drains to 0 and the events are republished. The old ephemeral consumer failed to attach here.
  • TestDLQRetryTwoReplicasNoConsumerCollision has two connections bind the shared durable; both must succeed. This is the exact filtered consumer not unique scenario.
  • TestDLQRetryDefersBackedOffEvents fetches a backed-off event once and asserts it is not redelivered within the run, guarding the spin.

Adds nats-io/nats-server/v2 as a test dependency for the embedded server. go build ./... and go test ./internal/activityprocessor/... pass. Root-cause writeup with prod evidence in #216.

Post-merge

Once the durable exists, the 177-event backlog drains: project events re-evaluate cleanly (milo CEL fix live in v0.30.1). Connector and billing events keep cycling on proper backoff until their policy fixes land (datum-cloud/network-services-operator#241, milo-os/billing#75), deferred by NakWithDelay rather than spinning.

Fixes #216

The DLQ retry controller created an ephemeral pull consumer per batch on
the ACTIVITY_DEAD_LETTER stream, which uses workqueue retention. Workqueue
streams permit only one consumer per non-overlapping subject filter, so
concurrent replicas and the per-policy narrow filter collided with
"filtered consumer not unique on workqueue stream" and the DLQ never
drained (totalSucceeded: 0 every run; prod backlog flat for 42h).

Bind a single shared durable consumer instead, provisioned declaratively
like every other stream's consumer. Per-policy filtering stays client-side,
so no second overlapping consumer is created. Defer redelivery of
backed-off events with NakWithDelay so a periodic run reaches processed==0
and stops instead of re-fetching the same events until the run deadline.

Key changes:
- Add dlq-retry-consumer.yaml durable pull consumer (filter activity.dlq.>)
  and wire it into the nats-streams kustomization
- Bind the durable in processRetryBatch; drop the ephemeral consumer and
  the overlapping per-policy server-side filter
- NakWithDelay backed-off events instead of immediate Nak

Fixes #216
@ecv
ecv requested review from JoseSzycho and scotwells July 9, 2026 13:48
Add in-process JetStream tests covering the #216 failure modes: draining a
workqueue-retention DLQ stream, two replicas binding the shared durable
consumer without a "filtered consumer not unique" collision, and a
backed-off event not being redelivered within the same run.

Adds nats-server/v2 as a test dependency for the embedded server.
@ecv ecv self-assigned this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DLQ periodic retry never drains: NATS consumer attach fails (filtered consumer not unique on workqueue stream)

1 participant