Skip to content

feat: implement Dead Letter Queue (DLQ)#111

Description

@LeonardoVieira1630

馃摌 Context

Currently, when the system encounters a message it cannot process (e.g., unknown trigger), it enters an infinite reprocessing loop due to NACK with requeue. This causes:

  • High redelivery rate (200+ messages/second)
  • Blocking of other messages in the queue (prefetch=1)
  • Unnecessary resource consumption
  • Inability to process valid messages

馃悶 Problem Identified

During an accidental deployment of development code to production, voting-power-changed messages were generated. When production code was restored, these messages got stuck in the queue causing an infinite loop, as the dispatcher doesn't have a handler for this trigger type.

馃敡 Tasks

  1. Update RabbitMQConsumer to automatically create DLQ

    • Modify consumer.ts to configure DLQ for all queues
    • Configure queue with x-dead-letter-exchange and x-dead-letter-routing-key
    • Automatically create DLQ with .dlq suffix
    • Set 7-day TTL for messages in DLQ
  2. Implement retry logic with counter

    • Add x-retry-count header to messages
    • After 3 attempts, use channel.reject(msg, false) to send to DLQ
    • Include error information in message header
  3. Add default handler for unknown triggers

    • Warning log for unregistered trigger
    • Direct ACK after logging (don't send to DLQ)
  4. Create monitoring script

    • Script to check messages in DLQs
    • Prepare structure for n8n integration
  5. Tests

    • Unit test for retry behavior
    • Integration test with unknown trigger message
    • Verify messages go to DLQ after 3 attempts

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions