Message Queues

Message Queues

Design reliable message-driven systems.

Queue Types

| Queue | Persistence | Ordering | Use Case |

|-------|------------|----------|----------|

| RabbitMQ | Optional | Per queue | Task distribution, RPC |

| Apache Kafka | Durable (disk) | Per partition | Event streaming, logs |

| AWS SQS | Durable | Best effort (std) / Strict (FIFO) | Serverless decoupling |

| Redis Pub/Sub | None | Per channel | Real-time notifications |

RabbitMQ Patterns

Work Queues (Competing Consumers)

Producer → Queue → Consumer 1
                 → Consumer 2
                 → Consumer 3

• Messages distributed round-robin
• Ack on success, nack on failure (requeue or DLQ)
• Prefetch count controls concurrency

Pub/Sub (Exchange → Binding → Queue)

• Fanout: broadcast to all queues
• Direct: route by routing key
• Topic: route by pattern (user.*, user.created)
• Headers: route by header values

Kafka Patterns

Topics & Partitions

• Messages within a partition are ordered
• Partitions enable parallelism
• Consumer group = one instance per partition

Producer

await producer.send({
  topic: 'order-events',
  messages: [{ key: orderId, value: JSON.stringify(order) }],
});

Consumer

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    await processOrder(message.value);
  },
});

Dead Letter Queues

• Messages that can't be processed go to DLQ
• Analyze DLQ periodically for systemic issues
• DLQ messages can be replayed after fix
• Set max retry count before DLQ

Best Practices

• Idempotent consumers (same message processed twice = safe)
• Monitor queue depth, consumer lag, error rate
• Set message TTL to prevent infinite backlog
• Use structured message schemas (Avro, Protobuf)
• Test with network failures and consumer crashes