caching-queues • Advanced

Message Queues: Kafka Partitions, Consumer Groups & DLQ

Distributed message queues (Kafka, RabbitMQ, SQS) decouple microservices. Kafka uses partitioned append-only commit logs where consumer groups maintain offset pointers, supporting millions of events/sec.

Intuitive Mental Model

The Assembly Line Conveyor Belts: 4 parallel conveyor belts (Partitions). 4 workers (Consumer Group) each monitor 1 belt. Each worker has a clicker counter (Offset) remembering which box they processed last.

Architecture Blueprint & CodeProduction Standard

async function processMessage(msg: Message) {
  try {
    await handlePayment(msg);
    await commitOffset(msg.offset);
  } catch (err) {
    if (msg.retryCount < 3) {
      await publishWithDelay(msg, Math.pow(2, msg.retryCount) * 1000);
    } else {
      await routeToDLQ(msg); // Dead-Letter Queue
      await commitOffset(msg.offset);
    }
  }
}

Key Architectural Takeaways

•Strict Ordering Within Partition: Kafka guarantees message ordering within a single partition, but not across partitions.
•Consumer Rebalancing: If an instance crashes, its assigned partition is reallocated to remaining group consumers.
•Dead-Letter Queue (DLQ): Isolates poisonous or malformed payloads from blocking the entire event stream.

Common Architectural Pitfall

Having more consumers in a group than partitions in a Kafka topic; idle consumers will sit completely dormant doing zero work.

Production Best Practice

Scale topic partition count to match maximum desired consumer parallelism.

Next: Fault Tolerance: Circuit Breaker, Bulkhead & Exponential Backoff