Deconstruct distributed message brokers from the inside out. Explore partition hashing, OS PageCache Zero-Copy, consumer group rebalancing, and dead-letter retry policies.
Choose the right architecture: transient work dispatch or persistent replayable event streams.
Messages are transient work items. Once consumed and acknowledged, the message is permanently deleted from the broker memory and disk.
Messages are written to an immutable sequential on-disk ledger. Multiple independent consumer groups read at their own pace and can rewind to replay history.
The difference between competing consumer work queues and broadcast event channels.
Navigating At-Most-Once, At-Least-Once, and Idempotent Exactly-Once processing.
Why Kafka partitions use sequential disk writes, OS PageCache, and Zero-Copy for millions of ops/sec.
Bypassing user-space memory buffers using Linux sendfile() and kernel PageCache.
Distribute partition workloads across multiple worker instances with automatic failover.
Detecting processing bottlenecks by monitoring Log-End-Offset vs Consumer-Committed-Offset.