Commit Logs / 01ADVANCED

Append-Only Commit Logs & Partitions

“Instead of deleting messages upon read, systems like Apache Kafka and Redpanda store events in an immutable, append-only commit log on disk. Topics are split into multiple partitions for horizontal scaling.”

Track Domain

Append-Only Commit Logs

Why Kafka partitions use sequential disk writes, OS PageCache, and Zero-Copy for millions of ops/sec.

Physical World Analogy

An immutable courtroom transcript ledger: stenographers only append new lines to the end of the page. Readers can reread past testimony at any speed simply by remembering their bookmark page number.

Architecture Implementation

Production Pattern & Code Contract

kafka-commit-log.tsPROD_VERIFIED

// Producing with Key-Based Partitioning
const producer = kafka.producer();
await producer.send({
  topic: 'customer-transactions',
  messages: [
    { 
      key: 'user_4821', // All events for user_4821 route to the SAME partition
      value: JSON.stringify({ action: 'TRANSFER', amount: 500 }),
      timestamp: Date.now()
    }
  ]
});

Guarantees sequential FIFO execution for each user without locking other users.

Execution Workflow

Step-by-Step State Flow

Workflow step 01 selected.

Critical Pitfalls to Avoid

Creating thousands of tiny partitions per broker, overwhelming OS file descriptors and metadata sync

Using random or missing keys when strict per-entity ordering is required

Treating Kafka like a traditional transient queue and attempting to delete individual messages

Systems Engineering Optimization

Sequential disk I/O approaches raw physical drive write speeds (hundreds of MB/s), outperforming random memory access in traditional database engines.

Related Messaging Primitives

Scaling / 01

Consumer Groups & Partition Rebalancing

View concept Commit Logs / 02

Zero-Copy & OS PageCache Mechanics

View concept

Loading MQ::STREAM...