Back to 20 Concepts
distributed-data • Expert
Database Sharding, Partition Keys & Write-Ahead Logging (WAL)
Sharding horizontally splits huge database tables across independent physical database servers using a Shard Key. Write-Ahead Logs (WAL) append sequential disk writes before mutating in-memory buffers to guarantee ACID durability.
Intuitive Mental Model
The Multi-Volume Encyclopedia: Instead of binding 1,000,000 pages into 1 gargantuan book that crushes the table, you divide the encyclopedia into 26 alphabetical volumes A-Z (horizontal sharding).
Architecture Blueprint & CodeProduction Standard
function getShardServer(userId: string, shardClusters: string[]): string {
const hash = murmurHash3(userId);
const shardId = hash % shardClusters.length;
return shardClusters[shardId];
}Key Architectural Takeaways
- •Shard Key Selection: Must have high cardinality and uniform query distribution to prevent celebrity hot-spot shards.
- •WAL Sequential IO: Appending to an append-only log file on NVMe disk is orders of magnitude faster than random B+ Tree disk page mutations.
- •Re-sharding: Requires dual-writing, background data backfill, and dynamic routing updates.
Common Architectural Pitfall
Choosing a timestamp (e.g. created_at) as a shard key, which routes 100% of current write traffic to the single most recent shard (hot-spotting).
Production Best Practice
Use a high-cardinality, uniformly distributed hash of UUIDs or user IDs combined with range lookup routing.