Back to 20 Concepts
resilience • Advanced
Fault Tolerance: Circuit Breaker, Bulkhead & Exponential Backoff
Microservice architectures prevent cascading death spirals using Circuit Breakers (Closed -> Open -> Half-Open), Bulkhead Thread Isolation, and Exponential Backoff with Random Jitter.
Intuitive Mental Model
The Submarine Watertight Doors (Bulkheads): If torpedo shrapnel floods Compartment 3, the submarine closes the bulkhead doors to isolate the leak, keeping the other 9 compartments dry and the ship floating.
Architecture Blueprint & CodeProduction Standard
class CircuitBreaker {
state: 'CLOSED' | 'OPEN' | 'HALF_OPEN' = 'CLOSED';
failureCount = 0;
failureThreshold = 5;
resetTimeoutMs = 10000;
lastFailureTime = 0;
async call(fn: () => Promise<any>) {
if (this.state === 'OPEN') {
if (Date.now() - this.lastFailureTime > this.resetTimeoutMs) {
this.state = 'HALF_OPEN';
} else {
throw new Error('CircuitBreaker: OPEN (Fast Fail)');
}
}
try {
const res = await fn();
this.reset();
return res;
} catch (err) {
this.recordFailure();
throw err;
}
}
}Key Architectural Takeaways
- •Circuit Breaker (Fast Fail): Stops sending requests to broken downstream dependencies, preventing thread pool starvation.
- •Bulkhead Isolation: Partitions thread pools so a failing recommendation service cannot consume all API Gateway threads.
- •Exponential Backoff + Full Jitter: Sleep = random(0, min(maxBackoff, base * 2^attempt)).
Common Architectural Pitfall
Retrying failed network calls immediately in a tight loop across thousands of clients, instantly taking down a recovering backend.
Production Best Practice
Always use Exponential Backoff with randomized full jitter on retries.