streams • Advanced

Streams & Backpressure with highWaterMark

Streams process data piece-by-piece in chunks without loading entire files into RAM. Backpressure occurs when a readable stream produces data faster than a writable stream can consume it.

Intuitive Mental Model

The Funnel & Kitchen Sink: If you pour a gallon of water into a narrow funnel (fast reader, slow writer), water overflows unless you pause pouring until the funnel drains.

Node.js ESM / CJS ImplementationNode.js v22 LTS

import fs from 'fs';

const readable = fs.createReadStream('./large_10gb_file.mp4', { highWaterMark: 64 * 1024 }); // 64KB chunks
const writable = fs.createWriteStream('./output.mp4', { highWaterMark: 16 * 1024 }); // 16KB buffer

// Handling Backpressure Manually:
readable.on('data', (chunk) => {
  const canContinue = writable.write(chunk);
  if (!canContinue) {
    readable.pause(); // High-water mark exceeded! Pause reading.
  }
});

writable.on('drain', () => {
  readable.resume(); // Buffer cleared. Resume reading!
});

// Idiomatic Solution: pipeline automatically manages backpressure:
import { pipeline } from 'stream/promises';
await pipeline(readable, writable);

Key Architectural Takeaways

•highWaterMark defines the maximum internal buffer threshold (default: 64KB for fs, 16KB for object streams).
•writable.write(chunk) returns false when buffer is full, signaling backpressure.
•Always use stream.pipeline() or .pipe() instead of raw event listeners to handle errors and backpressure automatically.

Common Production Mistake

Using fs.readFile() on massive 2GB files in API endpoints, blowing up heap memory and causing Out-Of-Memory (OOM) crashes.