javascript / expert
Snippet
Fine-tuning Stream Backpressure with highWaterMark
Backpressure occurs when data is read faster than it can be written. The `highWaterMark` defines the internal buffer size. When `write()` returns `false`, the buffer is full. Handling the 'drain' event allows for memory-efficient data processing without overwhelming the system's RAM.
snippet.js
javascript
1
2
3
4
5
6
7
8
9
10
11
12
const { createReadStream, createWriteStream } = require('fs');const source = createReadStream('large.file', { highWaterMark: 64 * 1024 });const dest = createWriteStream('output.file', { highWaterMark: 16 * 1024 });source.on('data', (chunk) => {const canContinue = dest.write(chunk);if (!canContinue) {source.pause();dest.once('drain', () => source.resume());}});
nodejs
Breakdown
1
highWaterMark: 16 * 1024
Sets the internal buffer limit to 16KB for the writable stream.
2
dest.once('drain', ...)
Resumes the source only after the destination buffer has been cleared.