2026-09-16 · 11 min
Node.js Stream Processing for Large Datasets: A Production Guide
Most memory problems I have debugged in Node.js backends were not caused by leaks — they were caused by someone reading a whole file into memory and then wondering why the container kept getting OOM-killed at 3 AM. The fix is almost always the same: use streams.
This is not a theoretical article. It covers the patterns I use in production pipelines that process daily transaction exports from our warehouse in Bitung Jaya Industrial Estate, Cikupa — files ranging from 400MB to 4GB in CSV or NDJSON format. The same patterns apply whether you are processing bank statements, telemetry, log archives, or bulk product catalogs.
The memory problem with naive file processing
Here is the code most people write first:
const fs = require('fs');
const csv = require('csv-parse/sync');
// This will OOM on anything over ~500MB
const raw = fs.readFileSync('/data/transactions.csv', 'utf8');
const records = csv.parse(raw, { columns: true });
for (const row of records) {
await db.insert(row);
}
Two problems. readFileSync loads the entire file before your loop even starts. Then csv.parse creates another in-memory array of all records. For a 2GB file you are allocating somewhere around 4–5GB before doing any real work.
The stream version uses roughly constant memory:
const fs = require('fs');
const { pipeline } = require('stream/promises');
const { parse } = require('csv-parse');
const { Transform } = require('stream');
// Memory usage stays flat at ~20-30MB regardless of file size
await pipeline(
fs.createReadStream('/data/transactions.csv'),
parse({ columns: true, skip_empty_lines: true }),
new Transform({
objectMode: true,
async transform(row, _encoding, callback) {
try {
await db.insert(row);
callback();
} catch (err) {
callback(err);
}
}
})
);
stream.pipeline() is the correct abstraction here — not pipe(). It handles backpressure, propagates errors, and destroys all streams in the chain on failure. A .pipe() chain that throws mid-stream will leak the readable.
Understanding backpressure before it bites you
Backpressure is the most misunderstood part of streams. When a Writable cannot keep up with a Readable, Node.js accumulates chunks in an internal buffer. If you ignore this, that buffer grows until your process crashes.
The signal is the return value of writable.write(). When it returns false, the writable’s buffer is full and you should stop pushing. Resume when the drain event fires.
stream.pipeline() manages this automatically. But when you build a custom Readable — say, you are producing rows from a database cursor — you must respect it:
const { Readable } = require('stream');
class DbCursorReadable extends Readable {
constructor(cursor) {
super({ objectMode: true });
this.cursor = cursor;
this._reading = false;
}
async _read() {
if (this._reading) return;
this._reading = true;
try {
// Fetch one batch at a time; only fetch the next
// batch after the consumer signals it is ready
const rows = await this.cursor.fetchMany(500);
if (rows.length === 0) {
this.push(null); // signal end of stream
return;
}
for (const row of rows) {
// push() returns false when the internal buffer is full
if (!this.push(row)) {
// Stop fetching until _read() is called again
this._reading = false;
return;
}
}
this._reading = false;
} catch (err) {
this.destroy(err);
}
}
}
Ignoring the push() return value here means fetching thousands of rows ahead of what the consumer can handle — which lands right back in memory trouble.
Parallel transform with a worker pool
CSV parsing and simple field mapping are cheap. But some pipelines require per-row work that is genuinely CPU-intensive: decompressing embedded payloads, running regex across large text fields, computing cryptographic checksums. Running that on the main thread means your HTTP server stalls during the pipeline run.
The pattern I use is a fixed-size worker pool fed from a stream:
// worker-pool.js
const { Worker, isMainThread, parentPort, workerData } = require('worker_threads');
const os = require('os');
// In the worker — receives one row, returns transformed row
if (!isMainThread) {
parentPort.on('message', (row) => {
// CPU-intensive work lives here
const result = heavyTransform(row);
parentPort.postMessage(result);
});
}
// Pool class for the main thread
class WorkerPool {
constructor(workerFile, size = os.availableParallelism()) {
this.workers = Array.from({ length: size }, () => new Worker(workerFile));
this.queue = [];
this.idle = [...this.workers];
for (const worker of this.workers) {
worker.on('message', (result) => {
const { resolve } = this.queue.shift();
resolve(result);
this.idle.push(worker);
this._drain();
});
}
}
run(data) {
return new Promise((resolve, reject) => {
this.queue.push({ resolve, reject, data });
this._drain();
});
}
_drain() {
while (this.idle.length > 0 && this.queue.length > 0) {
const worker = this.idle.pop();
const { data } = this.queue[this.queue.length - 1];
worker.postMessage(data);
}
}
async destroy() {
await Promise.all(this.workers.map(w => w.terminate()));
}
}
Wire the pool into a Transform stream:
const pool = new WorkerPool('./heavy-transform-worker.js', 4);
const parallelTransform = new Transform({
objectMode: true,
async transform(row, _encoding, callback) {
try {
const result = await pool.run(row);
callback(null, result);
} catch (err) {
callback(err);
}
}
});
await pipeline(
source,
parallelTransform,
sink
);
await pool.destroy();
Four workers on a 4-core machine keeps CPU saturated without over-subscribing. os.availableParallelism() is the right default here — it respects cgroup limits in containers, unlike os.cpus().length which reads the host CPU count.
Handling errors without losing progress
Long-running pipelines fail mid-stream. A corrupted row, a transient DB error, a malformed date — these should not kill a pipeline that is 80% through a 3GB file.
The pattern is a per-row error handler that routes bad rows to a separate error log rather than re-throwing:
const { createWriteStream } = require('fs');
const { stringify } = require('csv-stringify');
const errorStream = stringify({ header: true });
errorStream.pipe(createWriteStream('/data/errors.csv'));
const safeParse = new Transform({
objectMode: true,
transform(row, _encoding, callback) {
try {
const validated = validateRow(row);
callback(null, validated);
} catch (err) {
// Route bad row to error log, emit warning, continue
errorStream.write({ ...row, _error: err.message });
process.emitWarning(`Skipped row ${row.id}: ${err.message}`);
callback(); // no error argument — pipeline continues
}
},
flush(callback) {
// Close error stream when main pipeline finishes
errorStream.end();
callback();
}
});
After the pipeline completes, inspect the error CSV to decide whether re-processing is needed. For idempotent inserts (upsert by primary key), you can re-run the whole file safely without worrying about duplicates.
Monitoring and observability for long pipelines
A pipeline that processes 4GB silently for 45 minutes is uncomfortable. Add lightweight progress tracking without disturbing the stream:
let rowCount = 0;
let byteCount = 0;
const startTime = Date.now();
const counter = new Transform({
objectMode: true,
transform(row, _encoding, callback) {
rowCount++;
// Approximate byte count from JSON representation
byteCount += JSON.stringify(row).length;
// Log progress every 10,000 rows
if (rowCount % 10_000 === 0) {
const elapsed = (Date.now() - startTime) / 1000;
const rps = Math.round(rowCount / elapsed);
console.log(`[${new Date().toISOString()}] ${rowCount.toLocaleString()} rows | ${rps} rows/s | ~${(byteCount / 1e6).toFixed(1)} MB processed`);
}
callback(null, row);
}
});
In production this log goes to Loki, and a Grafana dashboard shows rows/sec over time. A sudden drop in throughput — visible as a slope change on the graph — usually means the database connection pool is saturated or the downstream write is waiting on a lock.
What not to do
A few patterns I see repeatedly that cause problems:
Collecting chunks to process in batches but letting the batch grow unbounded. If you buffer rows into an array and only flush when it hits some size, you need a hard upper bound on that size. An INSERT batch of 50,000 rows is not meaningfully faster than 1,000 rows and uses 50x the memory per batch.
Using fs.promises.readFile() and assuming it is streaming. It is not. It reads the entire file. The fs promise API is convenient but not appropriate for large files.
Forgetting to handle error events on streams when not using pipeline(). Unhandled error events crash the process. If you must use .pipe() for some reason, attach .on('error', handler) to every stream in the chain.
Creating a new Transform instance per row in a loop. Streams have initialization overhead. Create one Transform, push many rows through it.
When streams are not enough
Streams solve the I/O memory problem. They do not solve throughput. If inserting a row takes 10ms and you have 10 million rows, a single-threaded stream will take about 28 hours. At that point the question is not about Node.js streams — it is about bulk loading.
For PostgreSQL, COPY FROM STDIN via pg-copy-streams is an order of magnitude faster than individual inserts and works natively with Node.js writable streams. For ClickHouse, the HTTP bulk insert endpoint accepts NDJSON directly from a stream. For DynamoDB, BatchWriteItem with 25-item batches is the ceiling before you start hitting provisioned throughput limits.
The stream pipeline gets you constant memory. The write strategy determines whether you finish in 30 minutes or 28 hours.
Streams in Node.js have been production-ready for years. The API has sharp edges — backpressure is easy to ignore, error propagation in .pipe() chains is surprising — but stream.pipeline() smooths most of them. If your team is still reaching for readFileSync on files over 50MB, the payoff from switching is immediate and measurable.
Ditulis oleh Reza Pradipta