122: Buffers, Binary Data, Encodings, Streams, Pipelines, and Backpressure

Learning objectives

You will learn to:

explain what a byte is and why Node needs Buffer;
distinguish text encoding from binary data;
create and inspect Buffers safely;
understand UTF-8 boundaries and byte length;
understand readable, writable, duplex, and transform streams;
consume streams using events, async iteration, and pipeline;
understand backpressure;
build transform pipelines;
avoid buffering large payloads unnecessarily;
handle stream errors and cancellation;
know when Web Streams and Node streams interact.

Why binary data matters

JavaScript strings are abstractions for text. They are convenient for displaying and manipulating characters, but they are not the representation used for every kind of data a server handles.

Networks, files, compressed data, images, and cryptographic material are ultimately sequences of bytes. Node's Buffer is the byte-oriented data type used to work with those sequences. It is built on typed-array concepts, but provides APIs that are especially useful for Node's I/O systems.

const buffer = Buffer.from('hello', 'utf8');

console.log(buffer);
console.log(buffer.length);

buffer.length reports the number of bytes in the Buffer, not the number of JavaScript string characters that were used to create it. Those numbers happen to match for some ASCII text, which is why the distinction is easy to miss.

UTF-8 byte length

The difference becomes visible as soon as the text contains characters whose UTF-8 representation uses more than one byte:

console.log('A'.length); // 1
console.log(Buffer.byteLength('A', 'utf8')); // 1

console.log('₹'.length);
console.log(Buffer.byteLength('₹', 'utf8'));

JavaScript string .length counts UTF-16 code units. It is therefore a language-level text measurement, not a measurement of the encoded payload. Buffer.byteLength() answers the question that matters when the value will be sent or stored as UTF-8.

Network and request-body limits usually care about bytes. Do not enforce upload or request limits using only string character counts; a string with relatively few characters can occupy substantially more encoded bytes.

Create Buffers

The constructor you choose communicates how the memory should be initialized and where its contents come from.

From text:

const a = Buffer.from('hello', 'utf8');

Allocate initialized memory. The bytes start as zeroes:

const b = Buffer.alloc(1024);

Allocate uninitialized memory:

const c = Buffer.allocUnsafe(1024);

allocUnsafe can be faster because Node does not clear the allocated memory first. That speed comes with a strict obligation: every byte must be overwritten before the Buffer is read or exposed.

Do not send uninitialized Buffer contents to users. Apart from producing invalid output, doing so can disclose data left in reused memory.

Encoding and decoding

Encoding turns a sequence of bytes into a representation such as hexadecimal or Base64. Decoding turns that representation back into bytes or text. The representation you choose affects how the bytes are displayed, not the underlying security properties.

const bytes = Buffer.from('hello', 'utf8');

console.log(bytes.toString('hex'));
console.log(bytes.toString('base64'));
console.log(bytes.toString('utf8'));

Base64 is an encoding, not encryption. Anyone who can read a Base64 value can decode it.

Do not treat:

base64(secret)

as secure. Confidentiality requires an appropriate cryptographic design, not a different textual spelling of the same bytes.

Hex and binary identifiers

Cryptographic hashes commonly display as hex because two hexadecimal characters represent one byte and the result is easy to log or compare visually:

const digest = Buffer.from([0xde, 0xad, 0xbe, 0xef]);

console.log(digest.toString('hex'));
// deadbeef

The hex string is a representation of the digest; it is not a different digest. For random tokens, use cryptographically secure random APIs rather than Math.random, whose output is not suitable for security-sensitive identifiers.

Buffer slicing/views

Buffers inherit typed-array behaviors. In particular, some slicing and subarray operations create a view into existing storage instead of allocating new storage.

Be careful: some slice/subarray operations can share underlying memory.

const original = Buffer.from([1, 2, 3, 4]);

const view = original.subarray(1, 3);

view[0] = 99;

console.log(original);

Changing view can therefore change original. If you require an independent copy, allocate one explicitly:

const copy = Buffer.from(view);

Know whether your code is passing a view or an owned copy when handling mutable binary data. This distinction matters when a buffer is handed to another function that may retain or modify it.

Binary parsing

Binary formats describe fields at byte offsets. A parser must verify that the bytes for a field actually exist before attempting to read them.

Never read fields beyond bounds.

function readUInt32(buffer, offset) {
  if (offset + 4 > buffer.length) {
    throw new RangeError('Not enough bytes');
  }

  return buffer.readUInt32BE(offset);
}

This check is part of the parser's input validation, not an optional defensive extra. Untrusted binary parsers need strict length validation for every field and should validate formats and relationships between fields before using the parsed values.

Streams

A stream processes data over time, in chunks, instead of requiring the complete value to be present in memory before processing begins. That model is useful for I/O because the producer and consumer can make progress concurrently.

Core categories:

Readable

A readable stream is a source of data:

file read
HTTP request body
process.stdin

Writable

A writable stream is a destination for data:

file write
HTTP response
process.stdout

Duplex

A duplex stream is both readable and writable. Its input and output sides can represent related but independently flowing data:

TCP socket

Transform

A transform stream is a duplex stream whose output is derived from its input:

gzip
cipher
line parser

The useful distinction is direction and responsibility: readable streams produce, writable streams consume, and transform streams connect the two while changing the data.

Why streams matter

For a huge file, reading the whole value first creates an avoidable memory peak:

const data = await readFile('./huge.log');

await upload(data);

The entire file exists in memory before the upload can consume it. With a stream, the file can move through the application in chunks:

disk → chunks → network

This keeps memory bounded by the stream buffers and the chunks currently being processed, rather than by the total file size. Streaming does not make the data free; it changes when and how much data must be resident at once.

Readable stream events

One way to consume a readable stream is event mode:

import { createReadStream } from 'node:fs';

const stream = createReadStream('./notes.txt', {
  encoding: 'utf8',
});

stream.on('data', (chunk) => {
  console.log(chunk);
});

stream.on('end', () => {
  console.log('done');
});

stream.on('error', (error) => {
  console.error(error);
});

The data handler receives chunks, not necessarily complete application-level records. The end event means the readable side has no more data, while error must be handled so a failed read does not become an uncaught failure.

Event mode is useful to understand, but modern async iteration can be clearer when the consumer is already written as an async function.

Async iteration

const stream = createReadStream('./notes.txt', {
  encoding: 'utf8',
});

for await (const chunk of stream) {
  console.log(chunk);
}

Async iteration expresses consumption as a loop that pauses while the next chunk becomes available. It works naturally with async functions and makes the order of asynchronous work easier to follow. Stream errors still need to be handled by the surrounding async control flow.

Writable streams

process.stdout is a writable stream, so a small write looks like this:

process.stdout.write('hello\n');

For a writable stream:

const canContinue = writable.write(chunk);

The return value is a flow-control signal. If it returns false, the internal writable buffer has reached its high-water threshold. Wait for the drain event before producing more data.

This producer-consumer coordination is backpressure. Ignoring the return value can allow an upstream producer to keep allocating data while the destination is still trying to catch up.

Backpressure

The failure mode is easiest to see as a chain:

producer faster than consumer
→ buffer grows
→ memory grows
→ latency grows
→ process can fail

Backpressure lets the consumer signal:

slow down

Streams build this signal into their flow-control model. A well-connected pipeline allows the slowest stage to affect upstream production instead of accumulating unlimited work in memory.

Manual backpressure example

If code must write chunks manually, check the return value and wait for drain:

import { once } from 'node:events';

async function writeAll(writable, chunks) {
  for (const chunk of chunks) {
    if (!writable.write(chunk)) {
      await once(writable, 'drain');
    }
  }

  writable.end();
}

The loop does not assume that every write can be accepted immediately. It pauses only when the writable asks it to, then finishes the stream with end() after all chunks have been submitted.

Most application code should use pipeline instead of manually wiring all of these lifecycle details.

`pipeline`

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('./access.log'),
  createGzip(),
  createWriteStream('./access.log.gz'),
);

pipeline:

connects streams;
propagates errors;
coordinates closure;
respects backpressure.

It is the preferred high-level composition primitive for many stream pipelines because it keeps the normal data path concise while handling the connections between stages. Its returned promise gives the caller one completion or failure point.

Transform stream

This transform converts each incoming text chunk to uppercase:

import { Transform } from 'node:stream';

const upper = new Transform({
  decodeStrings: false,

  transform(chunk, encoding, callback) {
    callback(null, chunk.toUpperCase());
  },
});

Then it can sit between standard input and standard output:

await pipeline(
  process.stdin,
  upper,
  process.stdout,
);

For line-oriented text, remember that chunks have arbitrary boundaries. One chunk is not guaranteed to equal one line, so a transform that needs complete lines must retain partial input and emit records only when the framing delimiter has arrived.

Chunk boundaries

Network and file streams can split data at any byte boundary. The logical input:

hello\nworld\n

may arrive as:

"hel"
"lo\nwo"
"rld\n"

The chunks are transport-level pieces, not protocol messages. Do not parse line or protocol messages on the assumption that each data chunk contains exactly one message.

Maintain a framing buffer or use a parser. The parser owns the incomplete tail, combines it with the next chunk, and emits complete records while retaining anything that is still incomplete.

StringDecoder

There is an additional boundary issue for text. A UTF-8 character can occupy multiple bytes, and a stream can split those bytes across two chunks.

When decoding streaming multi-byte text manually, a UTF-8 character can be split across chunks. Node's string-decoder utilities and stream encoding support can preserve incomplete character bytes correctly until the remaining bytes arrive.

Prefer setting stream encoding or using proper text-decoding abstractions rather than calling chunk.toString() independently on arbitrary split bytes when correctness matters. Otherwise, an incomplete character can be decoded incorrectly before the next chunk supplies its remaining bytes.

Object mode

Streams can process JavaScript objects instead of byte strings or Buffers:

new Transform({
  objectMode: true,

  transform(task, encoding, callback) {
    callback(null, {
      ...task,
      title: task.title.trim(),
    });
  },
});

Object-mode highWaterMark represents an object count rather than a byte size. That changes the unit of the buffering threshold; it does not make the objects cost-free, so large objects can still create significant memory pressure.

Do not use streams merely because you have an array of 20 objects. Use object-mode streams where incremental flow, composition, or asynchronous production and consumption actually matter.

Stream errors

Every stage in a stream pipeline can fail:

source read error
transform parse error
destination closed
client disconnect
disk full

With the promises API, pipeline() rejects when the composed operation fails.

try {
  await pipeline(...);
} catch (error) {
  logger.error({ error }, 'pipeline failed');
}

The caller should decide what failure means for the surrounding operation: log it, clean up partial output, report an appropriate failure, or retry when that is safe. Do not continue sending HTTP success after the pipeline failed.

Cancellation

Modern stream and pipeline APIs can work with AbortSignal in supported forms. A controller gives the application a way to stop work when a client disconnects, a timeout expires, or an operator cancels the operation.

Example concept:

const controller = new AbortController();

await pipeline(source, transform, destination, {
  signal: controller.signal,
});

Verify exact signatures for your supported Node version. APIs and overloads can differ across Node releases, so do not assume that an example written for one version applies unchanged to another.

Cancellation should tear down all involved resources. Stopping only the visible consumer while leaving a file descriptor, network connection, or transform running defeats the purpose of cancellation.

HTTP request bodies are streams

An incoming native Node HTTP request is readable. Its body can be consumed incrementally:

for await (const chunk of request) {
  ...
}

Never accumulate an unbounded request body:

let body = '';

for await (const chunk of req) {
  body += chunk;
}

without a maximum byte limit. The concatenation can continue until the process exhausts memory. Attackers can exploit this by sending a request that never reaches an acceptable size or by sending an extremely large body.

Framework JSON parsers should also have explicit body limits. A framework does not remove the need to define an application-appropriate maximum, and the limit should be measured in bytes where that is what the transport and memory budget require.

Streaming HTTP responses

For a large export, the server can begin sending output as it becomes available:

await pipeline(
  createReportStream(),
  response,
);

This can begin the response earlier and avoid buffering the full report. It also means the response lifecycle must be designed around partial output.

After headers or body data have started, error mapping becomes harder because the HTTP status may already be committed. Design the stream failure behavior: decide how to terminate the response, how to log the failure, and whether the client can distinguish an incomplete result from a complete one.

File upload

A multipart parser can keep an upload incremental and apply policy as bytes arrive:

HTTP body
→ multipart parser
→ size/type policy
→ virus scanner / object storage

Avoid this design for large inputs:

HTTP body
→ entire Buffer in RAM
→ then upload

Streaming does not eliminate validation. It gives the application a place to enforce size and type rules before accepting more data or forwarding it downstream. Avoid buffering the entire input when a stream can safely connect the parser, policy checks, scanner, and storage.

Web Streams

Modern Node includes web-compatible stream types:

ReadableStream
WritableStream
TransformStream

It also provides interoperability utilities for connecting Web Streams and Node streams. fetch() response bodies are Web Streams.

Example:

const response = await fetch(url);

for await (const chunk of response.body) {
  ...
}

Node provides bridging APIs where you need to connect the Web and Node stream ecosystems. Before composing a pipeline, know which stream type a library expects; similar names do not guarantee identical methods, events, or lifecycle behavior.

High water mark

Streams buffer up to configured thresholds. A high-water mark is a flow-control threshold, not a promise that the stream can never temporarily hold more data.

Changing highWaterMark affects:

memory;
throughput;
syscall frequency;
latency.

Do not tune it without measurement. A larger threshold may reduce some call overhead while increasing memory use and latency; a smaller one may reduce buffering while increasing coordination overhead. Defaults are usually a good starting point.

`stream.finished`

Use stream.finished to await stream completion when you are not using a full pipeline. It is useful when a single stream's lifecycle must be observed independently or when the surrounding code is responsible for composing the operation.

Prefer the promises helper where appropriate so completion and failure fit naturally into async control flow.

Compression

Node's zlib module provides compression transforms such as createGzip:

import { createGzip } from 'node:zlib';

Compression is CPU work, and some zlib operations use the libuv thread pool. At high throughput, compression can become CPU or thread-pool pressure even when the underlying disk and network are fast.

A reverse proxy or CDN may be a better owner for compression when it can apply compression centrally and more efficiently. That is an architectural choice to measure, not an automatic rule: consider where the bytes are generated, who owns caching, and where CPU capacity is available.

Security

Compression bombs

A tiny compressed file can expand massively when decompressed. Set limits on decompressed size when processing untrusted archives or compressed content, and stop processing when those limits are exceeded.

Zip Slip

Archive extraction paths can contain traversal entries. Use secure extraction libraries and policies that validate the resolved destination path before writing files.

Binary parsers

Validate lengths and formats before reading or acting on parsed data. A malformed input must not be allowed to turn an offset or size field into an out-of-bounds read or an excessive allocation.

Secret buffers

Sensitive Buffer contents can remain in process memory. Avoid unnecessary copies and logging. Passing a secret through multiple transformations can leave more copies to account for, and logs are especially difficult to retract once emitted.

Failure clinic

`Buffer.allocUnsafe` exposed

Data leak. Uninitialized memory must never be returned to a client, written to a file, or included in a log.

Treating chunk as message

Protocol corruption. Reconstruct the protocol's framing across chunks instead of trusting transport boundaries.

Ignoring writable `false`

Memory growth. Stop producing temporarily and resume after drain or use a pipeline that manages backpressure.

`readFile` on huge upload

OOM. Stream the upload or file through the processing path rather than creating one full in-memory value.

No body size limit

DoS risk. Enforce a maximum body size before an attacker can use the request to exhaust memory or downstream resources.

Pipe without error strategy

Partially written output and uncaught failures. Use pipeline or explicitly define error, cleanup, and partial-output behavior.

Worked project: streaming log compressor

This small project composes a file reader, gzip transform, and file writer. The promise rejects if one of the stages fails:

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

async function compress(input, output) {
  await pipeline(
    createReadStream(input),
    createGzip(),
    createWriteStream(output),
  );
}

try {
  await compress(
    process.argv[2],
    process.argv[3],
  );
} catch (error) {
  console.error(error.message);
  process.exitCode = 1;
}

The basic version is intentionally focused on the data path. Extend it with:

validation;
progress;
cancellation;
temp-file + rename;
tests.

Each extension exercises a production concern: reject invalid paths or arguments, observe progress without defeating streaming, stop cleanly, avoid presenting a partial destination as complete, and verify both successful and failed pipelines.

Exercises

Compare string length and UTF-8 byte length.
Encode/decode hex and base64.
Demonstrate a shared Buffer view.
Read a 1 GB file with stream and measure memory.
Build stdin uppercase Transform.
Write manual backpressure handling.
Replace it with pipeline.
Parse newline-delimited JSON correctly across chunk boundaries.
Add AbortSignal cancellation.
Design safe streaming upload limits.

Mastery checklist

Explain:

Buffer;
encoding;
byte length;
stream categories;
chunks;
backpressure;
highWaterMark;
pipeline;
object mode;
cancellation;
Node versus Web Streams;
streaming security.