131: Node Debugging, Profiling, Memory Leaks, `--inspect`, `perf_hooks`, and APM

Learning objectives

By the end of this lesson, you should be able to:

use Node's inspector;
debug with breakpoints;
understand heap versus RSS at a practical level;
identify common memory leaks;
take heap snapshots carefully;
use CPU profiles;
measure event-loop delay/utilization;
use perf_hooks;
understand diagnostic reports;
understand logging, metrics, traces, and APM;
debug slow Node services systematically.

The goal is not to memorize a list of tools. It is to learn how to move from a symptom to evidence, and from evidence to the most likely boundary where the problem lives.

Debugging starts with a hypothesis

When a Node service is slow or unhealthy, random changes make the investigation harder. Start with a hypothesis about the kind of work that is failing, then collect evidence that can support or reject it.

Ask whether the problem is caused by:

Is it CPU?
event-loop blocking?
memory?
database?
upstream network?
too many requests?
connection pool?
GC?
stream buffering?
worker queue?

These possibilities overlap, but they do not lead to the same fix. A CPU profile helps with a hot JavaScript function; it will not explain a database query waiting on an index. An event-loop measurement can show that callbacks are being delayed, but the next step is still to determine whether the cause is synchronous work, garbage collection, or overload.

Collect evidence before changing the implementation. Useful evidence includes latency percentiles, CPU, memory, event-loop behavior, database and upstream timings, queue depth, and the version or deployment that introduced the change.

`--inspect`

Start a Node process with the inspector enabled:

node --inspect src/server.js

You can then connect Chrome DevTools or an IDE debugger to the process. This gives you access to breakpoints, stack traces, variables, profiles, and other runtime diagnostics.

To pause execution before the application reaches its first line, use:

node --inspect-brk src/server.js

That mode is useful when startup behavior matters or when you need to inspect initialization before requests arrive.

The inspector is not a harmless read-only status endpoint. It can execute code and control the process. Do not expose the inspector port publicly. For production diagnostics, bind it securely and, ideally, enable it only through a controlled, temporary procedure rather than leaving it available to the network.

Breakpoints

Prefer source breakpoints in your debugger. At a breakpoint, inspect the local variables, the closure, the call stack, the async stack, and the request data that led to the current execution.

local variables;
closure;
call stack;
async stack;
request data.

The async stack is particularly useful in server code because the function handling a request may have been reached through several promise or timer boundaries. Source maps also matter when the code running in Node is transpiled or bundled.

Do not accidentally leave a debugger statement in a production hot path:

debugger;

Even if it is harmless when no debugger is attached, it is an unexpected pause point when an inspector connection is present. Treat debugging changes as operational changes and remove or gate them before deployment.

Stack trace

A stack trace is more useful when you know which parts of the execution it represents. Keep these boundaries in mind:

sync call stack
async boundaries
source maps
cause chain

The synchronous call stack shows the currently nested calls. Async boundaries show that work was resumed later, rather than being one uninterrupted call chain. Source maps map generated code back to the source a developer recognizes. A cause chain can preserve the lower-level error that explains why a higher-level operation failed.

Enable and use source-map support appropriately for transpiled code. Otherwise, a correct stack may still point at generated files and make the actual failure unnecessarily difficult to locate.

Process memory

Start with the process-level view:

console.log(process.memoryUsage());

The fields describe different parts of the process. Confusing them is a common source of incorrect leak diagnoses.

Common:

`heapUsed`

heapUsed is the amount of V8-managed JavaScript heap currently in use. It reflects ordinary JavaScript objects that V8 is tracking, not every byte held by the process.

`heapTotal`

heapTotal is the V8 heap that has been allocated for use. It can be larger than heapUsed, because V8 may retain capacity for future allocations.

Read these values as a trend rather than treating one sample as proof of a leak. A process under load may legitimately allocate more heap and later return to a stable post-GC baseline.

`rss`

rss, or resident set size, is the resident memory for the process. It includes more than the JavaScript heap, including:

V8;
native code;
stacks;
Buffers;
mapped memory.

`external`

external is memory associated with C++ or other external objects, including Buffers in many cases.

The useful diagnostic distinction is this: RSS can grow while heapUsed remains stable. That pattern may point to native or Buffer behavior rather than ordinary retention of JavaScript objects. It is a clue, not a complete diagnosis, so follow it with more targeted runtime or library-level investigation.

Garbage collection

V8 automatically collects objects that are no longer reachable. Garbage collection does not, however, know whether an object is still useful to your application. If a reference remains reachable, V8 must treat the object as live.

A memory leak means that objects or resources remain reachable, or that native resources remain allocated, for longer than intended.

Common examples include:

a global Map that never deletes entries;
an EventEmitter listener leak;
an unbounded cache;
timers that are never cleared;
request closures that are retained;
Buffers accumulating in a queue;
sockets that are not closed;
worker or job references that are never released;
a database cursor or session that is never ended.

The same symptom can have different causes. A steadily rising JavaScript heap suggests retained objects, while stable heap with growing RSS suggests that you should also investigate external memory and native allocations.

Classic leak

This route stores every loaded user in a process-wide cache:

const cache = new Map();

app.get('/user/:id', async (req, res) => {
  const user = await loadUser(req.params.id);

  cache.set(req.params.id, user);

  res.json(user);
});

If IDs are unbounded, the Map grows forever. The entries remain reachable through the global cache, so garbage collection cannot reclaim them. A process restart may temporarily hide the symptom, but it does not fix the retention policy.

Possible fixes include:

no cache;
a bounded LRU;
a TTL;
an external cache;
correct invalidation.

A cache is controlled memory retention by design. That makes it useful, but it does not make it exempt from memory limits. Decide how many entries or bytes it may hold, how long entries may live, and when stale data must be invalidated.

Listener leak

This handler adds a listener for each request:

app.get('/events', (req, res) => {
  bus.on('update', handler);
});

If the listener is never removed after the client disconnects, the bus retains the handler and anything captured by its closure. Repeated connections can therefore retain request-specific state and also produce duplicate work for future updates.

Add cleanup when the request closes:

req.on('close', () => {
  bus.off('update', handler);
});

Alternatively, use an abortable listener API where the API and Node version support it. The key requirement is the same: the subscription lifetime must be bounded by the client or operation that created it.

Heap snapshots

Take heap snapshots through the inspector or other diagnostic tooling. A snapshot is powerful because it can show retaining paths, but it is also an intrusive and sensitive operation. It can:

pause the process;
use significant memory;
contain sensitive strings, tokens, or user data.

Do not casually capture production heaps. Prefer a replica or staging environment, or follow a controlled production procedure with an understood impact, access controls, and secure handling of the resulting artifact.

To investigate a suspected leak, compare snapshots at meaningful points:

before load
after repeated load
after GC/idle

The post-GC or idle snapshot is important because a normal allocation burst may disappear once the garbage collector runs. Inspect retaining paths to determine which object or collection is keeping the suspected data alive.

CPU profiling

When CPU is high, use a sample profile and inspect the functions consuming the samples. Measure the relevant algorithm and look for work such as JSON processing, regular expressions, compression, or cryptography. Also compare native work with JavaScript work; the profile may point outside application-level functions.

sample profile;
inspect hot functions;
measure algorithm;
identify JSON/regex/compression/crypto;
compare native vs JS.

Do not optimize a function merely because it looks familiar or easy to change. If it is absent from the profile, it may not be contributing meaningfully to the observed CPU problem.

Event-loop delay

Node can monitor how long callbacks wait before the event loop services them:

import {
  monitorEventLoopDelay,
} from 'node:perf_hooks';

const histogram = monitorEventLoopDelay({
  resolution: 20,
});

histogram.enable();

setInterval(() => {
  console.log({
    meanMs: histogram.mean / 1e6,
    maxMs: histogram.max / 1e6,
  });

  histogram.reset();
}, 10_000);

The histogram values are in nanoseconds, which is why the example divides by 1e6 to report milliseconds. Resetting the histogram after each reporting interval makes each output describe a new window rather than the entire process lifetime.

High event-loop delay means the loop cannot service callbacks promptly. Possible causes include:

CPU-blocking JavaScript;
synchronous I/O;
large garbage-collection pauses;
overload.

This measurement tells you that callbacks are delayed; it does not, by itself, identify which cause is responsible. Pair it with CPU, GC, request, and workload measurements.

Event-loop utilization

performance.eventLoopUtilization() can help estimate how busy the event loop is. Use trends rather than a single sample, because one reading has little context about the interval it represents.

Interpret utilization alongside CPU and latency. High utilization with high latency can indicate saturation or blocking work. Low utilization while a request remains slow points you toward waiting elsewhere, such as a database, upstream service, pool, or queue.

Keep the measurement interval consistent when comparing deployments or incidents. A utilization value without a time window is difficult to compare meaningfully.

Performance marks/measures

For timing stages inside one process, add performance marks and measure the interval between them:

import { performance } from 'node:perf_hooks';

performance.mark('start');

await doWork();

performance.mark('end');

performance.measure('work', 'start', 'end');

console.log(
  performance.getEntriesByName('work'),
);

This is useful for local application stages, such as separating validation, service logic, and repository work. For a distributed system, local marks do not automatically connect work across processes. Distributed tracing is stronger when you need to follow one request through several services.

Histograms

Node performance APIs can build histograms for durations. In a production metrics system, record the stages that help distinguish where time is being spent:

request duration
DB duration
upstream duration
queue wait
worker duration

Use percentiles to understand the distribution:

p50
p95
p99

An average alone hides tail latency. A service can have a reasonable average while a meaningful portion of requests waits several seconds, so the percentile view is essential for diagnosing user-visible slowness.

Diagnostic reports

Node can produce diagnostic reports containing runtime and process information that is useful during crashes and performance incidents.

Reports may include sensitive environment and process details. Protect access to them, store them securely, and treat them as operational data rather than harmless logs.

Use the current Node documentation for the exact flags, signals, and API behavior. These details can vary by Node release, and the current documentation is the right source when designing an incident procedure.

Core dumps

Advanced crash analysis can use core dumps and native tooling. This is specialized operational debugging, generally used when ordinary logs and JavaScript-level diagnostics cannot explain a native crash or low-level failure.

Do not enable core dumps without planning for security and storage. A dump can be large and may contain process memory, including sensitive data.

Structured logs

Structured logs keep fields queryable instead of burying all context in a formatted sentence. A useful error record might include:

{
  "level": "error",
  "time": "...",
  "requestId": "...",
  "route": "PATCH /tasks/:id",
  "durationMs": 123,
  "errorCode": "DB_TIMEOUT"
}

Do not log the whole request body by default. Request bodies may contain credentials, tokens, personal data, or other sensitive values. Log the minimum context needed to investigate, and apply the application's redaction and retention rules.

Metrics

Useful service metrics include:

request rate
error rate
latency
event-loop delay
CPU
RSS/heap
GC
DB pool
upstream failures
worker queue depth
active connections

Use labels carefully. Do not use high-cardinality labels such as raw userId or requestId in ordinary metrics systems. Those values can create an unbounded number of time series, making the metrics expensive and less useful. Keep request-level identifiers in logs or traces, where they can still correlate one operation without exploding metric cardinality.

Tracing

A distributed trace follows an operation across its boundaries:

browser/API request
→ Node route
→ service
→ Mongo
→ external API

Spans capture timing and context for each part of that path. They help answer whether a slow request spent its time in the Node handler, the database, or an external dependency.

OpenTelemetry is a common standard for collecting and propagating this context. Do not manually invent incompatible trace IDs when ecosystem tooling can propagate standard context. Interoperability is part of the value of using a shared tracing model.

AsyncLocalStorage and traces

Tracing and logging systems often use async context to propagate the current span or request ID through asynchronous work. This lets code deeper in the call chain associate its logs with the originating request without passing an identifier through every function signature.

Avoid mutating a shared global "current request" variable. Asynchronous execution interleaves users, so a global value can be overwritten by another request and cause logs or trace data to be attributed to the wrong user.

APM

Application Performance Monitoring tools can provide:

automatic HTTP traces;
database timing;
error aggregation;
service maps;
continuous profiling.

Evaluate their overhead, privacy implications, and sampling behavior. Automatic instrumentation is convenient, but it may add work, collect data you did not intend to export, or sample away the very tail requests you need to investigate.

Do not depend on an APM product without understanding basic runtime metrics. You should still know how CPU, RSS, heap, event-loop delay, latency, and dependency timing behave so that an APM view can be checked against the process itself.

Slow-request investigation

Suppose p95 latency jumps from 100ms to 3s. Treat that as an investigation, not as evidence that one particular subsystem is broken. Check the major sources of saturation and waiting in an order that narrows the search:

request rate;
CPU;
event-loop delay;
Mongo/upstream spans;
pool saturation;
worker queue;
GC/memory;
deployment/version change.

If event-loop delay is low but the database span is 2.8s, investigate Mongo query diagnostics. Moving the work to worker_threads would not address time spent waiting for the database. The measured boundary should determine the next tool and the next fix.

Load testing

Use a controlled load generator rather than an improvised burst of traffic. Measure:

throughput;
latency percentiles;
errors;
saturation.

Do not load-test production irresponsibly. A test can exhaust connection pools, fill queues, overload dependencies, and affect real users. Use an environment and workload plan that account for those effects.

Warm-up, realistic payloads, keep-alive behavior, authentication, and think time all affect the result. A test that omits them may measure a system that behaves very differently from the real service.

Memory load test

For memory leaks, run a sustained workload rather than a 30-second test. Some leaks are gradual and may need hours before the trend is visible.

Track heap after garbage collection or during steady periods. A sawtooth heap graph is normal when V8 allocates and then collects. The important question is whether the post-collection baseline remains bounded. If each cycle returns to a higher baseline, retained objects are more likely.

Compare the same workload and observation window when you repeat the test. Otherwise, a different request mix can look like a memory regression even when the retention behavior is unchanged.

Logging performance

Logging is work performed by the service, not a free observation layer. Synchronous logging or very large JSON serialization can block the event loop.

Use an appropriate structured logger and output strategy. Even container stdout can become a bottleneck when the application emits enormous volumes, because the output pipeline may block or consume substantial CPU and I/O capacity.

Source maps and error monitoring

Deploy source maps to the monitoring service or to private storage so errors can be mapped back to the source developers maintain. Do not necessarily serve .map files publicly; source maps can reveal implementation details and source content.

Ensure the release version identifies the correct map. A source map from a different build can produce a plausible-looking but incorrect stack location, which is especially dangerous during an incident.

Keep the mapping process part of the release procedure, not an improvised step after an outage begins. The error-monitoring record and the deployed artifact should identify the same release.

Failure clinic

These are common ways to make a debugging or observability situation worse:

inspector open to the internet;
average latency only;
heap snapshot taken on a tiny-memory production pod;
cache without a limit;
high-cardinality metrics;
logging secrets;
CPU optimization when the database is the bottleneck;
profiling dev mode only;
no baseline before a change.

Each failure removes either safety or useful comparison. Secure diagnostic access, capture distributions and baselines, and test under conditions that resemble the system you are trying to understand.

Exercises

Debug a route with --inspect.
Create an intentional Map leak and inspect the memory trend.
Create an EventEmitter listener leak and fix it.
Measure event-loop delay during a CPU loop.
Add performance marks around a service and repository.
Build a request-duration histogram.
Add structured request-ID logging.
Sketch OpenTelemetry spans.
Run a load test and identify the first saturation point.
Write an incident checklist for high p99 latency.

Mastery checklist

Explain the following in your own words and connect each item to an investigation:

inspector;
heap/RSS/external;
GC/leaks;
heap snapshot;
CPU profile;
event-loop delay/utilization;
perf hooks;
diagnostic reports;
logs/metrics/traces;
APM;
evidence-driven debugging.