132: Production Node Operations — Configuration, Graceful Shutdown, Health, Logging, Process Managers, Containers, and Reliability

Learning objectives

You will learn to:

design a production startup/shutdown lifecycle;
validate configuration before the process begins listening;
order dependencies during startup;
handle SIGTERM/SIGINT;
stop accepting new traffic;
close HTTP, database, and worker resources;
use readiness and liveness health checks;
understand process supervisors such as systemd, PM2, and container orchestrators;
understand stateless horizontal scaling;
design timeout, retry, and circuit-breaker boundaries;
understand logging and deployment releases;
avoid common graceful-shutdown and retry-storm mistakes.

The recurring theme is explicit ownership. A production process needs to know when it is ready, how it becomes unavailable, which resources it owns, and who is responsible for restarting it when it exits.

Production lifecycle

A service has distinct phases. Treating them as one undifferentiated node server.js step makes failures harder to diagnose and shutdown behavior harder to reason about.

boot
↓
validate config
↓
connect dependencies
↓
listen
↓
serve
↓
draining
↓
close dependencies
↓
exit

Make these phases explicit. In particular, listening is the boundary at which the service is telling the rest of the system, "I can receive work." That boundary should not be crossed before the required dependencies are usable.

Startup

This version creates a server before its database connection is ready:

app.listen(3000);

connectDatabase().catch(console.error);

The server can accept traffic while the database is still connecting, or after the connection has failed. Requests then fail in a noisy and inconsistent way instead of the process failing before it advertises readiness.

A better arrangement is to complete required startup work before calling listen:

async function start() {
  const config = loadConfig();

  const database = await connectDatabase(config.databaseUrl);

  const app = createApp({
    database,
    config,
  });

  const server = app.listen(config.port);

  return {
    server,
    database,
  };
}

try {
  const runtime = await start();
  installShutdown(runtime);
} catch (error) {
  logger.fatal({ error }, 'startup failed');
  process.exitCode = 1;
}

Here configuration is loaded first, the required database is connected second, and only then is the application made available to callers. Returning the owned resources gives shutdown code a clear inventory of what it must close.

Fail fast before serving. A startup failure should be visible to the process supervisor as a failed process, not hidden behind a server that accepts requests it cannot handle.

Dependency readiness

Not every dependency has the same role. If an optional dependency fails, decide explicitly whether the service should:

fail startup;
degrade the related feature;
retry.

The right choice depends on what the application can honestly serve without that dependency. Define the policy rather than letting an uncaught promise, a lazy connection, or a route-level error decide it accidentally.

For a primary database required by every route, accepting traffic without it often creates more noise than value. If the service can only return errors, it is usually better to remain unready or fail startup and let the supervisor handle the failure.

Graceful shutdown

When a process receives SIGTERM, the goal is not simply to terminate more slowly. The goal is to stop taking new work, finish a bounded amount of work already in progress, and release resources in an order that does not create new failures.

mark not ready
↓
stop accepting new requests
↓
allow bounded in-flight completion
↓
stop background polling/jobs
↓
close database/pools
↓
flush critical telemetry/logs
↓
exit

Use a deadline. A request, worker, or dependency can hang, and a graceful shutdown that waits forever prevents the supervisor from replacing the instance. Never wait forever.

Re-entrant shutdown

Signals and errors can arrive twice: for example, a second signal may be sent while the first cleanup is still running. Shutdown must therefore be re-entrant-safe.

let shuttingDown = false;

async function shutdown(reason) {
  if (shuttingDown) return;

  shuttingDown = true;

  ...
}

The guard ensures that two shutdown paths do not close the same server or pool concurrently. Protecting this lifecycle state is small, but it prevents cleanup races during the exact period when the process is already under stress.

Native server close

Node's native HTTP server exposes a close operation:

server.close((error) => {
  ...
});

Modern Node includes additional connection-closing helpers in supported versions. The exact helpers available depend on the Node version, so check the runtime documentation when you need to close idle or active connections more aggressively.

Understand the different connection states involved:

new connections;
idle keep-alive connections;
active requests.

Do not assume close() instantly ends every socket. In-flight requests may still need time to complete, and keep-alive behavior can make an apparently idle connection remain open. A deadline and, where appropriate, supported force-closing helpers provide the final safety boundary.

Shutdown deadline

A shutdown timer prevents one stuck resource from holding the process indefinitely:

const shutdownTimer = setTimeout(() => {
  logger.fatal('forced shutdown deadline exceeded');
  process.exit(1);
}, 30_000);

shutdownTimer.unref();

Normal cleanup should clear the timer. unref() prevents the timer alone from keeping the event loop alive while the process is otherwise finished.

Use force-exit as a last resort. It is a safety net for a deadline that has been exceeded, not a replacement for closing servers, pools, workers, and telemetry cleanly.

Abort background work

Long-running background work needs the same lifecycle signal as the HTTP server. A single AbortController gives pollers, fetch calls, jobs, stream pipelines, and worker pools a common shutdown notification:

const lifecycle = new AbortController();

process.once('SIGTERM', () => {
  lifecycle.abort(new Error('shutdown'));
});

Pass the signal to:

polling loops;
fetch;
jobs;
stream pipelines;
worker pools.

The signal does not magically cancel arbitrary code. Each operation must accept and honor it, and cleanup code should distinguish an expected shutdown abort from an unexpected operational failure.

Polling loop

An abort-aware poller can stop both the work and the delay between iterations:

async function runPoller(signal) {
  while (!signal.aborted) {
    await pollOnce({ signal });

    await setTimeout(1000, undefined, {
      signal,
    });
  }
}

Use abort-aware timers where supported. Without that, a worker may finish its current operation but remain asleep until the interval expires, or continue starting another operation after shutdown has begun.

Do not create unbreakable while(true) workers. Every loop that owns resources or performs I/O needs a clear stop condition.

Liveness versus readiness

These checks answer different operational questions. Confusing them can turn a recoverable dependency outage into a restart storm.

Liveness

Is the process alive enough that the supervisor should not restart it?

Liveness is usually simple. It should normally establish that the process is running and able to respond to the probe. It should not report every temporary downstream problem as a dead process.

Readiness

Should this instance receive traffic now?

Readiness can become false during:

startup;
shutdown or drain;
a critical dependency outage, if the application cannot serve without that dependency.

The useful distinction is that an unready instance is removed from traffic, while a non-live instance is a candidate for restart. Do not make liveness depend on a temporary database outage and cause restart storms.

Health endpoints

A service may expose separate endpoints such as:

/live
/ready

Keep the response useful to the platform while protecting operational detail. Public health responses should not expose:

database password
internal hosts
stack traces
deployment secrets

Detailed diagnostics belong behind appropriate access controls or in internal telemetry. Health endpoints are often reachable by more people and systems than the application team initially expects.

Stateless scaling

Horizontal scaling adds instances behind a load balancer:

load balancer
→ Node A
→ Node B
→ Node C

The load balancer can send consecutive requests from one user to different processes. Therefore, application state that must be shared cannot live only in a single process's memory:

new Map()

Examples include:

sessions;
rate limits;
job status;
shared cache.

Use external state, or use a sticky architecture where there is a deliberate reason and the trade-off is understood. In-memory state is still useful for process-local concerns, but it must not silently become the source of truth for behavior that spans replicas.

Process supervisors

Different layers can own process lifecycle:

systemd

systemd is an operating-system service manager. It can start a service, restart it, and integrate it with host-level service management.

PM2

PM2 is a Node-oriented process manager. It provides Node-focused process management features, including multiple processes and log handling.

Docker/Kubernetes

Docker and an orchestrator such as Kubernetes control container lifecycle, restart behavior, placement, and scaling.

Choose clear ownership. Avoid stacking several independent supervisors without a reason:

PM2 cluster inside Docker pod
+
Kubernetes scaling
+
another supervisor

One process per container is common, but it is not an absolute rule. The important point is that the deployment should have an intentional process model rather than multiple layers competing to manage the same workers.

Restart policy

A crashed process should usually restart through its supervisor. A restart policy is not, however, a substitute for diagnosing why the process crashed.

Constant crash loops need:

backoff;
alerting;
health status.

Do not hide fatal bugs by restarting thousands of times per minute. Backoff gives the system room to recover and gives operators a visible signal that the process cannot start successfully.

`NODE_ENV`

Some libraries use this convention:

NODE_ENV=production

Do not use NODE_ENV as the full deployment configuration. It is a broad environment label, not a safe place to encode every business, regional, logging, or feature decision.

Use explicit settings such as:

APP_ENV
REGION
LOG_LEVEL
FEATURE flags

as needed. Do not make business behavior depend on dozens of implicit environment conditions. Explicit configuration is easier to validate, document, test, and inspect during an incident.

Secrets

Production secrets should come from a controlled runtime source, such as:

an orchestrator secret;
a cloud secret manager;
a vault;
a mounted secret file;
protected environment injection.

Rotate secrets. Rotation is part of operating a secret safely, not an exceptional event that can be postponed indefinitely.

Do not bake secrets into the container image. Images are copied, cached, logged, and retained in registries; removing a secret from the current image does not remove it from layers or old artifacts.

Logging

In containerized environments, write JSON or otherwise structured logs to stdout. The collector should handle transport and aggregation rather than requiring the application to manage log files inside an ephemeral container.

Include fields that let an operator correlate an event with the running service and request:

timestamp
level
service
release
requestId
message
error code

Avoid multiline, unstructured logs when machines need to parse them. Structured output makes filtering and correlation practical, especially when several replicas are producing interleaved events.

Log levels

Typical levels are:

debug
info
warn
error
fatal

The production level is configuration. Keep enough information to diagnose normal operation and failures, while avoiding uncontrolled volume.

Do not disable all useful logs for performance. Do not log every request body: request bodies may contain credentials or personal data, and recording them can create both security and cost problems. Log deliberately selected metadata and redacted error context instead.

Release metadata

Expose non-sensitive release information internally:

service version
git SHA
build ID

This information is for diagnostics. A health or status endpoint may include a non-sensitive version, but it should not expose deployment secrets or unnecessary internal details.

With release metadata, an observation such as:

"error started after deploy abc123"

becomes traceable. Without it, operators have to infer which artifact each replica is running from timestamps and deployment records.

Timeouts

Every network hop should have a deadline. A request can cross several independent boundaries:

client → Node
Node → Mongo
Node → upstream

Without a timeout, a dead or degraded dependency can consume a connection, request slot, worker, or memory indefinitely. Configure timeouts according to the service-level objective and the operation, rather than choosing one number blindly for every hop.

Retry policy

Retries are another form of load. Retry only when all of these conditions are satisfied:

the error is likely transient;
the operation is safe or idempotent;
the overall deadline still has time remaining;
the number of attempts is bounded.

Use exponential backoff with jitter. Backoff spaces out attempts, and jitter prevents many instances that failed at the same time from retrying in lockstep.

Never use an unbounded, full-speed retry loop:

while (true) {
  await fetch(...)
}

An unsafe write may be duplicated by a retry, and even a safe read can amplify an outage if every caller retries without a limit.

Retry storm

When a dependency fails, every instance can retry aggressively at once:

outage
→ more retries
→ dependency overloaded
→ recovery delayed

Use backoff, jitter, circuit breaking, and concurrency limits. The objective is to keep a dependency failure from consuming the capacity of the service that is trying to survive it.

Circuit breaker concept

A circuit breaker changes behavior after repeated failures:

open
→ fail fast temporarily
→ allow probes
→ close on recovery

While open, calls fail quickly instead of repeatedly waiting on a known-bad dependency. Later, limited probes test whether recovery has occurred. A successful recovery closes the breaker and permits normal traffic again.

This can protect the system. Do not add a circuit-breaker library blindly; timeouts and bounded retries come first. A breaker with no meaningful timeout, attempt budget, or capacity limit can merely hide the same underlying problem behind more state.

Bulkhead

Separate limited pools or queues so that one downstream cannot consume every resource in the process. This is the same isolation idea used in a ship's bulkheads: a failure in one compartment should not flood the entire vessel.

For example:

payment calls max 20 concurrent
report jobs max 4 workers

These limits prevent a noisy dependency or expensive workload from taking all available connections, workers, or event-loop-adjacent capacity.

Load shedding

When the service is overloaded, reject work early:

429
503

Returning a controlled rejection is safer than accepting work until queues grow without bound and the process reaches out-of-memory failure. Monitor queue depth and concurrency so load shedding reflects a visible capacity boundary rather than an unexplained collection of errors.

Database pool

Database pool size must be considered across the whole deployment. The number of connections per instance multiplied by the number of instances is the pressure placed on the database.

For example:

20 connections per instance
× 100 pods
= 2000 DB connections

Scale the application and database together. Do not set the maximum pool size for each process without global capacity planning. A setting that looks reasonable for one local process can exhaust the database as soon as autoscaling creates many replicas.

Zero-downtime deploy

A zero-downtime deployment depends on readiness and graceful shutdown working as a pair:

new instance starts
becomes ready
traffic shifts
old instance marked unready
SIGTERM
drain
exit

The new instance must be ready before traffic shifts to it. The old instance must become unready before it drains, so new requests stop arriving while existing requests receive their bounded completion window. Readiness and graceful shutdown are essential to this sequence.

Uncaught errors

An unexpected fatal error means the process state may no longer be trustworthy. The usual response is to:

log the error;
mark the instance unhealthy and drain it if possible;
exit;
let the supervisor restart it.

Do not catch globally and continue forever. Continuing after an unknown fatal condition can serve corrupted state, hide the original failure, and make a supervisor believe the process is healthy when it is not.

Startup migration caution

Running a database schema migration automatically in every replica at startup can cause races and long deploys. Several replicas may attempt the same migration, or all replicas may remain unavailable while a migration runs.

Use an explicit migration job or process when the database type requires migrations. Mongo schema migration strategies are covered later; the operational boundary remains the same: schema changes need an intentional deployment step, not an accidental side effect of every web replica starting.

Node watch and nodemon

For development, Node can restart the process when source files change:

node --watch src/server.js

Nodemon is another development option. Do not use a development watcher as the production process supervisor. Development watchers optimize local feedback; production supervisors need deliberate restart, health, signal, and backoff behavior.

PM2 basics

PM2 can:

restart processes;
manage logs;
run multiple processes;
provide startup scripts.

If the deployment already has Kubernetes or systemd managing lifecycle, evaluate whether PM2 adds value or merely duplicates supervision. Running two systems with overlapping ownership can make signals, process counts, logs, and restart behavior difficult to predict.

Container basics

A production image should:

pin a supported runtime base;
install the lockfile deterministically;
run as non-root where feasible;
copy only needed files;
avoid secrets;
use production dependencies and build artifacts;
define the stop signal and health behavior at the platform layer.

Container security is a broader DevOps topic, but a Node developer still needs to understand the lifecycle. The application must receive the platform's stop signal, expose the health behavior the platform expects, and avoid assuming that local filesystem state or a development process manager exists inside the container.

Common mistakes

Watch for these failure patterns:

listening before dependencies are ready;
checking the database in liveness and causing a restart storm;
omitting a shutdown timeout;
calling process.exit immediately on SIGTERM;
keeping shared state in memory across replicas;
nesting supervisors;
omitting downstream timeouts;
retrying unsafe writes;
multiplying a connection pool across replicas without capacity planning;
baking secrets into the image;
omitting a release ID;
running a development watcher in production.

Each mistake crosses an operational boundary: readiness, ownership, capacity, data safety, or observability. Reviewing those boundaries is more useful than memorizing a particular platform's command names.

Exercises

Write an explicit start() lifecycle.
Add SIGTERM drain with a deadline.
Add /live and /ready.
Make readiness false before server close.
Abort a poller on shutdown.
Calculate database pool capacity for 50 instances.
Design a timeout/retry policy for an upstream API.
Simulate a retry storm and add backoff/jitter.
Draft a Docker production checklist.
Draw a zero-downtime deployment flow.

Work through the exercises in order. The early tasks establish lifecycle ownership; the later tasks apply the same reasoning to capacity, failure isolation, and deployment behavior.

Mastery checklist

Explain:

startup sequencing;
graceful shutdown;
signals and deadlines;
liveness and readiness;
stateless scaling;
supervisors;
secrets;
logging and release metadata;
timeouts and retries;
retry storms and circuit breakers;
pool capacity;
zero-downtime deployment.

If you can explain these boundaries and connect them to observable process behavior, you can reason about more than just whether a Node server starts. You can diagnose why it should receive traffic, how it should leave service, and where a failure is consuming capacity.