124: Native Node HTTP/HTTPS, Fetch, URLs, Headers, Keep-Alive, and Consuming APIs

Learning objectives

By the end of this lesson, you should be able to:

explain the HTTP request/response lifecycle in Node;
create an HTTP server with node:http;
parse URLs and query parameters;
work with request and response headers;
explain HTTP bodies as streams;
set status codes and content types deliberately;
use fetch to consume HTTP APIs;
use AbortSignal and timeouts to cancel work;
describe connection reuse and keep-alive at a practical level;
distinguish transport errors from HTTP errors;
explain redirects, compression, proxies, and TLS at a high level;
build a boundary around an API client.

The goal is not to memorize every method in Node's HTTP modules. It is to build a model that helps you read framework code, choose appropriate status codes, and diagnose whether a failure happened in the network, the HTTP exchange, or your own application logic.

Why native HTTP before Express

Express builds on the same Node HTTP concepts. A framework gives you routing, middleware, parsing, and conventions, but it does not replace the underlying protocol.

If you understand these pieces:

IncomingMessage
ServerResponse
method
URL
headers
body stream
status
socket

then Express is a convenience layer rather than magic. When a route behaves unexpectedly, this lower-level model tells you where to inspect: the incoming request, the response you are constructing, or the socket carrying the exchange.

First server

Here is the smallest useful native HTTP server:

import http from 'node:http';

const server = http.createServer((request, response) => {
  response.statusCode = 200;
  response.setHeader('content-type', 'text/plain; charset=utf-8');
  response.end('Hello\n');
});

server.listen(3000, '127.0.0.1', () => {
  console.log('http://127.0.0.1:3000');
});

The callback runs once for each request. request describes what the client sent, while response is the object you use to choose the status, headers, and body that Node sends back. response.end() completes the response; if you never end it, the client may continue waiting.

This server listens only on the loopback address, so it is reachable from the same machine but not directly from other machines on the network. That is a useful default while experimenting locally.

IncomingMessage

The request object is an IncomingMessage. Among other properties, it includes:

method
url
headers
socket
body stream

You can inspect the basic request metadata like this:

console.log(request.method);
console.log(request.url);
console.log(request.headers);

The body is not a ready-made string or parsed object. It is a readable stream, which means data can arrive in multiple chunks and can arrive later than the first callback invocation. That distinction matters for both correctness and memory use.

Headers and URL data came from the network, so treat them as untrusted input. Validate values before using them for authentication, routing decisions, logging, or constructing another request.

URL parsing

request.url is normally a relative path and query string, such as /tasks?page=2. It does not, by itself, provide a complete absolute URL.

To use the standard URL API, provide a known origin:

const url = new URL(
  request.url,
  `http://${request.headers.host ?? 'localhost'}`,
);

The resulting object separates URL components instead of making you parse punctuation manually. For example:

url.pathname
url.searchParams.get('page')

pathname is the route portion, while searchParams handles query parameters. Query values are strings, so convert and validate them before treating them as numbers, identifiers, or limits.

There is a security boundary here. For security-sensitive origin or host logic behind proxies, do not blindly trust the Host header. Use deployment-aware configuration. A client can send a Host value that is syntactically valid but not an origin your application should trust.

Routing manually

With the parsed URL, a route can be handled directly:

if (request.method === 'GET' && url.pathname === '/health') {
  response.writeHead(200, {
    'content-type': 'application/json; charset=utf-8',
  });

  response.end(JSON.stringify({ ok: true }));
  return;
}

The return prevents later route or fallback code from attempting to write a second response. This style is clear for a few endpoints, but a larger application quickly accumulates method checks, path checks, parameter handling, and error paths. Manual routing becomes cumbersome, which is one reason frameworks exist.

JSON response helper

Repeated response construction is a good place for a small helper:

function sendJson(response, status, body) {
  const text = JSON.stringify(body);

  response.writeHead(status, {
    'content-type': 'application/json; charset=utf-8',
    'content-length': Buffer.byteLength(text),
  });

  response.end(text);
}

Content-Length is measured in bytes, not JavaScript characters. Those measurements differ for non-ASCII text, so do not use string length. Buffer.byteLength(text) reports the number of bytes that will be sent using the string's encoding.

The helper also keeps the status and content type together, making it less likely that one route returns JSON with a missing or inconsistent header.

Reading request body with a limit

For a small JSON request, you can consume the stream and buffer it up to a fixed limit:

async function readJson(request, maxBytes = 1_000_000) {
  const chunks = [];
  let total = 0;

  for await (const chunk of request) {
    total += chunk.length;

    if (total > maxBytes) {
      const error = new Error('Request body too large');
      error.status = 413;
      throw error;
    }

    chunks.push(chunk);
  }

  const text = Buffer.concat(chunks).toString('utf8');

  return text ? JSON.parse(text) : null;
}

This function still buffers the body, but it buffers only up to the configured limit. The limit protects the process from accepting an unexpectedly large payload, and the 413 status communicates that the request is too large. JSON syntax errors are a separate failure and should be mapped by the caller to an appropriate client-error response.

Buffering like this is reasonable for small JSON documents. Large uploads, downloads, and other potentially unbounded data should be processed as streams instead.

Content-Type

Do not parse arbitrary request bodies as JSON solely because an endpoint expects JSON. Check the request's media type first. The basic media type is:

application/json

Requests commonly include parameters:

application/json; charset=utf-8

That means a check for exact string equality may reject a valid request, while a careless check may accept a type you do not support. Real content-type parsing includes parameters, so use robust parsing or framework utilities and reject unsupported media types with an appropriate status such as 415 Unsupported Media Type.

Status codes

Status codes are part of the API contract. Common choices include:

200 OK
201 Created
202 Accepted
204 No Content
400 Bad Request
401 Unauthorized (actually unauthenticated in common API usage)
403 Forbidden
404 Not Found
409 Conflict
413 Content Too Large
415 Unsupported Media Type
422 Unprocessable Content
429 Too Many Requests
500 Internal Server Error
503 Service Unavailable

Choose a status from the semantics of the result. For example, a successfully created resource is different from an accepted asynchronous job, and a malformed request is different from a temporary service outage.

Do not return 200 { success:false } for every failure. Clients, monitoring systems, caches, and intermediaries use the HTTP status to decide how to handle the response. A body can still carry useful application-specific details, but it should not hide the protocol-level result.

Headers

HTTP header names are case-insensitive. Node normalizes how you access them, so these forms are typical:

request.headers.authorization
request.headers['content-type']

Response headers can be set before the response is sent:

response.setHeader('cache-control', 'no-store');

The values still have to be treated carefully. Never reflect arbitrary user-controlled header values into response headers without validation. Header injection or splitting protections are necessary, and the header's semantics must make sense for the response you are producing.

Cookies

A cookie is carried in an HTTP header. Clients send cookies with the Cookie header:

Cookie: session=...

Servers set or update them with Set-Cookie:

Set-Cookie: session=...; HttpOnly; Secure; SameSite=Lax

The attributes affect whether scripts can read the cookie, whether it is sent over an unencrypted connection, and when browsers send it cross-site. Security attributes matter; choose them based on the session's threat model. For production parsing, signing, and session management, use a mature cookie or session library rather than writing an incomplete parser.

HTTPS

The node:https module supports TLS. In production, TLS often terminates before the request reaches Node, at a:

reverse proxy;
load balancer;
CDN;
ingress.

In that arrangement, the Node application may receive plain HTTP from a trusted local proxy even though the client connected to the public service over HTTPS. Know where TLS terminates and how the client's scheme and IP address are forwarded.

Do not trust X-Forwarded-* headers from arbitrary internet clients unless proxy trust is explicitly configured. Otherwise a client may be able to impersonate a secure scheme or a different source address in application logic and logs.

Keep-alive

Opening a new HTTP connection can require TCP setup and, for HTTPS, TLS setup. Connection reuse avoids repeating that work for every request.

Modern Node fetch and HTTP internals manage connection pooling and keep-alive. The practical implication is that you usually should not create a brand-new custom Agent for every outbound request, because doing so can defeat pooling and add connection overhead.

When using a third-party HTTP client, understand its connection-pool lifecycle: know whether a client instance owns a pool, how idle connections are closed, and how shutdown interacts with in-flight requests.

`fetch`

Modern Node provides global fetch, so a basic API call looks like this:

const response = await fetch('https://api.example.com/tasks');

if (!response.ok) {
  throw new Error(`HTTP ${response.status}`);
}

const body = await response.json();

There is a distinction that causes frequent bugs: fetch() rejects on network or transport failures, not on an HTTP 404 or 500. A server can successfully return a response whose status indicates failure. Always check response.ok or inspect the status before treating the result as successful.

After that check, parsing the body is another operation that can fail, for example when the server labels invalid content as JSON. Error handling should account for both the HTTP status and body parsing.

Timeout/cancellation

A timeout should cancel the underlying request, not merely stop your code from waiting for its result. With an AbortController, the timer can signal cancellation:

const controller = new AbortController();

const timer = setTimeout(() => {
  controller.abort(new Error('upstream timeout'));
}, 3000);

try {
  const response = await fetch(url, {
    signal: controller.signal,
  });
} finally {
  clearTimeout(timer);
}

The finally block clears the timer after success, failure, or cancellation, so it does not remain active unnecessarily. Use supported AbortSignal timeout helpers where appropriate. Cancellation is especially important for servers, because an abandoned upstream request can otherwise continue consuming sockets, memory, and upstream capacity after your caller has given up.

Upstream API client

Application services should not each reinvent transport error handling. A small API-client boundary can normalize network failures, HTTP failures, and response parsing:

export class HttpError extends Error {
  constructor(message, { status, body, cause } = {}) {
    super(message, { cause });

    this.name = 'HttpError';
    this.status = status;
    this.body = body;
  }
}

export async function requestJson(
  url,
  {
    signal,
    headers,
    ...options
  } = {},
) {
  let response;

  try {
    response = await fetch(url, {
      ...options,
      signal,
      headers: {
        accept: 'application/json',
        ...headers,
      },
    });
  } catch (cause) {
    throw new HttpError('Upstream request failed', {
      cause,
    });
  }

  const contentType = response.headers.get('content-type') ?? '';

  const body = contentType.includes('application/json')
    ? await response.json()
    : await response.text();

  if (!response.ok) {
    throw new HttpError('Upstream returned an error', {
      status: response.status,
      body,
    });
  }

  return body;
}

The caller now gets one error type for transport and HTTP failures, with a status when the upstream actually returned one and a parsed body when available. Components and services no longer need to repeat the same content-type inspection and status check. In production, decide separately how to handle body-parse failures and whether sensitive upstream bodies should be retained or logged.

Redirects

Fetch follows redirects by default according to its API. That is convenient for ordinary API calls, but it changes the destination of a request, which matters when the original URL came from a user.

For a security-sensitive server-side fetch of user-provided URLs, a redirect can become an SSRF path:

user URL initially public
→ redirect to 169.254.169.254 / internal service

Do not build an arbitrary URL-fetch proxy without SSRF protections. The first URL may pass a superficial allowlist while the redirected destination reaches cloud metadata or another internal service.

SSRF

Server-Side Request Forgery, or SSRF, occurs when an attacker controls a server's outbound target. The server's network position and credentials can then be used to reach destinations the attacker could not reach directly.

Protection may include:

allowlist destinations;
parse the URL;
enforce the scheme;
resolve and validate IP addresses;
block private, link-local, and loopback ranges;
revalidate redirects;
enforce a network egress policy.

A regex like this:

url.startsWith('https://')

is not sufficient. It checks only a string prefix. It does not establish which host will be contacted, whether DNS resolution points to a private address, or where a redirect ultimately leads.

API retries

Retries can recover from transient failures, but they can also repeat side effects. Retry safe or idempotent operations carefully. A potential retry is:

GET

GET is often safe to retry, assuming the endpoint follows its expected semantics. By contrast, POST /payments can create duplicate side effects unless the server provides idempotency semantics and the client uses them correctly.

Use exponential backoff, jitter, and retry budgets. Without those controls, many clients can retry at once during an outage and create a retry storm that makes the original incident worse.

HTTP caching

Response headers can express caching policy and validation information:

Cache-Control
ETag
Last-Modified
Vary

Do not implement cache policy randomly. Private or authenticated responses need especially careful directives so that one user's data is not reused for another user. Conditional requests using validators such as ETag or Last-Modified can let a client or intermediary avoid transferring an unchanged representation, reducing bandwidth while still checking freshness.

Compression

Servers can compress responses with formats such as:

gzip
br

Reverse proxies and CDNs often handle compression more efficiently because they can centralize configuration and reuse compressed representations. Compression is not free: it consumes CPU, and the chosen settings have security and operational implications. Consider response type, size, cache behavior, and the trust relationship between compressed data and secrets.

HTTP streaming

Both requests and responses are streams. The receiver may see data in chunks rather than as one complete value.

Streaming a JSON array is not the same as receiving ordinary complete JSON until the entire array has arrived. A parser that expects one complete document cannot necessarily process each chunk independently.

For streaming structured data, use a format or protocol designed for incremental consumption, such as:

NDJSON;
SSE;
chunked text;
WebSocket.

Document client behavior: specify framing, reconnect rules, cancellation, and what a partial result means. Streaming is a contract between producer and consumer, not merely a performance switch.

Server-Sent Events

SSE uses a long-lived HTTP response with this content type:

Content-Type: text/event-stream

It is a good fit for one-way server-to-client updates. The connection remains open, so heartbeat behavior, reconnect handling, and proxy timeout settings become part of the design.

WebSocket is bidirectional and uses a different protocol upgrade. Do not treat SSE and WebSocket as interchangeable implementations of the same transport. The Node roadmap in this course focuses on HTTP fundamentals; realtime communication can be added as an advanced extension.

Client disconnect

Requests and responses can close before the work is complete. A client may navigate away, time out, or lose its connection after the server has accepted the request.

Long-running work should support cancellation when the client no longer needs the result, especially when the work consumes an expensive upstream request or database operation. Do not assume that every accepted request remains connected until your handler finishes.

Graceful server close

A native server can stop accepting new connections. That is only one part of a production shutdown.

Graceful shutdown must account for keep-alive connections, in-flight requests, and deadlines. The process needs a bounded way to wait for useful work to finish while avoiding an indefinite hang. The detailed shutdown procedure is covered later, but the HTTP lifecycle already explains why simply stopping the listener is not enough.

Common mistakes

Keep these failure modes in mind when debugging a native server or an API client:

no request body limit;
using string length for content length;
treating a fetch 500 as success;
timing out without aborting the request;
trusting proxy headers blindly;
allowing outbound URL SSRF;
retrying unsafe writes;
buffering a huge request or response;
returning a raw internal error;
creating HTTP client pools repeatedly with a custom client.

When investigating one of these, first identify the boundary where behavior diverges. Inspect the incoming request and route, the status and headers on the response, the transport error or upstream status, and finally the process's resource usage. That avoids treating every failure as a routing problem.

Exercises

Work through these in order. The first exercises establish the request/response mechanics; the later ones require you to reason about production behavior and security.

Build native /health and /tasks GET routes.
Add a JSON response helper.
Parse JSON with a 64 KB limit.
Reject the wrong Content-Type.
Consume an external API with fetch and explicit HTTP error handling.
Add an abort timeout.
Simulate a redirect/SSRF threat and design an allowlist.
Add the ETag concept to one read endpoint.
Stream a large file response.

As you test, inspect both sides of the exchange: the request method, URL, and headers sent by the client, and the response status, headers, and body returned by the server. A request that reaches the server but receives 404 is a different problem from one that never establishes a connection.

Mastery checklist

You should be able to explain:

a native HTTP server;
request and response streams;
URL parsing;
body limits;
headers and status codes;
fetch network errors versus HTTP errors;
cancellation;
connection reuse;
TLS and proxies;
SSRF;
retries and idempotency;
HTTP caching.

If you cannot explain one of these, return to the corresponding section and write a small request or failure case that makes the behavior observable. Being able to diagnose the boundary is more useful than recalling an isolated API name.