252: Proxies, Reverse Proxies, Load Balancers, and API Gateways

Learning outcomes

By the end of this lesson, you can:

explain and apply a forward proxy in a realistic implementation;
explain and apply a reverse proxy in a realistic implementation;
explain and apply layer 4 and layer 7 balancing in a realistic implementation;
explain and apply load-balancing algorithms in a realistic implementation;
explain and apply health checks in a realistic implementation.

The goal is not just to name these components. You should be able to place one in a request path, explain what it knows, identify which failure it owns, and justify the operational trade-off it introduces.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before reading, retrieve one concrete example from a previous project where one of these concerns appeared. It might have been a corporate proxy controlling outbound requests, a reverse proxy terminating HTTPS, a platform load balancer distributing traffic, or an API gateway enforcing authentication and quotas.

Use that example as a retrieval exercise. What entered the system, which intermediary saw it first, where was the decision made, and what happened when a dependency was unavailable? The purpose is not to memorize a vocabulary list. It is to make a defensible decision inside a large-scale distributed service, where requirements, traffic, failure modes, cost, and operational constraints must be explicit.

Terminology

Forward proxy: A forward proxy acts on behalf of clients. The client chooses, or is configured to use, the proxy for outbound traffic. The proxy can provide egress control, filtering, caching, or anonymity. The destination may see the proxy rather than the original client.
Reverse proxy: A reverse proxy accepts client traffic on behalf of servers. The client addresses the public service, while the proxy can terminate TLS, route by path or host, compress, cache, enforce limits, and hide backend topology.
Layer 4 and Layer 7 balancing: L4 balancing works at the transport level, commonly distributing TCP or UDP connections with limited application awareness. L7 balancing understands application metadata such as HTTP methods, hostnames, paths, headers, and sometimes cookies, which enables richer routing at higher processing cost.
Load-balancing algorithms: Round robin, least connections, weighted, hashing, and latency-aware approaches fit different workloads. The right algorithm depends on request cost, connection lifetime, instance capacity, state, and how quickly the system must react to change.
Health checks: Liveness is not readiness. A process can be alive and responding while still unable to serve useful application traffic. Health checks are therefore a precise routing and failure-detection mechanism, not merely a vocabulary exercise.
API gateway: An API gateway is a reverse-proxy-like boundary that can centralize authentication, quotas, transformations, routing, and observability for APIs. It can simplify common policy, but concentrating too much domain behavior there creates a tightly coupled failure and deployment boundary.

Mental model

Treat Proxies, Reverse Proxies, Load Balancers, and API Gateways as a design problem with observable inputs, outputs, invariants, and failure modes. A traffic intermediary is not automatically “the infrastructure layer that makes things scalable.” It is another component that receives input, makes decisions, consumes resources, and can fail.

These intermediaries are useful because they centralize routing and cross-cutting controls. The same boundary can enforce TLS policy, rate limits, request size limits, authentication checks, or access to approved destinations instead of making every application implement those concerns independently. The cost is that the boundary can become a bottleneck, obscure client identity, add latency, and make it less obvious which component owns a failure.

A strong implementation makes assumptions visible, narrows uncertainty at boundaries, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to prove why the design is safe. For example, “send traffic to a healthy instance” is not a sufficient invariant until “healthy” means “ready to serve this contract,” and until the system defines what happens when no instance satisfies that condition.

A useful interview and production sequence is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement directly to a library call or a product name. First state what must remain true. Then choose the mechanism that enforces it. If the requirement is to prevent arbitrary outbound access, that points toward an egress policy and a forward proxy. If the requirement is to route /payments separately from /catalog, the need for HTTP-aware routing points toward an L7 boundary. The terminology follows the behavior you need, not the other way around.

Deep dive

1. Forward proxy

When a service must reach many external systems, allowing every workload to make arbitrary outbound connections makes policy and investigation difficult. A forward proxy acts on behalf of those clients and becomes a controlled egress point. The client sends the outbound request through the proxy; the proxy applies policy and contacts the destination.

That arrangement can enforce an allowlist of destinations, record outbound requests, scan or filter traffic, provide a shared cache where that is safe, and keep internal network details away from the destination. It can also support privacy or anonymity in contexts where that is an explicit requirement. A forward proxy does not make a destination trustworthy, and it does not remove the need to authenticate or validate the response.

There is a practical distinction from a reverse proxy: with a forward proxy, the client is the party using the proxy to reach a server. With a reverse proxy, the public client reaches the proxy as though it were the server, and the proxy chooses or protects the backend. Confusing these roles leads to the wrong policy boundary and often to incorrect assumptions about which address represents the original client.

Decision rule: Use a forward proxy deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Define proxy failure behavior, timeout and cancellation policy, credential handling, destination policy, and whether the destination needs the original client identity. Do not put secrets in URLs or assume that a proxy can safely inspect encrypted traffic without an explicit, trusted certificate and privacy design.

2. Reverse proxy

A public service often needs one stable entry point even while its application instances change. A reverse proxy accepts client traffic for those servers and can terminate TLS, route paths and hosts, compress responses, cache eligible content, enforce request limits, and hide backend topology. It can also add or preserve forwarding metadata so the application can reason about the original request, but that metadata must be trusted only when it comes from a controlled proxy chain.

TLS termination is a useful example of the boundary trade-off. Terminating TLS at the reverse proxy centralizes certificate management and lets the proxy inspect HTTP for routing. It also means the connection from the proxy to the backend needs its own security decision; encryption from client to proxy is not automatically encryption from proxy to service. Similarly, caching or compression can reduce backend work and transfer cost, but only when the cache key, authorization behavior, content variation, and freshness rules are correct.

Decision rule: Use a reverse proxy deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Specify which headers are trusted, which paths are public, how request bodies are bounded, what timeout applies at each hop, and how retries interact with non-idempotent requests. A reverse proxy should not silently become the place where domain business rules live; those rules belong in the owning service.

3. Layer 4 and Layer 7 balancing

The useful distinction is what the intermediary can observe. L4 balancing distributes transport connections with limited application awareness. It can be efficient and protocol-agnostic, which is valuable for TCP or UDP services and for workloads where the intermediary should not parse application data. But it cannot normally choose a backend based on an HTTP path or header because those are application-level details.

L7 balancing understands HTTP-level metadata and can route api.example.com differently from admin.example.com, or /v1 differently from /v2. It can apply method- or header-aware policies and often has better request-level observability. That capability requires parsing more data and may introduce more CPU, memory, latency, configuration complexity, and protocol-specific failure modes.

The choice also affects connection behavior. L4 commonly makes a decision for a connection, while L7 may make decisions for individual requests on persistent HTTP connections. The exact behavior depends on the implementation and protocol, so do not infer guarantees from the label alone. Verify how the chosen component handles keep-alive, HTTP/2 multiplexing, WebSockets, client IP information, retries, and long-lived streams.

Decision rule: Use layer 4 and layer 7 balancing deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Choose L4 when transport-level distribution and protocol neutrality are the priority. Choose L7 when the requirement explicitly needs application-aware routing or policy, and budget for parsing, configuration, observability, and security at that boundary.

4. Load-balancing algorithms

The algorithm is a policy for choosing among eligible backends; it is not a substitute for health checks or capacity planning. Round robin rotates through instances and is easy to understand, but it assumes requests and instances are roughly comparable. Least connections favors the instance with fewer active connections and can help when connections have different lifetimes, although connection count is only a proxy for actual work.

Weighted algorithms account for unequal capacity, such as during a gradual rollout or when one zone has larger instances. The weights must be maintained as capacity changes. Hashing can provide stable placement based on a client, key, or session, which is useful for locality or legacy in-memory state. It can also create hotspots and makes failover behavior important. Latency-aware approaches use observed response behavior to favor faster instances, but measurements can be noisy and feedback loops can overreact to temporary conditions.

Session affinity solves some legacy state problems by repeatedly sending related traffic to the same backend. It fights even distribution, complicates failover, and can hide an architectural issue: durable session state usually belongs in a shared store or in a stateless service design. The algorithm should match the workload, and the design should state whether a retry is safe, whether a request is idempotent, and whether a backend can handle the request without prior local state.

Decision rule: Use load-balancing algorithms deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Start with the simplest policy that meets measured requirements, then validate distribution, queueing, tail latency, and behavior during backend removal. A balanced number of requests does not necessarily mean balanced resource usage.

5. Health checks

Liveness is not readiness. A process may accept TCP while lacking database connectivity, required credentials, a warmed cache, a mounted dependency, or the state needed to serve its contract. Sending traffic to such an instance can turn a recoverable startup condition into visible application failures.

A liveness check answers whether the process should be restarted or considered alive. A readiness check answers whether this instance should receive traffic now. Readiness should be narrow enough to represent the service contract, but not so broad that a temporary optional dependency removes every instance from service. The check itself needs a timeout, interval, failure threshold, recovery behavior, and protection against causing more load than the traffic it is meant to protect.

Health is also a distributed-systems judgment. A dependency may be briefly slow rather than unavailable, and independent observers may disagree about an instance. Use hysteresis or consecutive-failure rules where appropriate so one transient result does not cause unnecessary flapping. During deployment, mark an instance unready before shutting it down and allow in-flight work to drain when the platform supports it.

Decision rule: Use health checks deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Document what each check proves, what it does not prove, and what callers observe when no ready backend remains. A green health endpoint is evidence about that endpoint, not proof that every user request will succeed.

6. API gateway

An API gateway can provide one managed boundary for authentication, quotas, transformations, routing, and observability. Centralizing these policies can make behavior consistent across several services and gives operators a useful place to measure requests, reject oversized input, and attach correlation information.

That centralization has limits. Authentication at the gateway does not eliminate authorization decisions that require domain knowledge inside the service. A gateway can validate the shape of input, but the service still owns rules about whether a user may modify a particular resource. Transformations can help clients evolve, but they can also conceal incompatible contracts and make debugging harder. Quotas need a defined identity, scope, time window, and failure response.

Avoid putting domain business logic into a giant gateway that becomes a tightly coupled monolith. A gateway should make cross-cutting boundary policy easier to operate, not become the only place where the business can be understood or deployed. Keep timeouts, retries, error mapping, and observability explicit, especially when one incoming request fans out to multiple services.

Decision rule: Use an API gateway deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Put shared edge policy at the gateway, and keep resource ownership, domain authorization, and transactional rules in the relevant service.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be made explicit. Start by writing the requirement in one sentence, then list the input and output contracts. Identify which concept above owns each failure mode instead of treating “the load balancer” or “the gateway” as a single magical component.

The important move is separation: parsing or validation belongs at the boundary; domain rules belong in the domain or service layer; persistence rules belong in the database or repository; presentation rules belong in the client. Mixing these concerns makes a happy-path demo look shorter, but it makes edge cases, retries, and ownership much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                         |          |
                         +------> Primary datastore
                         |
                         +------> Queue / Stream -> Workers

Read this from left to right as a request path, then inspect the side effects separately. DNS finds the public entry point; the CDN or edge may serve cacheable content or apply an outer policy; the load balancer selects an eligible API instance; the API may use a cache, primary datastore, and asynchronous queue. Each arrow is a possible timeout, retry, identity boundary, or partial failure. A queue can remove slow work from the request path, but it also introduces eventual completion and a need for retry or duplicate handling.

Walk the example with at least four cases: the normal path, an empty or missing value, a duplicate, retry, or concurrent path where relevant, and a dependency failure. For each case, state which layer detects the problem and what the caller observes. For example, malformed input should be rejected at the API boundary; a conflicting update may require service and datastore coordination; an unavailable cache might be a recoverable miss; and an unavailable primary datastore may require a bounded error rather than an unbounded retry storm.

This is the level of explanation expected in a senior code review or technical interview. Do not just say that traffic is “handled by the infrastructure.” Name the component, the signal it uses, the invariant it protects, and the evidence you would inspect when it fails.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high cardinality. A new backend may pass a shallow health check while failing real requests; a retry may improve availability for a read while duplicating a charge; and a metric label containing an unbounded URL can create an observability cost problem.

Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after you can identify the bottleneck or risk with evidence. Measure both averages and tail behavior where latency matters, and distinguish connection count, request count, queue depth, error rate, saturation, and dependency latency rather than treating them as interchangeable signals.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and the network input is untrusted. Also define how client identity is propagated through proxies and how logs avoid exposing tokens, credentials, or sensitive request data.

Guided lab

Design ingress for an API with three services. Compare L4 vs L7 balancing, define readiness checks, route and version policies, rate limits, and what happens when one backend zone becomes unhealthy. Your design should say whether TLS terminates at the edge, what traffic is sent to each service, which headers are trusted, how a request is rejected, and whether a retry is safe.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

For the inspection step, trace a request through the diagram and record the selected backend, health state, status code, latency, and correlation identifier. Then remove or mark one backend unready and observe whether new traffic stops reaching it, whether existing work drains, and what the caller receives if capacity is exhausted. Do not paste unknown JavaScript into DevTools or expose real credentials while testing an edge configuration.

Edge cases and failure modes

Forward proxy: test absence of proxy configuration, malformed destinations or proxy responses, duplicate requests where the operation is not idempotent, ordering and concurrency where applicable, and behavior at the smallest and largest credible request or payload sizes. Also test an unreachable proxy, a denied destination, timeout and cancellation, credential failure, and whether logs reveal sensitive outbound data.
Reverse proxy: test absent or malformed host, path, header, and body input; duplicate requests; ordering and concurrency where applicable; and behavior at the smallest and largest credible sizes. Also test an expired certificate, an untrusted forwarding header, an oversized body, cache variation by authorization, backend timeout, and graceful shutdown.
Layer 4 and Layer 7 balancing: test absence of a connection or required HTTP metadata, malformed protocol input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Compare persistent connections, multiplexed requests, long-lived streams, WebSockets, and routing that requires L7 visibility.
Load-balancing algorithms: test absence of eligible backends, malformed or missing hash keys, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Test unequal instance capacity, a hot key, changing weights, a failed instance, stale affinity, and the effect on tail latency rather than only average distribution.
Health checks: test absence of a response, malformed check output, duplicate checks, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Test startup, dependency failure, slow dependency response, flapping readiness, stale health information, rolling deployment, and the case where every backend is unready.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values.
Testing only the happy path and therefore discovering contracts only after integration.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees.

For debugging, reproduce the smallest failing case and follow the request boundary by boundary. Inspect the actual value or execution plan, trace where the invariant first becomes false, and fix the owning layer rather than adding a downstream patch. In practice, check DNS and edge routing first, then the proxy's access and error logs, selected backend and health state, server route logs, dependency timings, and finally the database or queue. Compare the client-visible status and latency with the internal evidence; a gateway timeout does not tell you by itself whether the backend timed out, was never selected, or returned an invalid response.

Interview questions

What problem does a forward proxy solve, and what trade-off or failure mode would make you choose a different approach?
What problem does a reverse proxy solve, and what trade-off or failure mode would make you choose a different approach?
What problem does layer 4 and layer 7 balancing solve, and what trade-off or failure mode would make you choose a different approach?
What problem do load-balancing algorithms solve, and what trade-off or failure mode would make you choose a different approach?
What problem do health checks solve, and what trade-off or failure mode would make you choose a different approach?

When answering, do not stop at a definition. Give the request path, state what the component can observe, name one invariant, and explain what evidence would distinguish a configuration problem from a backend failure.

Checkpoint

Without notes, explain Proxies, Reverse Proxies, Load Balancers, and API Gateways to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.