254: Scaling Compute: Vertical/Horizontal Scaling, Stateless Services, Autoscaling, and Load Shedding

Learning outcomes

By the end of this lesson, you can:

explain and apply vertical scaling in a realistic implementation;
explain and apply horizontal scaling in a realistic implementation;
explain and apply stateless request tier in a realistic implementation;
explain and apply autoscaling signals in a realistic implementation;
explain and apply warm-up and cold start in a realistic implementation.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before reading, retrieve one concrete example from a previous project where this concern showed up. Perhaps one process was running out of CPU, one server was carrying all in-memory sessions, or a queue kept growing faster than workers could drain it. The point is not to memorize a set of scaling terms. It is to make a defensible decision for a large-scale distributed service, where requirements, traffic, failure modes, cost, and operational constraints all need to be explicit.

Terminology

Vertical scaling: Increase the capacity of one machine by giving it more CPU, memory, or other resources. Larger machines are operationally simple and useful early, but they have finite ceilings, create larger failure domains, and often cost more than linearly as capacity increases.
Horizontal scaling: Add more instances and distribute work among them. Multiple instances can increase throughput and failure tolerance, but they also require load balancing, distributed state handling, coordinated deploys, and shared dependencies that may become the next bottleneck.
Stateless request tier: Keep durable state and user session state outside individual request-serving instances when practical. Requests can then move between instances without depending on the history or memory of one particular process.
Autoscaling signals: Measurements used to decide when capacity should change. CPU, request rate, queue depth, latency, and custom saturation metrics each reveal different parts of the system and differ in how quickly and reliably they indicate pressure.
Warm-up and cold start: The time and work required before a new instance can serve traffic. That work may include downloading an image, initializing the runtime, running JIT compilation, filling caches, and opening connections.
Load shedding: Reject or degrade lower-priority work early when dependencies are saturated, rather than allowing queues to grow until every request times out. This is a form of backpressure that protects the system's ability to recover.

Mental model

Treat Scaling Compute: Vertical/Horizontal Scaling, Stateless Services, Autoscaling, and Load Shedding as a design problem with observable inputs, outputs, invariants, and failure modes. A useful invariant for a request tier is that any healthy instance can handle any request that the load balancer sends to it; durable state must not be trapped in one instance. For a worker system, a corresponding invariant is that incoming work cannot be accepted indefinitely when the dependency or worker capacity is already saturated.

Compute scaling works best when request-serving instances are disposable and shared state lives in explicit stores. That does not mean every local cache is wrong. A local cache is reasonable when a miss is safe and correctness does not depend on the cache being present. Autoscaling also is not an instant capacity switch: it needs time to observe a signal, start an instance, make that instance ready, and route work to it. Capacity limits and overload behavior therefore belong in the design from the beginning.

A strong implementation makes assumptions visible, narrows uncertainty at boundaries, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to show why the design is safe. A useful interview and production sequence is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement directly to a library call or a cloud setting. First state what must remain true. Then choose the mechanism that enforces it, and identify what happens when that mechanism is slow, unavailable, or at its limit.

Deep dive

1. Vertical scaling

If one instance is close to its CPU or memory limit, moving it to a larger machine may be the smallest safe change. It preserves a simple deployment and avoids introducing load balancing and distributed coordination before those costs are justified. That simplicity is real, especially for an early service or a workload that needs a large memory footprint in one process.

The trade-off is a ceiling. A machine can only be enlarged so far, and the instance remains one failure domain: if it fails or is unavailable during maintenance, the whole capacity unit is affected. Larger machines can also have non-linear cost, so buying twice the resources does not necessarily cost twice as much. Vertical scaling can buy time, but it does not remove the need to understand the workload's eventual limit.

Decision rule: Use vertical scaling deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. State the limit you are accepting and the signal that would tell you it is time to move to another approach.

2. Horizontal scaling

Horizontal scaling adds instances and spreads requests or jobs across them. It can raise throughput and reduce the impact of a single instance failure, but it moves complexity into the system around those instances. A load balancer must know which instances are healthy, deploys must account for several copies, and state that used to be local must now be coordinated or moved to a shared dependency.

That shared dependency may become the actual bottleneck. Adding API instances does not help if every instance waits on one saturated database, cache, or downstream service. Horizontal scaling is therefore a capacity design, not just a replica-count setting. Measure the limiting resource and check that each dependency can support the additional concurrency.

Decision rule: Use horizontal scaling deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Be clear about what is being distributed, where its state lives, and which dependency remains a single or shared limit.

3. Stateless request tier

An instance is easiest to replace or add when it does not own durable user state. Keep sessions, durable data, and other state needed for correctness in an explicit shared store when practical. Then a retry or the next request can reach any healthy API instance instead of depending on a sticky route to the process that handled the previous request.

Local memory is still useful for temporary caches, connection pools, and other performance optimizations. The boundary is correctness: if removing an instance loses information the system must retain, that information is not merely local request state. If a local cache disappears, the design should tolerate a miss and retrieve the value from the authoritative store.

Decision rule: Use a stateless request tier deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Ask what happens during a restart, a retry, a deployment, and a request routed to a different instance.

4. Autoscaling signals

Autoscaling is only as good as the signal it follows. CPU utilization can be useful for CPU-bound work, but it may remain low while requests wait on a database. Request rate describes incoming demand but not necessarily the work per request. Queue depth is often a direct signal for worker pressure, while latency tells you what the caller is experiencing, although latency can rise only after the system is already overloaded. A custom saturation metric can be the most useful signal when it measures the resource that actually limits throughput.

Signals also differ in timing. A lagging signal can react too late to a burst, and an overly sensitive signal can cause replicas to oscillate. Scaling policies need sensible thresholds, bounds, and stabilization behavior, and they must account for the time it takes for a new instance to become ready. More replicas do not fix a saturated shared dependency, so that dependency's capacity and error behavior must be part of the policy.

Decision rule: Use autoscaling signals deliberately when they make the contract or invariant easier to prove. If they only reduce typing while hiding an assumption, prefer the more explicit design. Tie each signal to a concrete bottleneck, verify that it leads the failure you care about, and define minimum and maximum capacity rather than assuming scaling is unlimited.

5. Warm-up and cold start

A new instance is not useful merely because it has been created. It may first download its image, initialize the runtime, perform JIT compilation, load configuration, establish database or service connections, and populate caches. Until those steps are complete, routing traffic to it can increase errors or latency rather than adding useful capacity.

Capacity planning must include time-to-ready and minimum warm capacity. A system that scales from zero may be inexpensive when idle but unable to absorb a sudden burst quickly. A system that keeps warm instances pays for that readiness but has a better response to short spikes. Readiness checks should represent the ability to serve the relevant traffic, not just whether a process has opened a port.

Decision rule: Use warm-up and cold start deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Include startup duration, connection limits, cache behavior, and the minimum warm capacity in the failure analysis.

6. Load shedding

When a dependency is saturated, continuing to accept every request usually makes recovery worse. Work waits in longer queues, memory is consumed by pending requests, and callers eventually see timeouts instead of a quick, understandable failure. Load shedding rejects or degrades lower-priority work early so critical work retains a chance to complete.

Shedding must be intentional: define which requests are lower priority, what response or fallback they receive, and how retries are controlled. Otherwise clients may retry a rejection and recreate the same overload. Backpressure protects recovery by keeping queues and resource usage bounded; it is not a substitute for capacity planning, but it prevents overload from spreading through the dependency graph.

Decision rule: Use load shedding deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Make the priority, threshold, response, and retry behavior observable so operators can distinguish deliberate shedding from an unexplained outage.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be made explicit. Start by writing the requirement in one sentence. For example, the request tier may need to serve traffic through instance replacement while asynchronous work is processed without allowing an unbounded queue. Then list the input and output contracts and identify which concept above owns each failure mode.

The important move is separation. Parsing and validation belong at the boundary; domain rules belong in the domain or service layer; persistence rules belong in the database or repository; presentation rules belong in the client. Mixing these concerns can make a happy-path demo look shorter, but it makes edge cases, retries, and partial failures much harder to reason about. The same separation applies to scaling: an API instance can be disposable, while the datastore and queue have their own capacity and consistency contracts.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

Read the diagram as a set of boundaries rather than as a guarantee that every request uses every component. The edge may cache or route traffic; the load balancer distributes requests among API instances; the cache can reduce repeated reads; the primary datastore owns durable data; and the queue or stream separates accepted asynchronous work from workers that process it. Each boundary introduces a possible timeout, stale value, retry, or capacity limit.

Walk the example with at least four cases: the normal path, an empty or missing value, a duplicate/retry/concurrent path where relevant, and a dependency failure. For each case, state which layer detects the problem and what the caller observes. Also ask whether a retry can land on another API instance, whether the operation is safe to repeat, and whether the queue is allowed to grow. This is the level of explanation expected in a senior code review or technical interview: name the behavior, the owning layer, and the evidence you would use to verify it.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high cardinality. Scaling adds more operational questions: can instances drain in-flight work before termination, does a readiness check prevent premature routing, and do metrics distinguish a slow dependency from a CPU-bound process?

Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after you can identify the bottleneck or risk with evidence. A larger machine, more replicas, or a bigger queue can hide a problem temporarily without changing its cause.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and the network input is untrusted. These rules still apply when requests are routed through several instances; scaling does not move authorization or validation responsibility to the client.

Guided lab

Design autoscaling for a stateless API and worker pool. Choose signals, minimum and maximum replicas, warm-up behavior, queue thresholds, and which low-priority requests are shed during overload. Explain why each choice matches the bottleneck you are trying to control. Include what happens when the API scales out but the primary datastore does not, and what happens when a worker is terminated while processing a job.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

The lab is not complete if it only produces a replica count. A useful result explains the signal-to-action path: what observation triggers scaling, how long new capacity takes to become ready, what bound prevents uncontrolled growth, and what a caller sees when work is shed.

Edge cases and failure modes

Vertical scaling: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Confirm that a larger instance does not change correctness assumptions or simply move the bottleneck to a dependency.
Horizontal scaling: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Include a request that is retried on a different instance and an instance becoming unhealthy during work.
Stateless request tier: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Restart an instance and verify that durable or session state remains available without relying on process-local memory.
Autoscaling signals: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Check bursts, lagging metrics, scaling bounds, and the case where the chosen metric improves while a shared dependency remains saturated.
Warm-up and cold start: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Measure time-to-ready and verify that traffic is not sent to an instance before its runtime, connections, and required initialization are ready.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong. A queue, cache, or extra replica is not automatically justified just because it appears in a familiar diagram.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values. These choices can conceal whether the system is rejecting malformed work, retrying it, or silently losing information.
Testing only the happy path and therefore discovering contracts only after integration. Scaling and failure behavior must be tested at boundaries, not inferred from a successful local request.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement. More instances can increase contention, connection usage, and cost when the real limit is elsewhere.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees. Any instance receiving a request must enforce the server-side contract.

For debugging, reproduce the smallest failing case, inspect the actual value or execution plan, trace the boundary where the invariant first becomes false, and fix the owning layer rather than adding a downstream patch. Check the request rate, queue depth, latency, CPU, memory, dependency errors, and instance readiness together; one healthy-looking metric is not proof that the system is healthy. During overload, determine whether the request was processed, timed out, retried, or deliberately shed.

Interview questions

What problem does Vertical scaling solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Horizontal scaling solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Stateless request tier solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Autoscaling signals solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Warm-up and cold start solve, and what trade-off or failure mode would make you choose a different approach?

Checkpoint

Without notes, explain Scaling Compute: Vertical/Horizontal Scaling, Stateless Services, Autoscaling, and Load Shedding to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code. Be specific about the resource that limits the example, the signal that exposes that limit, and what the caller experiences when the limit is reached.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.