274: Availability, Reliability, SLOs, SLIs, Error Budgets, and Capacity Headroom

Learning outcomes

By the end of this lesson, you can:

explain and apply an SLI in a realistic implementation;
explain and apply an SLO in a realistic implementation;
explain and apply an SLA in a realistic implementation;
explain and apply an error budget in a realistic implementation;
explain and apply redundancy and headroom in a realistic implementation.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before reading, retrieve one concrete example from a previous project where a similar concern appeared. Perhaps a checkout endpoint depended on a database, or a background worker had to absorb a traffic spike. The point is not to memorize a list of expansions. It is to make a defensible decision inside a large-scale distributed service, where requirements, traffic, failure modes, cost, and operational constraints all need to be explicit.

As you read, keep asking four questions: What does the user experience? What events count as eligible? What does the system promise internally and externally? What capacity remains when something fails? Those questions connect the vocabulary to design decisions rather than leaving it as monitoring terminology.

Terminology

SLI: A Service Level Indicator is a measured, user-relevant quantity, such as the ratio of successful requests or the proportion of requests completed below a latency threshold. The eligible event population must be defined carefully; otherwise a good-looking percentage can describe the wrong traffic.
SLO: A Service Level Objective is an internal target for an SLI over a specified window, for example, 99.9% of eligible requests succeeding during a calendar month. An SLO is an engineering objective, not automatically a customer contract.
SLA: A service-level agreement is an external or commercial commitment. It may define a measured service level, credits, exclusions, maintenance windows, and the process for making a claim. It is not interchangeable with an internal SLO.
Error budget: The gap between perfect reliability and the SLO is an error budget. For a 99.9% availability target, 0.1% of eligible time or requests may fail within the measurement window, according to the chosen SLI. That budget can guide release velocity and reliability investment based on actual user impact.
Redundancy and headroom: N+1 capacity and multi-zone redundancy allow a system to survive planned maintenance or a failure without immediately saturating the remaining instances. Redundancy supplies alternatives; headroom supplies room for those alternatives to handle the load.
Reliability math: Serial dependencies reduce end-to-end availability because the request succeeds only when every required component succeeds. Redundancy can improve availability, but only when the failure modes are sufficiently independent. Shared dependencies and shared regions can turn apparently redundant components into a correlated failure domain.

The useful distinction is between a measurement, a target, and a promise. An SLI is what you measure, an SLO is what the team aims for, and an SLA is what the organization commits to a customer. The error budget turns the target into an operating decision. Redundancy and headroom are design mechanisms that help the service stay within that target.

Mental model

Treat Availability, Reliability, SLOs, SLIs, Error Budgets, and Capacity Headroom as a design problem with observable inputs, outputs, invariants, and failure modes. Reliability becomes actionable when service behavior is measured against explicit objectives and teams budget both failure and change velocity. A strong implementation makes assumptions visible, narrows uncertainty at boundaries, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to show why the design is safe.

A useful interview and production sequence is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Start with the requirement. “The API should be highly available” is not yet precise enough to implement or verify. Ask which users and requests are included, what counts as success, what window is measured, how much latency is acceptable, and what happens during maintenance. Then identify the dependencies and the capacity available after a failure.

Do not jump from a requirement directly to a library call or a second deployment region. First state what must remain true. Then choose the mechanism that enforces it, and define the observation that will tell you whether it is working. This prevents a dashboard full of infrastructure metrics from being mistaken for evidence of user-facing reliability.

Deep dive

1. SLI

A Service Level Indicator is a measured user-relevant quantity such as successful request ratio or latency under a threshold, with a carefully defined eligible event population. For a read API, a useful availability SLI might be:

successful eligible requests / total eligible requests

The numerator and denominator must describe the same population. If client timeouts, gateway errors, and responses that contain an unusable payload affect the user, excluding them from the denominator makes the SLI misleading. Likewise, a latency SLI needs a threshold and a definition of which requests are included, rather than a vague claim that the endpoint is “fast.”

SLIs are not limited to availability. You might measure the fraction of checkout attempts completed successfully, the percentage of requests under 300 ms, or the freshness of data shown to a user. Choose the signal that represents the user-visible failure. CPU utilization can help diagnose a problem, but it is usually not the SLI for whether a user received a usable response.

Decision rule: Use an SLI deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. In practice, write down the event population, success condition, threshold, aggregation window, and source of truth before putting the value on a dashboard.

2. SLO

An internal objective sets the target for an SLI over a window, for example, 99.9% successful requests in a calendar month. More nines rapidly reduce allowable downtime or error and increase cost. The difference between 99% and 99.9% is not a small cosmetic improvement: it changes the allowed failure from roughly 7 hours 18 minutes to roughly 43 minutes 49 seconds in a 30-day month.

An SLO also needs scope. “99.9% availability” is incomplete unless the service, request class, exclusions, and measurement method are named. A write-heavy checkout path may need a different objective from an internal reporting endpoint. Targets should reflect user and business impact, not a race to add nines that the architecture and operating budget cannot support.

Decision rule: Use an SLO deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. A good SLO gives engineers a decision threshold: when the budget is healthy, planned change can proceed; when the budget is nearly spent, reliability work takes priority.

3. SLA

A service-level agreement is an external/commercial commitment and may include credits or exclusions. It is not interchangeable with internal SLO engineering targets. An SLA may promise a lower number than the internal SLO so the team has operating margin, and it may define availability differently from the product dashboard.

For example, an SLA could promise a monthly availability level for a particular API and offer service credits after a qualifying breach. That does not mean every error is creditable, nor does it mean the SLA captures every user experience problem. A slow but technically successful response may matter to users even if the commercial agreement measures only availability.

Decision rule: Use an SLA deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Before relying on an SLA, inspect its eligible service, measurement window, exclusions, latency definition, and remedy. Those details determine what the agreement actually guarantees.

4. Error budget

The gap between perfect reliability and the SLO is an error budget that can guide release velocity and reliability investment based on actual user impact. With an SLO of 99.9%, the monthly budget is 0.1% of eligible requests. If a read API receives 10,000,000 eligible requests in a month, the rough budget is 10,000 failed requests. For a time-based availability SLI over a 30-day month, 0.1% is about 43 minutes and 49 seconds.

The unit follows the SLI. A request-based SLI produces an allowed number of bad requests; a time-based SLI produces an allowed amount of unavailable time. Do not add those numbers together as though they were interchangeable. Also, an error budget is not permission to cause failures intentionally. It is a shared way to decide whether the risk of a deployment, migration, or experiment is justified by the remaining budget.

If a release consumes an unusually large part of the budget, investigate the cause and slow further change while the service is restored. If the budget is consistently untouched, the team might be over-investing in reliability, measuring the wrong user outcome, or leaving useful delivery capacity unused. The correct response depends on evidence, not on treating the budget as a score.

Decision rule: Use an error budget deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Record how incidents and failed requests consume the budget, and make the action triggered by a low budget explicit.

5. Redundancy and headroom

N+1 capacity and multi-zone redundancy let the system survive maintenance or a failure without immediately saturating remaining instances. If a service needs four instances to handle normal peak traffic, running five is a basic N+1 plan. It is only adequate if one instance can disappear while the four survivors still have enough CPU, memory, connection capacity, and downstream capacity.

Placement matters. Five instances in one failure domain are not equivalent to instances spread across independent zones. Even across zones, a shared load balancer, database, network path, quota, or deployment system can remain a single point of failure. Redundancy also has a cost: more compute, more replication, more operational complexity, and sometimes more consistency or coordination work.

Headroom is the unused capacity that absorbs variation and failure. Running every instance at its theoretical maximum may look efficient, but it leaves no room for a retry storm, uneven load balancing, a slow dependency, or a node being drained during deployment. Define headroom against a credible peak and failure scenario, then verify it with load tests and production measurements.

Decision rule: Use redundancy and headroom deliberately when they make the contract or invariant easier to prove. If they only reduce typing while hiding an assumption, prefer the more explicit design. State which component may fail, how much load remains, and whether the remaining system still satisfies latency and dependency limits.

6. Reliability math

Serial dependencies reduce end-to-end availability; redundancy can improve it only when failures are sufficiently independent. If an API requires both a cache and a datastore, and their availability is approximately 99.9% and 99.99%, the simple serial estimate is:

0.999 × 0.9999 = 0.9989001, or about 99.89001%

That is lower than either component's availability. It is only a rough model: retries, fallbacks, cached reads, maintenance exclusions, and the precise definition of success can change the result. Still, it exposes a common design mistake: multiplying several “good enough” dependencies can produce an end-to-end target that the product cannot accept.

For two redundant components with independent availability A, the chance that at least one is available is:

1 - (1 - A)²

Two 99.9%-available instances therefore have a theoretical availability of 99.9999%, but only under assumptions that often fail in production. A shared zone outage, bad rollout, common software defect, exhausted quota, or misconfigured load balancer can take both out together. Independence is a design property to investigate, not a number to assume because resources have different instance names.

Decision rule: Use reliability math deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Use the calculation to reveal risks, then validate the assumptions with failure-domain analysis, dependency documentation, testing, and operational evidence.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be made explicit. Start by writing the requirement in one sentence, list the input and output contracts, and identify which of the concepts above owns each failure mode. For example: “During the monthly peak, eligible read requests should meet a 99.9% success SLO, and checkout should fail safely without charging a customer twice.” That sentence still needs traffic and latency details, but it gives the design something observable to protect.

The important move is separation: parsing or validation belongs at the boundary; domain rules belong in the domain/service layer; persistence rules belong in the database or repository; presentation rules belong in the client. Mixing these concerns makes a happy-path demo look shorter, but makes edge cases much harder to reason about. An SLI should be computed from well-defined user-facing outcomes, not from whichever internal component happens to have an easy metric.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

Read the diagram as a set of failure boundaries. DNS and the edge determine whether traffic reaches the service. The load balancer distributes work, but it does not make the API instances or their dependencies available. The cache can reduce datastore load, while the primary datastore remains a required dependency for misses or writes. The queue and workers can move work out of the request path, but they introduce delay, retry, duplicate-delivery, and eventual-consistency behavior.

Walk the example with at least four cases: the normal path, an empty or missing value, a duplicate/retry/concurrent path where relevant, and a dependency failure. For each case, state which layer detects the problem and what the caller observes:

On the normal read path, the API validates the request, obtains data from the cache or datastore, and returns a response that counts as an SLI success if it is usable by the client.
For an empty or missing value, define whether the result is a valid empty response, a not-found response, or an error. Do not let a cache miss, a missing record, and malformed input collapse into the same unexplained status.
For a duplicate checkout request or retry, use an idempotency strategy at the service and persistence boundary. A client-side button state alone cannot prevent a repeated network request from charging twice.
For a dependency failure, apply a bounded timeout and a clear fallback or error. Record the user-visible outcome consistently so the SLI reflects the failure rather than hiding it behind an indefinitely pending request.

This is also where the SLO and capacity design meet. If one API instance is removed during a deploy, the remaining instances must have enough headroom to handle the traffic without crossing the latency target. If the datastore is serially required, its availability and failure behavior constrain the end-to-end SLO. If work is queued, decide whether accepting the request counts as success or whether completion is the user-relevant event; the SLI must match that decision.

For each case, state which layer detects the problem, what gets logged or measured, whether a retry is safe, and what the caller observes. This is the level of explanation expected in a senior code review or technical interview. The goal is not to claim that every failure can be eliminated. The goal is to show where it is detected, how much failure is acceptable, and how the system remains understandable when the happy path is gone.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high cardinality. Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after you can identify the bottleneck or risk with evidence.

When the topic involves an external dependency, define a timeout and cancellation strategy. An unbounded request can consume threads, connections, memory, and retry capacity while the caller is already gone. When it involves persistence, define transaction and consistency expectations, including what happens if a worker retries after the first attempt committed.

When it involves user-visible state, define loading, empty, error, stale, and success states. A successful HTTP response is not necessarily a successful user experience if the payload is stale or incomplete. When it involves security, assume the client can be modified and the network input is untrusted. Client checks can improve usability, but they do not replace server-side authorization or validation.

Operationally, distinguish symptoms from causes. A rising API error SLI may be caused by datastore saturation, a bad deployment, exhausted connection pools, or a shared network failure. Track enough dimensions to investigate without creating unbounded high-cardinality metrics. Include deployment markers and dependency outcomes so an error-budget burn can be connected to a change or failure mode.

Guided lab

Define SLIs/SLOs for a read API and checkout. Calculate rough monthly error budgets for two availability targets, then show how a critical serial dependency changes end-to-end availability.

For the two targets, use 99% and 99.9% over a 30-day month. The approximate time budgets are:

30 days × 24 hours × 60 minutes = 43,200 minutes
99%   -> 1% error budget   -> 432 minutes
99.9% -> 0.1% error budget -> 43.2 minutes

For the request-based version, if the read API receives 1,000,000 eligible requests, those same targets allow approximately 10,000 and 1,000 unsuccessful requests. State whether your checkout SLI measures the request response or successful order completion; those are different events and can have different objectives.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

For the reliability calculation, choose a critical serial dependency such as an API with a datastore. Use the approximate availability of each dependency, multiply them, and compare the result with the API's target. Then describe one mitigation, such as a cache or fallback, and explain what new stale-data, consistency, or failure-mode trade-off that mitigation introduces. A calculation without its assumptions is not a finished answer.

Edge cases and failure modes

SLI: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Also test whether client timeouts, gateway failures, and unusable successful responses are represented consistently in the eligible population.
SLO: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Check that the target has a defined window, scope, threshold, and action when the budget is nearly exhausted.
SLA: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Inspect exclusions, maintenance rules, measurement ownership, and the difference between a service credit and actual recovery of user impact.
Error budget: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Confirm that the budget's units match the SLI and that retries or duplicate events do not distort consumption.
Redundancy and headroom: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Remove an instance or zone in a controlled test and check remaining capacity, dependency limits, load balancing, and recovery time.

Failure modes are not limited to a component being completely down. Partial packet loss, slow responses, stale cache entries, uneven traffic, correlated zone failures, retry storms, and a deployment that is valid but incompatible with old clients can all consume reliability budget. Include these cases when deciding what the SLI should report and what the system should do next.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values.
Testing only the happy path and therefore discovering contracts only after integration.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees.

Other common reliability mistakes include using CPU as a substitute for a user-facing SLI, counting only successful requests in the denominator, treating an SLA number as the engineering target, and calling two components redundant when they share a failure domain. A dashboard can be precise and still answer the wrong question.

For debugging, reproduce the smallest failing case, inspect the actual value or execution plan, trace the boundary where the invariant first becomes false, and fix the owning layer rather than adding a downstream patch. Start at the user-visible symptom, then follow request IDs through the edge, API, cache, datastore, and worker path. Compare the event counts used by the SLI with raw logs or traces, and check whether a deploy or dependency change aligns with the first budget burn.

If an availability percentage suddenly changes, first verify the denominator and the measurement window. If latency rises without an availability change, inspect queueing, connection pools, dependency latency, and saturation rather than assuming the SLO is unaffected. If failover causes errors, check whether the surviving instances had the planned headroom and whether the supposedly independent dependency failed with them.

Interview questions

What problem does SLI solve, and what trade-off or failure mode would make you choose a different approach?
What problem does SLO solve, and what trade-off or failure mode would make you choose a different approach?
What problem does SLA solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Error budget solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Redundancy and headroom solve, and what trade-off or failure mode would make you choose a different approach?

When answering, do more than expand the acronym. Define the measured population or commitment, give a small calculation or example, identify an assumption, and name the failure mode that would invalidate the design. A strong answer distinguishes what the user experiences from what the infrastructure reports.

Checkpoint

Without notes, explain Availability, Reliability, SLOs, SLIs, Error Budgets, and Capacity Headroom to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code.

As a self-check, you should be able to explain why a 99.9% target permits about 43 minutes of unavailability in a 30-day month, why a serial dependency can lower the service's effective availability, and why two instances in one shared failure domain are not automatically independent redundancy. You should also be able to say which layer owns each decision and how you would verify it in production.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.