276: Disaster Recovery, Backups, RPO/RTO, Multi-Zone/Region Design, and Game Days

Learning outcomes

By the end of this lesson, you can:

explain and apply RPO in a realistic implementation;
explain and apply RTO in a realistic implementation;
explain and apply backup types in a realistic implementation;
explain and apply why replication is not backup in a realistic implementation;
explain and apply multi-zone versus multi-region design in a realistic implementation.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before you read, retrieve one concrete example from a previous project where this kind of concern appeared. Perhaps a database had to be restored, a deployment damaged data, or a service needed to keep operating after an infrastructure failure. The point is not to memorize a list of terms. It is to make a defensible decision for a large-scale distributed service, with its requirements, traffic, failure modes, cost, and operational constraints stated explicitly.

Terminology

RPO: Recovery Point Objective is the maximum amount of data loss the system can accept, expressed as a time interval. An RPO of five minutes means that, after recovery, losing the most recent five minutes of writes is within the stated contract.
RTO: Recovery Time Objective is the maximum acceptable time to restore service after a disruptive event. It includes the work required to detect the incident, choose a recovery path, restore data, and make the service available again.
Backup types: Full backups, incremental or differential backups, snapshots, and log/PITR approaches have different restore dependencies, recovery characteristics, and retention requirements.
Replication is not backup: Logical corruption, accidental deletion, ransomware, or a bad migration can be copied to every replica immediately. Replication improves availability or reduces data-loss windows, but it does not by itself provide a clean historical recovery point.
Multi-zone versus multi-region: Zones reduce the risk of a datacenter or local infrastructure failure within one region. Multi-region design can protect against a larger regional disaster, but introduces latency, data-consistency, cost, and failover complexity.
Restore testing and game days: Regularly restore production-like data, verify application compatibility, and rehearse failover with clear authority and communication paths. A runbook that has never been exercised is an assumption, not evidence of recoverability.

Mental model

Treat Disaster Recovery, Backups, RPO/RTO, Multi-Zone/Region Design, and Game Days as a design problem with observable inputs, outputs, invariants, and failure modes. High availability handles many local failures while the system is still running. Disaster recovery addresses loss, corruption, or site failure and depends on tested restoration paths, not merely on keeping another copy of the current production data. A strong implementation makes assumptions visible, narrows uncertainty at system boundaries, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to support the claim that the design is safe.

A useful sequence for both interviews and production design is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement directly to a library call or a cloud-product checkbox. First state what must remain true. For example, specify how much data may be lost, how quickly service must return, and which operations may be unavailable during recovery. Then choose the mechanism that enforces those properties and define how you will verify it.

Deep dive

1. RPO

The problem behind RPO is that a recovery process may not be able to preserve every write made before a failure. Recovery Point Objective defines the maximum acceptable data loss, measured in time. An RPO of zero implies that the recovery design must preserve every committed write that the business considers committed; an RPO measured in minutes permits recovery to an earlier point within that window.

Lower RPO generally requires more frequent or continuous replication, log shipping, or change capture, along with stronger operational controls. It can also increase bandwidth, storage, coordination, and recovery complexity. Define the RPO per data set or business operation when appropriate rather than assuming that every piece of data needs the same guarantee.

Decision rule: Use RPO deliberately when it makes the recovery contract or invariant easier to prove. If the number only appears in a document but the system has no mechanism or measurement that can meet it, the design is overstating its reliability. Make the assumption explicit and choose a more observable design.

2. RTO

The problem behind RTO is that having recoverable data does not mean the service will be usable immediately. Recovery Time Objective defines the maximum acceptable time to restore service. The clock may include detection, incident coordination, infrastructure provisioning, data restoration, configuration changes, DNS or traffic changes, warm-up, and verification, so do not measure only the database restore command.

RTO drives the recovery architecture: automation, pre-provisioned or warm capacity, DNS and traffic failover, dependency readiness, and runbook practice all matter. A cold backup-based recovery may be inexpensive but slow. A warm secondary can reduce RTO while costing more continuously. The target should be measured in a production-like exercise, not inferred from a vendor's individual component metric.

Decision rule: Use RTO deliberately when it makes the recovery contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. State which parts of the service are restored first and what degraded behavior is acceptable during recovery.

3. Backup types

Backups are useful only when they can be located, read, restored, and used by a compatible application. A full backup contains the selected data set and is straightforward to reason about, but can consume more time and storage. Incremental backups record changes since a previous backup, while differential backups record changes since a full backup; their restore chains have different operational costs. Snapshots can be fast to create and restore within a storage system, but their portability and independence from the original account or region must be checked. Log shipping and point-in-time recovery, or PITR, can reduce the recovery point window by retaining a sequence of changes.

These approaches have different restore dependencies and retention behavior. Encryption protects backup contents, while off-account and off-region copies reduce the chance that one compromised identity, account, or site can destroy both production and recovery data. Retention should cover the business and operational recovery window, including the possibility that corruption is discovered long after it first occurred.

Decision rule: Use backup types deliberately when they make the recovery contract or invariant easier to prove. If a backup is described as available but its dependency chain, retention, encryption keys, or restore procedure is unknown, it is not a complete recovery design. Prefer the simplest combination that meets the RPO, RTO, and retention requirements and can be tested.

4. Replication is not backup

Replication is often introduced to keep another copy of the current data available. That helps with availability and can reduce data loss after some infrastructure failures, but it does not create an independent historical record. Logical corruption, an accidental delete, ransomware, or a bad migration can be accepted and replicated immediately. A replica can therefore be perfectly synchronized with the wrong state.

Retained backups, especially immutable or otherwise protected copies, create a different failure boundary. Recovery may still require choosing a safe point in time, restoring into an isolated environment, validating the result, and deciding how to reconcile writes made after that point. Those steps are why “we have a replica” is not an answer to a backup or disaster-recovery requirement.

Decision rule: Use replication and backup as separate mechanisms when each makes a different contract or invariant easier to prove. If replication is being used to hide the absence of historical, isolated recovery points, the design does not cover corruption or operator error.

5. Multi-zone versus multi-region

Zones reduce the risk of a datacenter, power, or local infrastructure failure while keeping the deployment within one region. They often provide lower latency and simpler consistency behavior than a cross-region design. They do not, however, protect against every regional service failure, a regional control-plane problem, a widespread configuration error, or a disaster that affects the region as a whole.

Multi-region design can protect against those larger failures, but the second region is not a magic copy of the first. Cross-region links add latency and cost. Data may be asynchronous or require coordination, which affects consistency and the achievable RPO. Traffic failover, writes during a partition, identity and key availability, dependencies, cache state, and failback all need explicit decisions. A secondary region that cannot run the application, access its secrets, or accept traffic is not a tested recovery site.

Decision rule: Use multi-zone versus multi-region deliberately when it makes the contract or invariant easier to prove. If the actual failure requirement is local and the service cannot justify cross-region consistency and failover complexity, multi-zone may be the better design. If a regional disaster is in scope, document the additional assumptions rather than treating zones as equivalent protection.

6. Restore testing and game days

A recovery plan is credible only when the team has demonstrated it. Regularly restore production-like data into an isolated environment, validate that the application version can read it, check migrations and indexes, and confirm that required secrets and dependencies are available. Measure the elapsed time and the recovered data point instead of recording only that a command succeeded.

Game days extend that practice to failure and coordination. Rehearse scenarios such as loss of a zone, loss of a region, corrupted data, an unavailable dependency, or an operator mistake. Define who can declare the incident, who can authorize failover, how customer-facing communication works, and how the system returns to normal. Untested runbooks decay as infrastructure, credentials, schemas, and ownership change.

Decision rule: Use restore testing and game days deliberately when they make the recovery contract or invariant easier to prove. If an exercise cannot be performed safely, narrow its scope and use an isolated environment, but do not replace it with a document review. The result should produce evidence, findings, and follow-up work.

Worked example

Consider a large-scale distributed order service. Its requirements, traffic, failure modes, cost, and operational constraints must be made explicit before choosing a recovery design. Start with one sentence such as: “The order service must continue accepting orders after the loss of one zone, and after a regional disaster it must restore service within the stated RTO while losing no more than the stated RPO.” Then list the input and output contracts and identify which of the concepts above owns each failure mode.

The useful design move is separation. Parsing and request validation belong at the boundary. Domain rules belong in the domain or service layer. Persistence and transaction rules belong in the database or repository. Presentation rules belong in the client. Mixing these concerns can make a happy-path demo shorter, but it makes recovery state, retries, malformed data, and edge cases much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

Walk through at least four cases: the normal path; an empty or missing value; a duplicate, retry, or concurrent path where relevant; and a dependency failure. For each case, state which layer detects the problem, whether the operation is safe to retry, what data may already have been committed, and what the caller observes. For a regional failure, also state which traffic path becomes active, which data point is available, and how the team knows the recovered service is safe to expose. This is the level of explanation expected in a senior code review or technical interview.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the recovery design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high cardinality. A backup can exist while its restore is incompatible with the current application. A failover can succeed at the DNS layer while a dependency, secret, queue, or database write path remains unavailable. Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after evidence identifies a bottleneck or risk.

When the topic involves an external dependency, define timeout and cancellation behavior, including what happens to work that was already sent when the timeout fires. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states, including what the user sees during degraded recovery. When it involves security, assume the client can be modified and network input is untrusted. Backup and recovery credentials should also be protected from the same compromise that threatens production.

Guided lab

Create DR for an order platform with explicit RPO/RTO targets, backup and PITR retention, a multi-zone database, a secondary-region strategy, DNS or traffic failover, restore verification, and quarterly game-day scenarios.

Complete the lab with this discipline:

Write the requirement and two non-requirements. For example, specify what must recover and what can remain unavailable or be rebuilt later.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing. Record restore duration, recovered point in time, and failover observations where applicable.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it, including its cost or operational trade-off.
Record a short “what would break at 10× scale?” note.

Edge cases and failure modes

RPO: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Verify what happens when the latest log segment or replication link is delayed, missing, or only partially available.
RTO: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Measure the complete recovery path rather than only one restore command, including traffic switching and readiness checks.
Backup types: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Test broken dependency chains, expired retention, unavailable keys, incompatible schema versions, and restores into an isolated environment.
Replication is not backup: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Include accidental deletion, logical corruption, ransomware, bad migrations, replication lag, and a replica that contains the same corrupted state.
Multi-zone versus multi-region: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Include zone loss, regional loss, network partition, stale DNS or traffic state, cross-region lag, dependency failure, and failback.

Common mistakes and debugging

Solving the example instead of the requirement: a copied architecture can be syntactically or operationally correct while still failing the actual RPO, RTO, cost, or data-consistency requirement.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values. In recovery work, an unknown state should be visible and handled explicitly.
Testing only the happy path and therefore discovering contracts only after integration. A successful backup job is not proof that a restore will work.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement. Cross-region replication and warm capacity have ongoing costs that need justification.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees. The recovery path must preserve the same security and data rules as the normal path.

For debugging, reproduce the smallest failing case, inspect the actual value or execution plan, and trace the boundary where the invariant first becomes false. Check backup-job logs, replication lag, object or snapshot retention, key access, restore logs, application and migration compatibility, DNS or traffic state, and dependency health. Then fix the owning layer rather than adding a downstream patch that merely hides the symptom.

Interview questions

What problem does RPO solve, and what trade-off or failure mode would make you choose a different approach?
What problem does RTO solve, and what trade-off or failure mode would make you choose a different approach?
What problem do backup types solve, and what trade-off or failure mode would make you choose a different approach?
What problem does the distinction that replication is not backup solve, and what trade-off or failure mode would make you choose a different approach?
What problem does multi-zone versus multi-region design solve, and what trade-off or failure mode would make you choose a different approach?

Checkpoint

Without notes, explain Disaster Recovery, Backups, RPO/RTO, Multi-Zone/Region Design, and Game Days to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code. Be prepared to explain how you would verify the claimed RPO and RTO rather than only naming them.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.