257: Replication: Leaders, Followers, Multi-Leader, Quorums, and Replication Lag

Learning outcomes

By the end of this lesson, you can:

explain and apply leader-follower in a realistic implementation;
explain and apply synchronous versus asynchronous in a realistic implementation;
explain and apply read-your-writes in a realistic implementation;
explain and apply multi-leader in a realistic implementation;
explain and apply quorum intuition in a realistic implementation.

These are design skills, not terms to recite. You should be able to connect a replication choice to a user-visible guarantee, a failure mode, an operational cost, and a way to verify that the system is behaving as intended.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before reading, retrieve one concrete example from a previous project where the same concern appeared. Perhaps a request wrote to one datastore and then read through a cache, or a background worker consumed data after the original request had returned. Identify what could be stale, what had to be durable, and what the user was promised.

The goal is not to memorize terminology. It is to make a defensible decision inside a large-scale distributed service. Make the requirements, traffic, failure modes, cost, and operational constraints explicit before choosing a replication pattern. A design that is correct for a profile page may be wrong for a payment ledger, even if both systems store records and serve reads.

Terminology

Leader-follower: Writes go to a leader, and the leader replicates changes to followers. Followers can serve some reads or provide failover capacity, but they may be behind the leader.
Synchronous versus asynchronous: With synchronous replication, a write acknowledgement waits for one or more replicas to confirm the change. This can improve durability or freshness guarantees, but it adds latency and can reduce availability when a required replica is unavailable. Asynchronous replication lets the leader acknowledge earlier, which reduces write latency, but acknowledged data may not yet exist on followers when the leader fails.
Read-your-writes: After a user successfully changes data, later reads in that user's session should show that change when the product requires immediate visibility. The request can be routed to the leader, use session stickiness or version tokens, or wait until a follower reaches a required replica position.
Multi-leader: Multiple nodes or regions accept writes. This can improve regional write availability and reduce write distance, but concurrent changes need conflict detection, resolution, and ordering rules. It is appropriate only when the domain can reconcile those changes safely.
Quorum intuition: A read or write quorum is a required number of replica responses. Under stated assumptions, choosing quorum sizes so the sets overlap means a read can encounter information from a write. Real systems differ: they may use sloppy quorums, hinted handoff, repair, leader-based ordering, or weaker consistency, and failures can violate the assumptions behind the simple arithmetic.
Replication lag: Replication lag is the distance between a source's committed position and a replica's applied position. Measure it in time, log positions, sequence numbers, or another system-specific unit. It is a concrete engineering condition that affects correctness and user experience, not merely vocabulary.

Mental model

Treat Replication: Leaders, Followers, Multi-Leader, Quorums, and Replication Lag as a design problem with observable inputs, outputs, invariants, and failure modes. Replication creates multiple copies of state. That can improve availability, read capacity, geographic reach, and recovery options, but it also means the copies can temporarily disagree about what is current. The design must say which copy accepts a write, when a write is considered acknowledged, which reads may be stale, and what happens when replicas cannot communicate.

For example, a profile service might require that a successful password change is durable before the response is returned, while allowing a public display name to appear on a follower a few seconds later. Those are different contracts. Do not hide both behind a vague promise that “the database is replicated.”

A strong implementation makes assumptions visible, narrows uncertainty at boundaries, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to show why the design is safe. Ask which invariant matters: must every acknowledged write survive one node failure, must reads be globally current, or must only the writer see its own update immediately? The answer determines how much coordination you need.

A useful interview and production sequence is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement directly to a library call. First state what must remain true. Then choose the mechanism that enforces it. Finally, name the observation that would tell you the mechanism is failing: increasing apply lag, rejected writes, conflicting versions, quorum timeouts, or reads returning a version older than the session's required version.

Deep dive

1. Leader-follower

Writes go to a leader, which establishes the write order and sends the changes to followers. Reads from followers can scale horizontally and can be placed near users, but they may be stale, especially immediately after a user's own write. A follower is not automatically a safe source for every read just because it contains a copy of the data.

The leader-follower model is useful when one ordered write path makes invariants easier to enforce. A service can send ordinary, latency-sensitive reads to followers and send reads that require current state to the leader. It may also use followers for reporting or search projections when a small delay is acceptable. The routing decision belongs in the service contract, not in an accidental connection-pool default.

There are costs. The leader can become a write bottleneck, failover requires a promotion and a way to prevent split-brain writes, and followers consume storage and replication bandwidth. A follower that is alive but far behind can be more dangerous than an obviously failed follower if the application quietly serves obsolete decisions from it. Track health and freshness separately.

Decision rule: Use leader-follower deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. State which reads can tolerate staleness, whether failover is automatic, and how clients learn that a new leader has been selected.

2. Synchronous versus asynchronous

The central question is not whether one mode is universally better. It is when the system is allowed to tell the caller that a write succeeded. With synchronous acknowledgement, the leader waits for confirmation from the required replica or quorum before responding. This can improve the chance that an acknowledged write survives leader failure and can provide a known freshness point for subsequent reads. It adds network round trips, and a required replica outage may turn a write into a timeout or rejection.

With asynchronous replication, the leader acknowledges after its local durability condition is met, while followers catch up afterward. That usually reduces write latency and allows the leader to continue when a follower is temporarily unavailable. The trade-off is an acknowledgement window: if the leader fails before the change reaches a surviving replica, the acknowledged write may be delayed, lost, or require recovery from another log, depending on the system.

Synchronous does not mean “all replicas are instantly identical,” and asynchronous does not mean “the write is unsafe.” The exact durability boundary depends on the database, storage layer, acknowledgement policy, and failure model. Document whether “success” means locally persisted, replicated to one other failure domain, or accepted by a quorum. Also decide what the client should do after an ambiguous timeout, because retrying can create a duplicate unless the operation is idempotent.

Decision rule: Use synchronous versus asynchronous deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Choose the acknowledgement point from the consequence of losing or delaying the write, then budget for its latency and availability impact.

3. Read-your-writes

A common confusing sequence is simple: the update request returns success, the page immediately fetches the record, and the old value appears. Nothing necessarily rolled back. The read may have gone to a follower that had not applied the leader's change yet. This is replication lag observed by a user.

Read-your-writes is the session-level guarantee that a user's later read does not move backward past a write that the same user has already observed as successful. The usual solutions have different costs:

Route the user's relevant reads to the leader. This is straightforward, but concentrates read traffic and can increase latency.
Keep a session sticky to a leader or a suitable follower. This can reduce surprises, but routing changes and failover still need a policy.
Return a version, timestamp, or log position from the write and require the next read to reach that position. A follower can serve the read once it has caught up; otherwise the service can wait or route elsewhere.
Return the updated representation from the write response and let the client use it temporarily. This improves the immediate UI but does not by itself guarantee that a later independent read is current.

The right choice depends on the product contract. A social feed may accept eventual visibility, while an account settings screen usually should not show the old email immediately after confirming a change. Be careful with retries and multiple devices: a session token can express what one client has observed, but it does not automatically provide global ordering across all clients.

Decision rule: Use read-your-writes deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Write the guarantee in observable terms, such as “after UpdateProfile returns success, GetProfile for that session returns a version at least as new as the acknowledged version.”

4. Multi-leader

Multi-leader replication allows more than one region or node to accept writes. It is attractive for a global service because users can write near their region instead of crossing a long network distance to one leader. It can also keep regional writes available during some inter-region failures.

The price is concurrent change. Two leaders can update the same record before either has seen the other update. The system then needs a conflict policy: last-write-wins, field-level merge, a domain-specific merge, a rejection requiring user intervention, or an operation-based approach such as an explicitly commutative update. Clock timestamps alone do not prove causality, and silently choosing the “latest” value can discard meaningful user data.

Multi-leader also complicates uniqueness, counters, ordering, deletes, and side effects. A globally unique username cannot be safely checked only against a locally current replica unless the design reserves names or coordinates the check. A replicated event that triggers an email or payment must be deduplicated, because conflict resolution and retries can cause delivery more than once. Regional partition behavior must be specified: accept divergent writes, reject writes, or limit the writable fields.

Decision rule: Use multi-leader deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Choose it only after demonstrating that concurrent writes can be reconciled without violating domain invariants, and define how conflicts are observed, repaired, and explained to operators.

5. Quorum intuition

Quorum reasoning starts with a simple replicated set. If a write must reach W of N replicas and a read consults R, then choosing W + R > N means the two sets overlap in the idealized model. The overlapping replica may contain the write, so a read can discover it. Choosing W > N / 2 also makes two successful writes overlap, which can help establish a single winner or detect a conflict.

That arithmetic is useful intuition, not a complete consistency proof. It assumes the replicas agree on versions, the read selects the newest valid response, failures are classified correctly, and the system does not route around the configured set. Sloppy quorums may count a temporary substitute node. Hinted handoff may postpone delivery. Read repair may fix a stale replica only after a read. A leader-based database may use the word quorum while providing semantics that differ from a leaderless key-value store.

Quorum size also does not answer what happens during a partition. A larger required quorum can preserve stronger guarantees by rejecting more operations; a smaller quorum can preserve availability while allowing more stale or conflicting results. Always pair the numbers with a version-selection rule, timeout behavior, repair process, and failure-domain placement. Three replicas in one failure domain are not equivalent to three replicas across independent zones.

Decision rule: Use quorum intuition deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Treat N, R, and W as inputs to a stated failure and consistency model, not as a magic formula that guarantees safety by itself.

6. Replication lag

Replication lag is the gap between a source and a replica. Depending on the system, measure it as elapsed time, a log sequence-number difference, a commit timestamp difference, or the number of unapplied operations. A replica can be reachable and returning successful queries while still being too stale for a particular request.

Lag affects more than dashboards. A stale index or cache can make a newly created item appear missing. A follower used for authorization or inventory can make an already-revoked permission or already-sold item appear valid. A user may see an old profile immediately after editing it. These are different consequences and should not all be solved by simply “waiting longer.” Route critical reads appropriately, carry a version requirement, or change the acknowledgement contract.

Measure both the typical and worst-case lag, and alert on the values that violate a product or safety objective. Inspect whether lag is caused by network delay, a slow apply process, oversized transactions, lock contention, storage pressure, or an unhealthy replica. During an incident, distinguish a replica that is unavailable from one that is serving stale data; the mitigation and the risk are different.

Decision rule: Use replication lag deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Define the maximum tolerable staleness for each read path and make the system fail safely when that bound cannot be met.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be explicit. Suppose the service stores user profiles globally. A reasonable initial contract might allow follower reads for a public profile view, require read-your-writes after a profile edit, and require a stronger acknowledgement rule for security-sensitive fields. Those requirements lead to different read routes and should not be flattened into one “read from a replica” rule.

Start by writing the requirement in one sentence, list the input and output contracts, and identify which concept above owns each failure mode. For example, the input may be a validated profile patch and an authenticated user; the output may include the updated profile version; errors may distinguish invalid input, authorization failure, dependency timeout, and an ambiguous write result. A version returned by the leader can let the next read wait for or route around a lagging follower.

The important move is separation: parsing or validation belongs at the boundary; domain rules belong in the domain/service layer; persistence and replication rules belong in the database or repository; presentation rules belong in the client. Mixing these concerns makes a happy-path demo look shorter, but it makes edge cases much harder to reason about. A client-side optimistic update can make the screen look current, but it cannot replace server-side authorization or a durable write.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

In a replicated implementation, “Primary datastore” represents the write leader and its replication topology. The cache and workers may introduce additional freshness boundaries. A cache invalidation event can be delayed, and a worker consuming an asynchronous stream may observe the profile after the write response. That is acceptable only if each consumer's contract says so. Do not infer database freshness from the freshness of the API process.

Walk the example with at least four cases:

Normal path: The authenticated client sends a valid edit. The service applies the domain rules, writes through the leader, receives the configured acknowledgement, and returns the new version. A subsequent read either uses the leader or requires a follower to be at least that version.
Empty or missing value: Decide whether an omitted field means “leave unchanged” and whether an explicit empty value means “clear it.” Validate that distinction at the boundary and preserve it in the update command; do not let a serialization default accidentally turn both cases into the same operation.
Duplicate, retry, or concurrent path: A client retries after a timeout, or two regions edit the same profile. Use an idempotency key or conditional version where appropriate. In a multi-leader design, apply the documented conflict policy rather than silently assuming arrival order is global order.
Dependency failure: A follower is reachable but exceeds the allowed lag, the leader fails after accepting a write, or the inter-region link is partitioned. State whether the service rejects the operation, routes to another replica, returns an explicit stale result, or exposes an indeterminate outcome for reconciliation.

For each case, state which layer detects the problem and what the caller observes. This is the level of explanation expected in a senior code review or technical interview. Also name the evidence you would inspect: request and replication IDs in logs, acknowledgement latency, follower apply position, cache age, queue delay, and the version returned by the read. Without those observations, “the replica is probably behind” is a guess rather than a diagnosis.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high cardinality. Replication adds operational work: promotion, failback, resynchronization, lag monitoring, backup recovery, and protection against split brain. A failover plan that has never been exercised is an assumption, not a guarantee.

Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after you can identify the bottleneck or risk with evidence. Measure write acknowledgement latency, read latency by consistency route, replica lag, quorum timeout rate, conflict count, repair backlog, and the age of data served from caches or projections. Include failure-domain placement in the capacity and cost model.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and the network input is untrusted. In particular, do not use a lagging replica as the sole authority for an authorization decision unless the security contract explicitly accounts for revocation delay.

Guided lab

Design replicas for a global user profile service. Specify which reads may use followers, how read-your-writes works after a profile edit, and what happens during leader failure or regional partition. Your design should distinguish a public profile read from a security-sensitive or account-management read, and it should say what the client sees when the consistency guarantee cannot be met.

Complete the lab with this discipline:

Write the requirement and two non-requirements. For example, state whether public profile reads may be briefly stale, and explicitly state a feature the service does not promise, such as globally ordered edits across all regions.
List input, output, and error contracts before implementation. Include the acknowledged version or position if the read path will use it.
Implement the smallest correct vertical slice. Start with one leader and one follower route before adding multi-leader conflict handling.
Add at least one invalid-input test and one edge-case test. Useful edge cases include an edit followed immediately by a follower read, a retry after an ambiguous timeout, or a follower that exceeds its freshness bound.
Instrument or inspect the behavior instead of guessing. Record which replica served the read, the observed version, and the lag at the time of the request.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration. A consistency requirement or maximum acceptable lag should not exist only in a comment.
Explain one alternative design and why you did not choose it. Compare, for example, leader-only reads with version-aware follower reads, including their latency, availability, and operational costs.
Record a short “what would break at 10× scale?” note. Consider leader write capacity, cross-region bandwidth, cache invalidation volume, failover duration, and the number of metrics or conflicts operators must inspect.

Edge cases and failure modes

Leader-follower: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Also test leader promotion, a follower that is alive but stale, and a client that has cached the old leader endpoint.
Synchronous versus asynchronous: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Verify the result when a required acknowledgement times out and when the caller retries an operation whose outcome is unknown.
Read-your-writes: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Verify that the version or routing token cannot be silently discarded between the write response and the next read.
Multi-leader: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Include concurrent edits to the same field, edits to independent fields, deletes racing with updates, and a partition followed by conflict resolution.
Quorum intuition: Test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Exercise different replica failures, timeouts, sloppy-quorum behavior if supported, and the case where responses overlap but carry different versions.

In every case, define the invariant first. “The request returned 200” is not enough to establish that a write was replicated, that a read was current, or that a conflict was resolved safely.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong. A follower read is not a solution until its allowed staleness is known.
Treating synchronous acknowledgement as a guarantee that every replica is current, or treating asynchronous acknowledgement as proof that data is disposable. Inspect the actual durability and acknowledgement semantics of the chosen system.
Assuming quorum arithmetic alone provides strong consistency. Check version selection, failure domains, sloppy quorum behavior, repair, and partition handling.
Testing only the happy path and therefore discovering contracts only after integration. Include lag, failover, retry, duplicate, and concurrent cases.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values. An ambiguous write outcome should be represented and reconciled, not converted into a false success.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement. A multi-leader topology adds conflict and operational cost that may not be justified.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees. An optimistic UI is not evidence that the backend has accepted or durably stored the change.

For debugging, reproduce the smallest failing case, inspect the actual value or execution plan, and trace the boundary where the invariant first becomes false. Record the write version, the replica that served the read, that replica's applied position, cache age, and relevant queue delay. A successful write followed by an old read usually points to routing or apply lag; a missing write after failover requires checking the acknowledgement boundary and recovery log. Fix the owning layer rather than adding a downstream patch that merely masks stale state.

Interview questions

What problem does Leader-follower solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Synchronous versus asynchronous solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Read-your-writes solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Multi-leader solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Quorum intuition solve, and what trade-off or failure mode would make you choose a different approach?

Answer each question with a requirement, an invariant, a concrete failure mode, and an observation or metric you would use to verify the decision. Naming a pattern without naming its consistency and availability costs is not a complete design answer.

Checkpoint

Without notes, explain Replication: Leaders, Followers, Multi-Leader, Quorums, and Replication Lag to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code. Your example should make its acknowledgement point and stale-read behavior visible rather than leaving them implicit.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.
I can state what a successful write guarantees and what a follower read is allowed to return.
I can diagnose lag, failover, retry, and conflict behavior using observable evidence.