140: MongoDB Replica Sets and High Availability — Elections, Oplog, Failover, Read Scaling, and Recovery Semantics

Learning objectives

You will learn to:

understand replica-set architecture;
distinguish primary and secondary members;
understand replication oplog at a practical level;
understand elections;
understand failover behavior;
understand rollback risk;
understand majority commit point;
understand read preference trade-offs;
plan application behavior during failover;
understand connection-string discovery;
understand hidden/delayed/arbiter concepts at a high level;
avoid using replication as backup.

The goal is not to memorize a list of replica-set settings. You should be able to look at a deployment, explain where writes go, reason about what happens when a member disappears, and choose read and write behavior that matches the application's correctness requirements.

Replica set mental model

Start with the smallest useful model. One member is normally the primary. It accepts writes and records the operations that must be replicated in the replica set's oplog. Eligible secondary members follow that log and apply the operations locally.

Primary
  │ writes
  ↓
oplog
 ↙   ↘
Secondary A
Secondary B

Clients normally send writes to primary. A driver may also select a primary for reads because primary reads are the default behavior.

Secondaries replicate primary operations. They are not independent databases receiving unrelated application writes; their normal role is to reproduce the primary's state by applying the replicated operations.

If primary unavailable, eligible members can elect a new primary. That change is not instantaneous. During the election, the application can see a period in which no member is available for ordinary primary writes, and the driver must discover the new topology.

Why replication exists

Replication provides several related, but distinct, benefits:

high availability;
redundancy;
failover;
disaster-resilience building block;
read distribution for suitable workloads.

High availability means the service can continue after a member failure, provided the remaining members and voting configuration can support a new primary. Redundancy means multiple members hold the replicated data. Failover is the transition to a replacement primary. These properties help with node failures, but they do not remove the need to plan for larger failures or logical mistakes.

Replication is not backup.

If application deletes all records, deletion replicates too. A replica is designed to converge on the same current state as the primary, including an incorrect update, an accidental delete, or a destructive migration. A backup or point-in-time recovery system gives you a way to go back to an earlier state; replication by itself does not.

Primary

The primary accepts writes under normal replica-set operation. For an application using the default read preference, reads also go to the primary. This gives the application a simple consistency model: the same member processes the write and the subsequent ordinary read.

Only one primary at a time per replica set. Election and voting rules exist partly to prevent two partitions from both behaving as valid primaries. When leadership changes, the old primary must stop accepting primary-only work before the replacement can take over.

Secondary

A secondary replicates the oplog and applies operations to its own data files. It can serve reads when read preference permits, which is useful for carefully selected workloads such as reporting or geographically distributed read traffic.

A secondary may lag. The useful distinction is between a member that is healthy but slightly behind and a member that is so far behind that it can no longer use the available oplog history. Both are operationally relevant, but they require different responses.

Do not assume secondary is perfectly current. A secondary read can return a state that existed on the primary earlier, so routing a request to a secondary is an application-level consistency decision, not merely a free performance optimization.

Oplog

The replica set maintains a capped operation log. This oplog records the operations that secondaries need to follow, and secondaries read from it as they apply changes.

The oplog window represents how much history is retained. Think of it as the time span covered by the oldest available oplog entry through the newest one, not as a permanent archive. The window changes with write volume and the configured oplog capacity. A busy system can consume the same amount of history much faster than a quiet system.

If secondary falls too far behind beyond oplog history, it may need initial sync rather than catching up incrementally. In that case, the member must obtain a fresh copy of the data and then resume replication. Initial sync takes resources and can change the recovery or maintenance timeline, so the oplog window is an operational metric rather than an implementation detail.

Monitor oplog window for recovery/maintenance expectations. A window that is too short for the time required to repair or restart a member increases the chance that a normal catch-up will no longer be possible.

Replication lag

Replication lag is the difference between primary progress and secondary apply time. It can mean that a secondary has received an operation but has not applied it yet, or more generally that its replicated state trails the primary's state.

Common causes include:

network;
slow disks;
heavy writes;
long operations;
resource saturation.

Secondary read may be stale. The amount of staleness depends on the workload and the member's current lag, so do not infer freshness from the fact that the member is reachable.

Monitor. Look at lag together with disk, CPU, network, and write activity. A rising lag trend is often more useful than a single measurement because it tells you whether the member is catching up or falling further behind.

Election

When primary unavailable, members coordinate election. Eligible members use the replica-set voting process and their configuration to determine whether a new primary can be chosen. The exact election timing depends on the deployment and current conditions, so application code must handle the transition rather than assume a fixed duration.

During election:

writes can temporarily fail/pause
driver discovers new primary
retryable operations may retry

Applications must expect transient errors. A request may fail because the old primary became unavailable, because server selection is still discovering the replacement, or because an operation was interrupted during the leadership change. Drivers and retry settings can reduce the visible impact, but they do not make every operation automatically safe to repeat.

Do not treat a 1-second failover as “database corruption.” A short period without a primary is an expected availability event in a replica set. The right response is to inspect client errors, retry behavior, election metrics, and the resulting data semantics rather than to conclude that the data is corrupt from the pause alone.

Driver discovery

Use replica-set/Atlas connection string listing/discovering topology. The connection string should give the driver enough information to find the members and identify the current primary, rather than making the application responsible for hard-coding leadership.

The driver monitors cluster. It updates its view as members change state and uses that topology information when selecting a server for an operation.

Do not pin application permanently to one node IP. If that node is stopped, steps down, or becomes unreachable, an application with a single fixed endpoint may fail even though another eligible member is healthy.

Example Atlas SRV:

mongodb+srv://cluster...

Driver selects server based on operation/read preference. Writes that require the primary are directed accordingly, while read preference can allow a suitable secondary or another member to serve a read. The driver still needs reasonable selection and operation timeouts so the application can tolerate the normal discovery interval during a failover.

Majority

A write acknowledged with majority means majority of voting data-bearing members acknowledge according to replication semantics. It is not simply a promise that one server wrote the bytes locally. The acknowledgement depends on the voting configuration and the deployment's replication and storage behavior.

Majority commit point relates to durability/visibility. It identifies progress that has been acknowledged by the required majority and is therefore less exposed to a later leadership reconciliation than an operation known only to the former primary.

This reduces rollback exposure. It does not mean the application can ignore backups, network failures, or every durability detail. A write concern choice should follow the business consequence of losing or temporarily not observing a write.

Exact behavior depends on deployment/storage. Check the MongoDB version and current deployment documentation when the guarantee matters, especially across managed services, storage configurations, and unusual voting layouts.

Rollback

If former primary accepted write not majority committed and loses leadership, conflicting operations may be rolled back during reconciliation. The write may have been acknowledged by that member, yet not have the replication support required to survive a change in leadership. When the members reconcile their histories, the branch that is not retained can be rolled back.

This is why write concern matters for durability requirements. Choose acknowledgement semantics based on whether the operation is disposable, retryable, or business-critical. Also make the client behavior explicit: a retry after an uncertain result must not create a duplicate side effect merely because the original outcome is unknown.

Read concern majority

Reading majority helps avoid seeing data not majority committed. This can reduce the chance that a read observes a value that could later disappear as part of rollback, although it is a consistency choice with system costs.

Trade latency/availability. A stronger read guarantee can wait for more replication progress and may be less available when the required members cannot communicate.

Use business requirements. A dashboard may accept a different freshness and availability profile from an authorization decision, an inventory check, or a payment workflow.

Failover test

Production readiness requires testing the entire client and operational path, not just proving that a replacement primary can be elected:

sustained writes;
kill/step down primary;
observe client errors/retries;
new primary election;
verify no duplicates;
verify latency;
verify monitoring alerts.

Sustained writes expose behavior that a quiet-cluster test can miss. Killing or stepping down the primary creates the transition. Client errors and retries show what the application actually experiences. After the new primary is elected, verify both data outcomes and operational signals: a request that is technically successful but duplicated is still a production failure.

Do not discover failover behavior first time in outage. Run the test in a controlled test cluster, document expected transient errors, and make sure the test does not accidentally target production data.

`rs.stepDown()`

An administrative command can trigger primary stepdown for maintenance/testing. It is useful for deliberately exercising the election path without waiting for a hardware failure.

Use controlled environment and current docs. Command behavior, options, permissions, and timing can vary by MongoDB version and deployment.

Do not run blindly on production without operational plan. Confirm authorization, traffic impact, monitoring coverage, rollback procedures, and the expected client behavior before using an administrative stepdown.

Secondary reads

Secondary reads can improve:

analytics/reporting;
geographically closer reads;
primary load.

Those benefits apply when the endpoint can tolerate the member's freshness and consistency characteristics. Reporting often has that tolerance; a user-facing workflow that immediately follows a write often does not.

But secondary reads can harm:

read-after-write;
authorization;
inventory;
real-time queue.

For example, a user may create a record on the primary and then be sent to a secondary that has not applied the record yet. An authorization check against stale data can make a valid user appear unauthorized, and an inventory read can temporarily show stock that has already been reserved.

Match endpoint semantics. Choose read preference per workload or operation where the driver and application architecture allow it, rather than making every read a secondary read for a vague promise of scale.

Nearest

Read preference nearest selects based on latency window/topology, not necessarily geographically “nearest” in simple terms. It can choose among eligible members according to measured network latency and topology rules, so the selected server is not necessarily the member in the same city or availability zone that a human would call nearest.

It can return secondary data. That means the lower-latency result may still be stale relative to a recent primary write.

Do not use for correctness-critical reads solely for latency. If correctness dominates, define the read concern and read preference from that requirement first, then measure whether the resulting latency is acceptable.

Hidden member

A hidden member is a replica-set member hidden from normal client reads. Hiding it prevents ordinary topology selection from sending application reads there, while allowing the member to serve a specialized operational role.

It can be used for dedicated backup/reporting in some designs. Whether that is appropriate depends on the backup method, workload isolation, resource capacity, and the service's current recommendations.

Still participates replication/elections depending votes/priority configuration. Hidden does not mean detached from the replica set, and it does not by itself determine whether the member can vote or become primary.

Use current guidance. Configure votes, priority, hidden status, and workload purpose together; an apparently harmless setting can affect election behavior and durability.

Delayed member

A delayed member intentionally delays replication. It is deliberately kept behind the primary so that an operator may have a recovery window against some logical errors before the bad operation reaches that member.

It can provide recovery window against some logical errors. This is a specialized recovery technique, not a guarantee that every mistake can be undone cleanly.

But not substitute for backups and can complicate elections/capacity. A delayed member is behind by design, consumes resources, and must be excluded from inappropriate read or election roles. Keep independent backups and point-in-time recovery as the actual recovery strategy.

Arbiter

An arbiter votes but stores no data. Its vote can affect whether a majority exists, but it cannot provide a data-bearing copy after a member failure.

Modern production architecture often prefers data-bearing voting members for redundancy. More data-bearing members generally provide stronger redundancy than adding a vote-only member, assuming the placement and operational budget support them.

Use arbiters only when architecture/guidance supports; understand durability implications. A configuration that can elect a primary with an arbiter still may not have enough data-bearing copies to satisfy the durability goal you intended.

Three-member replica set

A common baseline is:

3 data-bearing members

This setup can tolerate one member failure and still maintain majority, assuming the remaining members can communicate and the voting configuration permits the required majority.

Placement across failure domains matters. Three members on different hosts do not necessarily mean three independent failure domains.

Three members in same physical failure zone do not protect zone outage. If a rack, availability zone, or facility fails and all members are there, the replica set can lose both data availability and the ability to form a majority at once.

Failure domains

Place members across:

availability zones;
racks;
regions where latency allows.

The choice depends on the failure you are designing against. Availability zones or racks can protect against local infrastructure failure; regions can improve geographic resilience but introduce greater network distance and operational complexity.

Trade-off:

geographic resilience
vs
write latency

Majority write across far regions costs more latency. If a business requires majority acknowledgement before a write is considered complete, the cross-region network path becomes part of the write's latency budget. Design the voting layout and write concern together instead of evaluating placement in isolation.

Network partitions

Replica-set voting prevents both partitions from independently having valid primary if majority rules work correctly. A partition that cannot establish the required majority should not continue as a valid primary simply because it still has a local copy of the data.

The minority side loses primary ability. That protects the system from accepting competing writes in both partitions, but it also means clients connected only to that side can lose write availability.

This prioritizes consistency. It is the reason a network partition is not treated as an ordinary brief server restart: the system must protect the single-primary model while connectivity is uncertain.

Clients on minority partition may lose write availability.

Distributed systems trade-offs are real. A design cannot simultaneously guarantee unrestricted writes in every isolated partition and guarantee that the partitions never produce conflicting primary histories.

Write availability

To elect primary/ack majority, enough voting/data-bearing members must communicate. The required members depend on the replica-set configuration and the operation's write concern.

If majority unavailable, writes can stop. That is an intentional consequence of protecting majority-based durability and primary authority, not evidence that the database is randomly refusing work.

Do not promise 100% write availability under any partition. State the expected behavior in the application's reliability design and make sure callers can handle temporary failures or queued work where appropriate.

Read availability

Depending read preference/read concern, some reads may continue from secondaries. A read preference that permits secondary reads can preserve some read service while the primary is unavailable, provided an eligible secondary is reachable.

But serving stale reads may violate product semantics. A page showing an older report may be acceptable; an authorization, inventory, or queue decision may not be.

Availability is operation-specific. Describe availability per operation, including its acceptable freshness and consistency, instead of labeling the entire database simply “available” or “unavailable.”

Maintenance

A rolling restart generally follows a sequence like this:

secondary restart
catch up
next secondary
primary stepdown/restart

Restart one secondary, wait for it to return and catch up, then continue with the next member. Handle the primary deliberately with a planned stepdown or the managed service's supported maintenance process.

Managed services automate much. They do not remove the need to understand what the service is changing, which failure domains are involved, and what your clients observe during the operation.

Check replication lag before proceeding. A member that is already far behind may not recover quickly enough to provide the safety margin you expect during a rolling operation.

Backup from replica

Backups often use secondary/snapshots to reduce primary impact. Moving backup work away from the primary can protect foreground write and read capacity, but it does not automatically make an arbitrary snapshot consistent or restorable.

Still require consistency-aware snapshot method. Use the backup mechanism supported for the MongoDB deployment and verify that the resulting backup has the required point-in-time and transaction semantics.

Do not copy live database files arbitrarily. Files copied while the database is changing may not form a usable, consistent backup, and a secondary copy is not a substitute for a documented restore test.

Connection pool during failover

The driver pools connections to topology. It does not simply maintain a permanent pool to whichever server happened to be primary when the process started.

Failover can cause transient pool churn/selection delays. Existing connections may become unusable, server selection may wait while the driver learns the new primary, and operations in flight may report errors.

Set reasonable server selection/operation timeouts. The values should reflect the expected election and network behavior, the request's deadline, and whether the caller can retry safely.

Do not set 100ms timeouts if elections can exceed that. An unrealistically short timeout can turn an expected election into a stream of application-level incidents, even though the database recovers normally.

Health checks

Application readiness should consider database availability without causing restart loops. A readiness failure can remove an instance from traffic; a liveness failure can cause the process to be restarted. Those are different consequences and should not be triggered by the same shallow check.

Liveness should not kill process simply because Mongo primary is electing. If every application instance restarts during the same brief election, the health check amplifies the database event into an application outage.

Use operation-aware readiness. Decide which database operations the application must be able to perform before it accepts traffic, and distinguish a temporary primary election from a permanently unhealthy process.

Monitoring

Track:

member state
replication lag
oplog window
elections
primary changes
connections
disk
CPU
cache
write latency
majority commit lag

These signals answer different questions. Member state and primary changes show topology transitions. Replication lag, oplog window, and majority commit lag show how far replicated progress has fallen behind. Connections, disk, CPU, cache, and write latency help explain why lag or selection problems are developing.

Atlas provides managed metrics/alerts. Configure alerts around the actual operating thresholds and connect them to a response procedure; collecting a metric without knowing what action it should trigger is not enough.

Replication and indexes

Indexes exist on members. The replica members need compatible index state so that reads and replicated operations behave as expected on each member.

Index builds consume resources across replica set. They can compete with application work and replication for CPU, disk, memory, and I/O, and the effect is not limited to the member receiving a client request.

Plan production index changes. Check the supported build behavior for the MongoDB version and deployment, watch replication health while the change runs, and avoid treating an index rollout as operationally free.

Replication and TTL

TTL deletes execute/replicate according to Mongo internals. The expiry process is asynchronous and is not a precise per-document timer.

Do not depend on exact deletion timestamp. If business behavior needs an exact retention boundary, model that requirement explicitly rather than assuming a TTL delete occurs at the precise instant a field reaches its expiry value.

Replication and transactions

Transactions rely on replica-set/sharded deployment. A standalone Mongo has different transaction capability limitations, so a local setup that works for ordinary CRUD may not exercise the topology-dependent behavior your production transaction code depends on.

Use production-like topology in integration tests for transactions/change streams. This lets tests cover session, election, retry, and topology behavior that a standalone process can hide.

Local development

A single standalone is easy but hides:

transaction topology;
failover;
retryable behavior;
change streams.

That makes a standalone useful for basic development, but insufficient as the only environment for features whose behavior depends on replica-set topology.

For advanced tests, run local replica set/container or Atlas test cluster. Keep the test environment isolated, use test data, and exercise the same driver configuration patterns that the application will use in deployment.

Disaster scenarios

Replica protects node failure. It gives the system another copy and can support failover when one member or host is lost.

It does not protect:

operator drops database;
bad migration;
ransomware with DB credentials;
corrupted app writes;
long-undetected logical error.

Each of these can be replicated to otherwise healthy members. An operator can issue a destructive command against every member through replication; compromised credentials can do the same; and an application bug can write consistently wrong data everywhere.

Backups/PITR needed. Point-in-time recovery is what lets the team select a state before the logical error, while tested restores demonstrate that the backups are actually usable.

Failure clinic

Common design and debugging failures include:

replication = backup misconception;
secondary used for auth check;
no failover tests;
app connects single hostname/node;
low timeout makes every election incident severe;
all members same failure domain;
arbiter chosen without durability reasoning;
no oplog/lag monitoring;
liveness restarts app during DB election;
standalone dev hides transaction topology behavior.

When investigating an incident, classify it first: topology discovery, election, replication lag, stale-read semantics, resource pressure, or data-recovery failure. That classification points you toward the relevant driver logs, replica-set metrics, write/read concern, and backup evidence instead of treating every database error as the same problem.

Exercises

Draw 3-member replica set.
Explain oplog.
Choose read preference for analytics versus checkout.
Simulate primary stepdown in test cluster.
Measure client retry behavior.
Calculate failure-domain placement.
Design monitoring alert for lag.
Explain why logical delete survives replicas.
Build readiness policy during election.
Document write concern/read preference matrix.

Work through the exercises in order. The first activities establish the member roles and replication path. The middle activities connect read preference, stepdown, retries, placement, and monitoring to observable behavior. The final activities require you to reason about logical failure and to document application policy rather than relying on a database default.

Mastery checklist

Explain:

primary/secondary;
oplog;
election;
failover;
lag;
majority;
rollback;
read preference;
failure domains;
partitions;
driver topology;
replication versus backup.

You should be able to explain each term in the context of a real application: where a request is routed, what can become stale or unavailable, which metric would reveal the problem, and what recovery mechanism applies. If you can name a feature but cannot describe its failure behavior, the topic still needs more practice.