143: MongoDB Backup, Recovery, Monitoring, Profiling, Maintenance, and Operational Diagnostics

Learning objectives

You will learn to:

• distinguish replication from backup;
• understand logical backup with mongodump/mongorestore;
• understand snapshot backups and point-in-time recovery;
• design recovery objectives;
• test restores;
• use monitoring metrics;
• use database profiler/current operations carefully;
• inspect slow queries;
• understand connection capacity;
• understand maintenance/index operations;
• plan version upgrades and feature compatibility;
• write an operational runbook.

These topics fit together around one operational question: if a MongoDB deployment is damaged, slow, full, or in the middle of an upgrade, can the team explain what is happening and recover it safely? The commands are useful, but the decisions around consistency, timing, access, and verification are what make them reliable in production.

Recovery starts with requirements

Before choosing a backup mechanism, establish what the business can tolerate. A backup that runs successfully but cannot meet the recovery deadline is not a suitable backup design.

RPO

Recovery Point Objective:

How much data loss is acceptable?

RPO describes how far back the recovered data may be compared with the moment of failure. It is usually expressed as a duration, not as a vague promise that backups exist.

Example:

RPO 5 minutes

An RPO of five minutes means the recovery process must be designed so that losing more than roughly five minutes of accepted changes is not expected. The actual mechanism might use frequent snapshots, continuous history, or another supported approach.

RTO

Recovery Time Objective:

How long may recovery take?

RTO is the maximum acceptable time from the recovery event to a usable service. It includes more than copying database files: provisioning or selecting the target, restoring data, validating it, changing application routing, and confirming that the application works all consume time.

Example:

RTO 30 minutes

Backup design must satisfy both. A solution can have an excellent recovery point but still fail the RTO if the restore process is too slow or has not been practiced.

“Daily backup” is not enough information. You still need to know which point in the day is captured, how much data can be lost, where the backup is stored, how quickly it can be restored, and whether the result has been verified.

Replication is not backup

Replication keeps multiple live members aligned with changes. It improves availability when a member fails, but it does not preserve a historical copy of every correct state.

Replica set mirrors changes.

Accidental:

db.orders.deleteMany({})

That valid delete replicates to the other members. The replica set has done exactly what it was configured to do, even though the operation was a serious business error.

Backup provides historical recovery. It gives you a way to return to an earlier state after a destructive command, bad migration, compromised credential, or other logical failure. Replication and backup solve different failure modes, so one cannot be counted as a substitute for the other.

Backup types

Logical

mongodump

Produces BSON/metadata dump.

Good for:

• smaller deployments;
• migration/export;
• selected DB/collections.

Logical backups read database content and write BSON data plus the metadata needed to restore it. They are convenient when you need a selected database or collection, or when moving data between environments.

Can be slow/large for huge DB. The tool must read and write the logical contents, so its duration and storage requirements can become significant as the dataset grows. Measure it against the RTO instead of assuming that a successful dump is fast enough.

Physical/snapshot

Storage-level or Atlas snapshots.

Physical or storage snapshots capture the underlying storage through a mechanism supported by the database and storage platform. They are generally efficient for large datasets because they avoid rewriting every logical document into a new dump format.

Need consistency-aware supported mechanism. A raw copy of files taken at an arbitrary moment is not automatically a usable backup, particularly while the server is changing data. Use the procedure documented for the deployment and storage system.

Point-in-time recovery

Uses snapshots + continuous oplog-like history to restore near target timestamp.

Point-in-time recovery, or PITR, combines a base snapshot with a continuous history of changes. That lets the operator choose a recovery point near the incident rather than being limited to the timestamp of the last full backup.

Atlas offers managed PITR depending tier/configuration. Confirm the exact retention, supported target range, and configuration for the Atlas tier in use; the feature is not identical across every deployment.

mongodump

Example:

mongodump \
  --uri="$MONGODB_URI" \
  --out="./backup"

Do not put secret URI directly in shell history when avoidable. Prefer the deployment's secret-management and credential-handling practices, and make sure the account used for backup has only the access it needs.

Protect output. A dump can contain the full contents of the selected data, including personal or otherwise sensitive information. Restrict filesystem and object-storage access, encrypt it as required, and avoid leaving temporary copies on an unsecured workstation.

mongorestore

mongorestore \
  --uri="$TARGET_URI" \
  "./backup"

Restore into isolated environment first. The isolated target gives you a place to verify that credentials, versions, indexes, data, and application assumptions all work before any production cutover.

Do not test restore by overwriting production. A restore test should be repeatable and disposable; it should not create a second incident or destroy the very data you may need to recover.

Namespace selection

Dump selected DB/collection can reduce scope. That can save time and storage when the recovery or export really is limited to a well-defined namespace.

But recovery consistency may require related collections together. An application's business state often spans several collections, even if a single collection appears to be the one that failed.

Example:

orders
payments
outbox

restored from different times can violate business invariants. For example, an order might be restored without the payment state that was associated with it, or an outbox entry might be replayed against a state that did not yet exist.

Design backup unit. Choose the smallest unit that is still consistent for the workflows the system must recover, rather than selecting namespaces only by convenience.

Consistent backups

On replica sets/shards, use supported procedures to ensure consistent point-in-time backup. Sharded systems add coordination concerns because related data may live across shards; the backup process must account for the deployment as a whole.

Do not copy data directory while server is running without supported snapshot mechanics. Files captured while writes, checkpoints, or other storage-engine activity are in progress may not form a recoverable, coherent backup.

Atlas managed backups reduce complexity. They do not remove the need to understand retention, access, restore targets, recovery timing, and restore testing.

Backup encryption

At rest and transit. Encrypt stored backups and protect transfers between the database, backup system, and restore environment. Encryption in transit does not replace encryption at rest, and vice versa.

Keys separate/protected. Keep key access separate from ordinary database access where the platform and policy allow it, and make sure the recovery process documents how authorized operators obtain the keys.

Backup access can be more dangerous than live DB because it contains broad historical data. A backup may include records that have since been deleted or changed, so treat it as a high-value data store with its own auditing, authorization, retention, and deletion requirements.

Retention

Example:

hourly 48h
daily 30d
monthly 12m

Depends compliance/business. Retention should reflect the period in which logical mistakes may be discovered, regulatory requirements, customer commitments, and the cost of retaining the data.

Retention costs storage and privacy obligations. More copies improve the chance of finding a usable recovery point, but they also increase cost and the amount of sensitive data that must be protected.

Expired data should be deleted from backup according to policy/legal requirements where feasible. Document exceptions when immutable storage or legal holds prevent immediate deletion.

Restore testing

A backup not tested is an assumption. Successful backup-job status proves that a job ran; it does not prove that the data can be restored, that the credentials still work, or that the restored service meets its deadline.

Scheduled drill:

1. select backup;
2. restore isolated;
3. run integrity checks;
4. run app smoke tests;
5. measure time;
6. document failures;
7. destroy test environment safely.

The integrity checks should cover both database-level consistency and important business invariants. Application smoke tests should exercise the paths that matter after recovery, not just confirm that a port is open.

Track actual RTO. Record the backup age, restore duration, validation duration, cutover steps, and any manual intervention. Those measurements expose whether the stated RTO is realistic.

Point-in-time recovery drill

Simulate:

bad deploy 14:05
detected 14:22
restore to 14:04:30

The recovery point is intentionally before the bad deploy, not merely before the time at which the problem was noticed. In a real drill, also record how the team identifies the target timestamp and prevents the bad deployment from writing into the restored environment.

Then reconcile external side effects after restore. Database recovery can rewind database state, but it cannot automatically rewind systems that have already received messages or completed actions.

Database recovery can rewind:

order status

but cannot undo email/payment already sent externally.

Cross-system recovery requires business procedures. Those procedures may include suppressing duplicate notifications, reconciling payment provider state, replaying or discarding outbox messages, and communicating the resulting customer impact.

MongoDB monitoring

Important dimensions:

connections
operations/sec
query latency
CPU
disk latency
disk space
WiredTiger cache
page faults/IO
replication lag
oplog window
locks/tickets
network
index usage
slow queries
shard balance

These metrics answer different diagnostic questions. Latency tells you that users are waiting; CPU, disk, cache, connections, and tickets help explain why. Replication lag and oplog window show how much recovery and failover headroom remains. Index usage and slow-query data help connect the symptom to a workload.

Atlas provides dashboards/alerts.

Self-managed uses monitoring stack. Whichever platform is used, alerts should be tied to service objectives and trends rather than only to a single absolute threshold. A brief spike and a steadily worsening value do not have the same operational meaning.

WiredTiger cache

MongoDB storage engine uses cache. Frequently accessed data and index pages can be kept in memory so operations do not need to read storage for every request.

Working set larger than memory causes more disk reads. That can increase latency, especially when the storage system is already under pressure. It is a capacity and workload signal, not automatically a database defect.

Do not interpret “Mongo uses lots of memory” as leak; database intentionally caches. The right question is whether cache pressure, eviction, latency, and system memory behavior are healthy for the workload.

Monitor eviction/cache pressure. Correlate those values with disk latency, query latency, and working-set changes before deciding whether to resize the host, change indexes, or change the workload.

Disk latency

Database performance depends strongly on storage. A query can have a reasonable execution plan and still be slow when reads or journal/checkpoint work wait on saturated storage.

High IOPS/latency causes:

• query slowness;
• replication lag;
• checkpoint issues.

CPU tuning will not fix slow disk. Check storage metrics and the operations competing for the device before adding application workers or increasing database CPU.

Connections

Drivers use pools. Each application instance generally maintains a pool of reusable connections, so the server sees the combined behavior of every process, pod, job, and administrative client.

Monitor current/available. A rising current count, a shrinking available count, or connection wait errors can indicate capacity pressure even when query CPU is low.

Too many app instances × huge pool can exhaust server.

A pool size of 100 per pod × 200 pods = 20,000 potential connections.

Capacity plan globally. Include all application replicas, background workers, deploy overlap, failover behavior, and operational tools. Configure pool limits with the database's connection capacity and the application's concurrency needs in mind, rather than copying a large default to every instance.

Slow query profiler

MongoDB profiler can collect detailed operation info. That detail helps identify namespaces, plans, durations, and patterns that aggregate metrics may hide.

Use carefully—profiling itself has overhead and may capture sensitive query values. Limit the scope and duration when possible, protect the resulting data, and account for the operational cost while diagnosing a live system.

In production prefer slow-operation thresholds/diagnostic tools according to Mongo guidance. The goal is usually to capture operations that need investigation without collecting every operation indefinitely.

Do not enable full profiling indefinitely without plan. Define what question the profiling is meant to answer, who can inspect the output, and when the setting will be returned to its normal state.

Current operations

Administrative commands can inspect current long-running operations. This is useful when an incident requires a view of what the server is doing right now, rather than only what appeared in historical logs.

Useful for incident:

what is stuck?
which namespace?
how long?

Use the answers to form the next investigation step: inspect the relevant query, lock or resource pressure, recent deployment, or client behavior. Killing operations is an emergency tool, not query optimization strategy. Terminating an operation may release pressure, but it does not fix an inefficient plan or prevent the workload from returning.

explain

First tool for query performance. Use it to examine how MongoDB plans and executes a query, then compare that evidence with the endpoint's expected result size and latency.

Compare:

nReturned
keys examined
docs examined
execution time
sort

The relationship between returned results and examined keys/documents is often more informative than execution time alone. A query returning ten records after scanning a very large collection is a different problem from a query returning a large result set that genuinely requires substantial work.

Don't start by increasing server RAM if query scans 50M docs for 10 results. First investigate the predicate, index design, sort, data model, and actual execution plan. More memory may reduce some I/O without addressing the unnecessary scan.

$indexStats

Can inspect index usage. This provides evidence about which indexes have been used during the observed period and can help identify indexes that deserve closer review.

Use to identify candidates for cleanup. Removing an index can save storage and write or maintenance work, but usage statistics must be interpreted in the context of the observation window and workload coverage.

But rare critical index may show low use. A monthly report, failover path, or emergency query may be important even if it is not frequently executed.

Combine workload knowledge. Review application query patterns, scheduled jobs, known incident queries, and rollout plans before dropping an index. Validate the change safely and keep a rollback path.

Database logs

Mongo logs:

• connections;
• slow operations;
• elections;
• checkpoints;
• warnings/errors.

Centralize and protect. Centralized logs make correlation across members and application instances possible, while access controls and retention policies prevent logs from becoming an unprotected copy of sensitive operational data.

Sensitive details may exist. Query shapes, namespaces, identifiers, or error context can reveal information that should not be broadly accessible. Apply the same care to log collection and export that you apply to database diagnostics.

Query comment

Drivers can attach comments/metadata to operations in supported APIs.

Useful to identify application operation in profiler/logs. A stable operation label can connect a slow database operation to an endpoint, job, or code path without guessing from the query shape alone.

Do not include sensitive user text. Comments should be controlled metadata, not a place to copy request bodies, tokens, email addresses, or arbitrary customer input.

Example logical:

comment: "GET /tasks list"

maxTimeMS

Bound expensive queries. A time limit gives the database and application a defined upper bound for work that might otherwise consume resources for an unbounded period.

Timeout protects availability. It can prevent one pathological operation from occupying resources indefinitely, but it should be paired with logging and investigation so the failure is visible.

But if normal query always hits timeout, fix index/model. A timeout is not evidence that the query is healthy; it is a boundary around an operation that may still be poorly designed for the data and workload.

Maintenance

Index creation/drop

Plan resource usage. Creating or dropping an index can affect CPU, memory, disk, I/O, query plans, and replication behavior. Treat the change as an operational deployment, not as a harmless metadata edit.

Compact/reclaim

Storage maintenance commands have significant impact and version-specific behavior.

Do not run old blog commands blindly. Check the MongoDB version, deployment type, current official guidance, expected lock or resource behavior, and whether the operation is actually needed before scheduling it.

Validation

validate command checks collection/storage consistency.

Use during diagnostics/maintenance according to guidance. Validation can be useful after an incident or when investigating suspected storage problems, but the command's cost and supported use should be considered for the specific environment.

Roadmap includes validate() concept. Treat that as a pointer to the operational topic, not as permission to run a resource-intensive check on a busy cluster without planning.

validate

Shell:

db.runCommand({
  validate: "tasks"
})

Can be resource intensive. Schedule and execute it with an understanding of its effect on the target collection and cluster.

Do not schedule on hot production blindly. Prefer an appropriate maintenance window or a safer diagnostic target when the situation allows, and record the result and follow-up actions.

Feature Compatibility Version (FCV)

During upgrades, Mongo uses FCV to control feature behavior. The binary version and the feature compatibility setting are related but are not interchangeable; the supported upgrade procedure determines when the feature set can move forward.

MongoDB 8.3 upgrade from 8.0/8.2 requires prescribed paths/FCV state.

Follow official upgrade guide exactly. Verify the current version, supported intermediate versions, driver compatibility, replica-set health, backup status, and FCV state before changing the deployment.

Do not skip versions because “files are compatible.” Upgrade support includes operational sequencing and feature behavior, not just whether a process can open existing files.

Upgrade plan

1. verify supported driver/Mongoose;
2. backup;
3. test staging;
4. review compatibility changes;
5. upgrade adjacent supported path;
6. verify members;
7. set FCV when instructed;
8. monitor;
9. have downgrade plan within supported window.

The sequence is deliberately operational. A backup is useful only if it is accessible and restorable, staging should exercise the application's real driver behavior, and member health should be checked after each meaningful topology change. A downgrade plan must also respect the supported window; do not assume that reverting binaries is always safe after features have been enabled.

Patch versions

Security/reliability fixes arrive in patches.

Production should not remain on .0 indefinitely. Establish a patch policy that balances testing time with the exposure created by delaying fixes, and make the policy visible to the team responsible for the cluster.

Track release notes/advisories. Include driver, server, operating system, and managed-service notices where they affect the deployment.

Driver compatibility

Before DB upgrade, verify Node Mongo driver compatibility. Also check the Mongoose version and the application's use of commands, retry behavior, transactions, and connection options.

Do not upgrade server to feature not understood by old driver. Server availability alone does not prove that the application can negotiate and use the upgraded behavior safely.

Change management

For major index/schema changes:

• canary;
• hidden index;
• dual reads;
• background migration;
• metrics;
• rollback.

The exact combination depends on the change, but the pattern is to reduce blast radius, observe behavior, and retain a way back. A hidden index can help evaluate an index before making it available to normal planning; dual reads or background migration can expose data differences before a hard cutover.

Database changes are deployments. They require review, observability, an owner, and a rollback or mitigation procedure just like application code changes.

Data integrity checks

Examples:

required counts
orphan references
tenant nulls
duplicate business keys
invalid type distributions
negative amounts
schemaVersion distribution

Build scripts/reporting. These checks should produce evidence that can be compared over time and attached to a restore drill, migration, or incident record. They should reflect the application's actual invariants, not only generic database structure.

Do not rely only on Mongo structural validation for business integrity. A document can satisfy collection-level rules and still reference a missing record, contain an impossible amount, or violate a workflow invariant.

Operational runbook

For “Mongo slow”:

1. check alerts/topology
2. confirm primary/election
3. query latency
4. CPU/disk/cache
5. connection saturation
6. slow ops/explain
7. recent deploy/index
8. replication lag
9. shard imbalance
10. mitigate

The ordering moves from broad system state to likely causes and then to action. Confirm topology before interpreting query symptoms, compare latency with resource metrics, and check recent changes before making a risky configuration change. “Mitigate” might mean rolling back a deployment, reducing traffic, disabling a problematic workload, or adding capacity, depending on the evidence.

For “disk 90%”:

• identify growth;
• backup safety;
• add capacity;
• retention;
• indexes;
• logs;
• avoid emergency deletes without plan.

First identify what is growing and whether backup or restore operations are also consuming space. Add capacity when possible, then review retention, indexes, and logs. Avoid emergency deletes without plan: deleting data or indexes under pressure can damage recovery options or create a second incident.

Disaster recovery runbook

Include:

who declares incident
backup location
credentials/key access
restore command/process
target cluster
DNS/connection switch
integrity validation
external-system reconciliation
communications
postmortem

Each entry should identify an owner or an unambiguous procedure. During an incident, operators should not have to discover where backups live, how keys are obtained, which cluster is safe to restore into, or how external side effects are reconciled.

Common mistakes

• no restore tests;
• backup on same server only;
• replication called backup;
• full profiler always on;
• giant connection pools;
• ignore disk latency;
• kill slow query instead of index;
• upgrade without FCV plan;
• use old maintenance commands;
• no patch policy;
• backup secrets exposed;
• RPO/RTO undefined.

These mistakes tend to reinforce one another. For example, an undefined RPO/RTO makes it hard to choose retention or test success, while untested backups make a high-level recovery promise impossible to verify. Treat each item as a prompt to add a concrete design decision, metric, owner, or drill.

Exercises

1. Define RPO/RTO for three systems.
2. Run mongodump/mongorestore on test DB.
3. Time restore.
4. Build PITR incident scenario.
5. Monitor connections/cache/lag.
6. Use explain on slow query.
7. Inspect index stats.
8. Write upgrade checklist for 8.3.
9. Design connection pool capacity.
10. Write production Mongo runbook.

For each hands-on exercise, record what you expected to observe and what would indicate a problem. In particular, compare the measured restore time with the stated RTO, identify which backup point satisfies the RPO, and connect slow-query observations to an actionable index or model decision. For the upgrade and runbook exercises, include ownership, verification, and rollback or mitigation steps rather than only a list of commands.

Mastery checklist

Explain:

• logical/snapshot/PITR;
• RPO/RTO;
• restore drills;
• monitoring;
• cache/disk;
• profiler/current ops;
• connections;
• validate;
• FCV/upgrades;
• patching;
• runbooks.

Being able to explain these terms should lead to operational decisions: which backup mechanism fits the dataset, what point can be recovered, which metrics narrow a performance investigation, how to protect connection capacity, when a maintenance operation is safe, and how a team executes and verifies recovery.

Official references

• https://www.mongodb.com/docs/manual/core/backups/
• https://www.mongodb.com/docs/database-tools/mongodump/
• https://www.mongodb.com/docs/database-tools/mongorestore/
• https://www.mongodb.com/docs/manual/administration/monitoring/
• https://www.mongodb.com/docs/manual/tutorial/manage-the-database-profiler/
• https://www.mongodb.com/docs/manual/release-notes/