MongoDB Replica Set Oplog Retention and CDC Reliability
Oplog window size and monitoring are the foundation of reliable MongoDB change data capture.
The oplog is the one physical layer every MongoDB change data capture pipeline reads from, and its retention window decides if a pipeline can recover from downtime or has to resynchronize from scratch. Most failures in production CDC trace back to one variable: the gap between the consumer's last processed position and the oplog's eviction boundary. Understanding how that window is set, monitored, and protected is the foundational discipline for reliable change data capture on MongoDB.
Why the oplog is the substrate every MongoDB CDC pipeline depends on
The oplog is not a CDC feature that someone turned on. It is the internal replication journal MongoDB already maintains for replica set synchronization, and every CDC mechanism, whether Change Streams or direct tailing, reads from it. Every write that modifies data on the primary is recorded there as an idempotent operation, and secondaries continuously tail the journal to stay in sync. A Change Stream, as Streamkap's CDC guide puts it, lets an application act as another consumer of this same oplog stream.
The oplog lives in the local database as a capped collection called oplog.rs. It is primarily associated with replica set members, though a standalone instance can also have one when temporarily restarted in standalone mode during maintenance or configured as a single-node replica set. Failed or no-op writes do not produce oplog entries. The oplog reflects only successful data modifications, so a CDC pipeline consuming it sees a complete record of committed changes, not all attempted writes.
Because the oplog is capped, it has a finite window. When the collection fills, old entries are evicted, and that eviction boundary creates the central reliability constraint this piece addresses. MongoDB replica set confirmation of one PRIMARY and two SECONDARY states occurs within a brief, bounded window, so the replication machinery itself depends on that journal staying intact behind the current position of every consumer.
How the oplog window is determined, and why it can be dangerously short by default
The oplog window is a function of two variables: maximum configured size and the cluster's write rate. The governing formula is straightforward: oplog window in hours equals oplog max size in gigabytes divided by write rate in gigabytes per hour. On busy clusters, the default configuration can produce a window of only a few hours. A modest server with moderate write volume can have a multi-gigabyte oplog that still holds only a few hours of retention under load.
MongoDB sets the oplog to roughly 5% of free disk space at initialization, with a minimum floor and a maximum ceiling. The window is not static. A write spike, whether from a bulk import, an index rebuild, or a schema migration, compresses it temporarily, narrowing the gap between the consumer's last position and the eviction boundary. This isn't a theoretical failure mode. MongoDB replication troubleshooting guidance calls out a short oplog window, meaning less than a few hours, as a primary cause of secondaries being forced into full resync.
What resume tokens are and why the oplog window determines their validity
Resume tokens are the mechanism that makes Change Stream consumers restartable, but their validity has a hard expiry set by the oplog window. If the oplog rolls past a token's position, the token is gone forever. The catch that makes oplog sizing critical is that the oplog is a capped collection. If the consumer is offline longer than the oplog window, the entry the token points to has already been evicted. The stream then raises a ChangeStreamHistoryLost error, and a full collection re-sync is the only recovery path.
That re-sync is not merely inconvenient. You reload every document in every collection in scope, re-establish the consumer's position, and accept a gap or duplication risk in downstream systems during the process. For direct oplog tailing rather than Change Streams, the resume point is a raw ts timestamp persisted to disk. Resumption requires saving the last processed ts and querying for entries with ts greater than lastTs. The same eviction risk applies to that timestamp as to a resume token, because both point into the same capped collection.
Default MongoDB oplog size is 5% of free disk space. That number matters here because it sets the baseline for how long a token survives. The oplog window is not merely a replication tuning knob.
How oplogMinRetentionHours changes the retention contract and when to use it
The oplogMinRetentionHours parameter, available since MongoDB 4.4, separates oplog retention from write volume. It guarantees a time-based floor even when the capped-size constraint would otherwise trigger eviction. Before this parameter existed, the only lever was oplog size. Operators had to estimate write rates and over-provision storage to achieve a target window, with no protection against write spikes compressing the window below the target.
OplogMinRetentionHours prevents the oplog from discarding entries younger than the specified period even if the collection is full in the capped sense. MongoDB respects whichever constraint produces the larger oplog, whether that's the size limit or the time floor. The operational trade-off is real. When the retention floor is active, the oplog can grow beyond its configured maximum size, and disk space consumption can exceed what was configured. Monitoring actual disk usage becomes a requirement, not a recommendation.
Sizing guidance follows directly from that retention contract. Set the retention period to cover the longest expected maintenance window, planned or unplanned, plus a safety margin. If a four-hour failover recovery takes a CDC consumer offline, it needs a window larger than four hours. The maximum MongoDB oplog size is 50 GB, which places a hard ceiling on how much a pure size-based approach can achieve on a busy cluster. A time-based floor remains the only reliable protection for long outages.
How to size, inspect, and monitor the oplog window in a production CDC context
Correctly sizing the oplog and alerting on consumer lag before the window closes are the two operational practices that prevent ChangeStreamHistoryLost from ever becoming the recovery path. The primary diagnostic command is rs.printReplicationInfo(), which reports the current oplog size and the estimated window in hours based on recent write rate. A window of less than a few hours is a warning threshold worth acting on immediately. Additional signals come from rs.status() and rs.printSecondaryReplicationInfo(), which show per-member lag, while db.serverStatus().metrics.repl exposes deeper replication metrics for dashboarding.
To resize the oplog without restarting, use db.adminCommand with replSetResizeOplog set to 1 and size specified in megabytes. A practical calculation approach is to measure one hour of oplog growth, then multiply to reach the target retention period. The sizing guidance that holds up in production is to cover worst-case recovery time multiplied by two. That factor-of-two margin absorbs write-rate spikes that compress the window during the outage being recovered from.
Alerting logic belongs on the consumer side. Track the lag metric, meaning time since the last successfully processed token, against the current oplog window. Alert when lag reaches a configured fraction of the window, well before the eviction boundary, not when ChangeStreamHistoryLost fires. The oplog window is a living metric that changes as write volume grows. Monitoring it is a continuous practice, not a one-time configuration.
How to handle ChangeStreamHistoryLost when it does occur
Every production CDC consumer needs explicit handling for ChangeStreamHistoryLost because no monitoring setup eliminates the possibility of an outage longer than the oplog window. An unhandled error here stalls the pipeline indefinitely. The correct consumer-side response is to detect the ChangeStreamHistoryLost error code, delete the stale resume token, and trigger a full collection re-sync before re-opening the Change Stream.
Delivery semantics during and after recovery must be understood precisely. Change Stream delivery is at-least-once from the consumer's perspective. The last event before a crash may be reprocessed on restart, depending on when the token was saved. Full re-syncs introduce the same risk at scale. Idempotent upserts at the sink are therefore a hard requirement, not an optimization.
The cost of a full re-sync scales with collection size. A team with hundreds of millions of documents faces hours of re-snapshot time, and downstream systems get either stale data or a flood of replayed events. That cost is the real argument for keeping the oplog window wide. Slow consumers are a precursor risk worth watching. A consumer holding its cursor open but processing slowly consumes oplog window without triggering the error immediately. Instrument consumer lag continuously, not just at restart, and alert on the rate of approach to the boundary.
Change Streams vs. direct oplog tailing: which layer to build CDC on
Change Streams are the right default for new CDC pipelines, and direct oplog tailing is a narrow-use tool. This matters because the two have different resume semantics, different sharded-cluster behavior, and different Atlas availability. Change Streams are built on top of the oplog but expose a higher-level API: structured change events rather than raw oplog entries, automatic resume token management, unified cross-shard ordering via mongos, and pre-image and post-image support with appropriate configuration. The oplog tailing guidance recommends Change Streams for new applications explicitly.
Direct oplog tailing grants access to raw operation types, raw update diffs, and compatibility with older deployments where Change Streams are unavailable. But resume is manual, requiring the ts timestamp to be persisted and a reconnect to the new primary after failover. Pre-image support is absent, and sharded clusters require tailing every shard independently and merging by cluster time in application code.
The oplog format can also change between major MongoDB versions. A consumer that parses raw entries may break on upgrade, but Change Streams abstract the format behind a stable event schema. Total failover time in MongoDB with default configuration is 10 to 15 seconds. The oplog window governs the validity of both resume tokens and raw ts positions.
Sharded cluster caveats that change the oplog reliability calculus
In a sharded MongoDB cluster, oplog retention must be managed independently on every shard, and a single cold or slow shard can stall the entire Change Stream even when all others are healthy. Each shard's primary maintains its own oplog. There is no single shared oplog on a sharded cluster, so sizing and monitoring decisions must be applied per shard, not cluster-wide.
Change Streams on a sharded cluster must be opened via mongos, which merges per-shard streams and delivers events in cluster-time order. But the merged stream is only as fast as the slowest shard. A cold shard with little activity introduces latency because mongos must wait for its cursor before advancing. Shard removal can close an open change stream cursor in a way that may not be fully resumable. If you perform shard topology changes, account for that in your pipeline recovery plans. Direct oplog tailing on a sharded cluster requires connecting to every individual shard's primary and merging by cluster time in application code. That is much of what the Change Stream API handles automatically, which is another reason direct tailing remains a narrow-use tool.
Schema evolution in the oplog stream and how CDC consumers must handle it
Because MongoDB's document model is schema-flexible, new fields, restructured documents, and collection-level changes can appear in the change stream at any time. A consumer that is not prepared for them will stall or corrupt downstream data. When a document's structure changes, downstream consumers receive change events with a shape they have not seen before: a new field, a removed field, a changed type. If your consumer layer enforces a strict schema, it will reject or drop those events.
The recommended pattern is to detect new fields at a staging layer, then add corresponding columns in the destination with documented defaults or route the events to a schema-change review pipeline. Raw CDC events should not hit analytical or BI systems directly. Collection-level DDL events, including drops, renames, and index changes, appear in the oplog as command entries with op set to "c" and require explicit handling. AWS DMS, for example, does not support collection renames during the CDC phase, meaning such events can silently break replication.
A schema registry with compatibility checks enforces that producers and consumers agree on evolving shapes. For strict analytical use cases, that registry prevents a careless application-side schema change from propagating corruption downstream. Schema evolution is a CDC pipeline reliability problem, not just an application concern.
Why managed CDC changes the operational burden without changing the underlying mechanics
Everything described in this piece, from oplog sizing and retention configuration to resume token management, lag alerting, schema evolution handling, and re-sync recovery, is real operational work that must be performed by someone. Whether a team builds it themselves or uses a purpose-built service, the work does not disappear. Building a production-grade MongoDB CDC pipeline from scratch means owning the oplog window sizing decisions, the consumer crash-recovery logic, the ChangeStreamHistoryLost handler, the lag monitoring stack, and the schema evolution strategy simultaneously. Getting any one of them wrong produces data loss or pipeline outages.
The distinction that matters for evaluation is whether a managed service actually addresses those failure modes. A service that simply wraps a Change Stream cursor without addressing oplog window monitoring, re-sync automation, and schema evolution is not solving the reliability problem. It is just moving the operational surface to a different team. Evaluate any CDC tooling against the specific failure modes this piece describes. What happens when the consumer is offline for six hours? Does it detect ChangeStreamHistoryLost and handle re-sync automatically? Does it monitor oplog lag and alert before the window closes? Does it propagate schema changes to the destination without manual intervention? A tool that cannot answer those questions leaves the operator holding the same risk, just one layer further from the oplog.
