Delayed Replica Set Members
On this page
Delayed members contain copies of a replica set's data set. However, a delayed member's data set reflects an earlier, or delayed, state of the set. For example, if the current time is 09:52 and a member has a delay of an hour, the delayed member has no operation more recent than 08:52.
Because delayed members are a "rolling backup" or a running "historical" snapshot of the data set, they may help you recover from various kinds of human error. For example, a delayed member can make it possible to recover from unsuccessful application upgrades and operator errors including dropped databases and collections.
Considerations
Requirements
Delayed members:
Must be priority 0 members. Set the priority to 0 to prevent a delayed member from becoming primary.
Must be hidden members. Always prevent applications from seeing and querying delayed members.
Do vote in elections for primary, if
members[n].votesis set to 1. Ensuring that delayed members are non-voting by settingmembers[n].votesto 0 can help improve performance.
Important
If your replica set contains delayed members ensure that the delayed members are hidden and non-voting.
Hiding delayed replica set members prevents applications from seeing
and querying delayed data without a direct connection to that member.
Making delayed replica set members non-voting means they will not
count towards acknowledging write operations with write concern
"majority".
If you do not hide delayed members and one or more nodes become unavailable, the replica set has to wait for the delayed member and the commit point lags. A lagged commit point can lead to performance issues.
For example, consider a Primary-Secondary-Delayed replica set configuration where the delayed secondary is voting with a 10 minute delay.
With one non-delayed secondary unavailable, the degraded configuration
of Primary-Delayed must wait at least 10 minutes to acknowledge a write
operation with "majority".The majority commit point
will take longer to advance, leading to cache pressure similar
performance issues with a
Primary with a Secondary and an Arbiter
(PSA) replica set.
For more information on the majority commit point, see Causal Consistency and Read and Write Concerns. For additional details on resolving performance issues see the replica set maintenance tutorial.
Behavior
Delayed members copy and apply operations from the source oplog on a delay. When choosing the amount of delay, consider that the amount of delay:
must be equal to or greater than your expected maintenance window durations.
must be smaller than the capacity of the oplog. For more information on oplog size, see Oplog Size.
Write Concern
Delayed replica set members can acknowledge write operations issued
with w: <number>. For write operations
isued with w : "majority", however,
delayed members must also be voting members (i.e.
members[n].votes greater than 0) to acknowledge the
"majority" write operation. Non-voting replica set members
(i.e. members[n].votes is 0) cannot contribute to
acknowledging write operations with majority write concern.
Delayed secondaries can return write acknowledgment no earlier than the
configured secondaryDelaySecs.
Sharding
In sharded clusters, delayed members have limited utility when the balancer is enabled. Because delayed members replicate chunk migrations with a delay, the state of delayed members in a sharded cluster are not useful for recovering to a previous state of the sharded cluster if any migrations occur during the delay window.
Example
In the following 5-member replica set, the primary and all secondaries have copies of the data set. One member applies operations with a delay of 3600 seconds (one hour). This delayed member is also hidden and is a priority 0 member.
Configuration
A delayed member has its
members[n].priority equal to 0,
members[n].hidden equal to true, and
its members[n].secondaryDelaySecs equal to the
number of seconds of delay:
{ "_id" : <num>, "host" : <hostname:port>, "priority" : 0, "secondaryDelaySecs" : <seconds>, "hidden" : true }
To configure a delayed member, see Configure a Delayed Replica Set Member.