replSetReconfig

term Replica Configuration Field

MongoDB 4.4 adds the term field to the replica set configuration document. The term field is set by the primary replica set member. The primary ignores the term field if set explicitly in the replSetReconfig operation.

Reconfiguration Can Add or Remove No More than One Voting Member at a Time

Starting in MongoDB 4.4, replSetReconfig by default allows adding or removing no more than 1 voting member at a time. For example, a new configuration can make at most one of the following changes to the cluster membership:

• Adding a new voting replica set member.

• Removing an existing voting replica set member.

• Modifying the votes for an existing replica set member.

To add or remove multiple voting members, issue a series of replSetReconfig operations to add or remove one member at a time.

Issuing a force reconfiguration immediately installs the new configuration even if it adds or removes multiple voting members. Force reconfiguration can cause unexpected behavior, such as the rollback of "majority" committed write operations.

Reconfiguration Waits Until a Majority of Members Install the Replica Configuration

Starting in MongoDB 4.4, replSetReconfig waits until a majority of voting replica set members install the new replica configuration before returning success. A voting member is any replica set member where members[n].votes is 1, including arbiters.

Replica set members propagate their replica configuration via heartbeats. Whenever a member learns of a configuration with a higher version and term, it installs the new configuration. The reconfiguration process has two distinct 'waiting' phases:

1) Wait for the current configuration to be committed before installing the new configuration.

The "current" configuration refers to the replica configuration in use by the primary at the time replSetReconfig is issued.

A configuration is committed when:

• A majority of voting replica set members have installed the current configuration, and

• All writes which were "majority" committed in the previous configuration have also replicated to a majority in the current configuration.

Typically, the current configuration has already been installed on a majority of voting replica set members. However, the majority committed writes in the previous configuration may not all be committed in the current configuration. Delayed members or members that are lagging behind the primary can increase the time spent in this phase.

If the operation was issued with a maxTimeMS limit and the operation exceeds the limit while waiting, the operation returns an error and discard the new configuration. The limit is cumulative and does not reset after proceeding to the next phase.

2) Wait for a majority of voting members in the new configuration to install the new configuration.

The "new" configuration refers to the replica configuration specified to replSetReconfig.

The primary installs and begins using the new replica configuration before propagating the configuration to the remaining replica set members. The operation only waits for a majority of voting members to install the new configuration, and does not require waiting for the new configuration to be committed.

If the operation was issued with a maxTimeMS limit and the operation exceeds the limit while waiting, the operation returns an error but continues using and propagating the new configuration.

Issuing a force reconfiguration immediately installs the new configuration regardless of the commitment status of the previous configuration. Force reconfiguration can cause unexpected behavior, such as the rollback of "majority" committed write operations.

To check the commitment status of the current replica configuration, issue replSetGetConfig with the commitmentStatus parameter on the replica set primary.

Access Control

To run the command on deployments that enforce access control, the user must have replSetConfigure privilege action on the cluster resource. The clusterManager built-in role, available in the admin database, provides the required privileges for this command.

Locking Behavior

replSetReconfig obtains a special mutually exclusive lock to prevent more than one replSetReconfig operation from occurring at the same time.

Mixed Version Replica Set

Availability

A majority of the set's members must be operational for the changes to propagate properly.

replSetReconfig can trigger the current primary to step down in some situations. Primary step-down triggers an election to select a new primary:

• Starting in MongoDB 4.4, when the new primary steps up, it increments the term field to distinguish configuration changes made on the new primary from changes made on the previous primary.

• Starting in MongoDB 4.2, when the primary steps down, it no longer closes all client connections; however, writes that were in progress are killed. For details, see Behavior.

• In MongoDB 4.0 and earlier, when the primary steps down, it closes all client connections.

The median time before a cluster elects a new primary should not typically exceed 12 seconds, assuming default replica configuration settings. This includes time required to mark the primary as unavailable and call and complete an election. You can tune this time period by modifying the settings.electionTimeoutMillis replication configuration option. Factors such as network latency may extend the time required for replica set elections to complete, which in turn affects the amount of time your cluster may operate without a primary. These factors are dependent on your particular cluster architecture.

During the election process, the cluster cannot accept write operations until it elects the new primary.

Your application connection logic should include tolerance for automatic failovers and the subsequent elections. MongoDB drivers can detect the loss of the primary and automatically retry certain write operations a single time, providing additional built-in handling of automatic failovers and elections:

Compatible drivers enable retryable writes by default

To further reduce potential impact to a production cluster, reconfigure only during scheduled maintenance periods.

{ force: true }

Drop Outgoing Connections After Removing a Member

Using replSetReconfig to remove a replica set member does not automatically drop open outgoing connections from other replica set members to the removed member.

By default, replica set members wait for 5 minutes before dropping connections to the removed member. In sharded replica sets, you can modify this timeout using the ShardingTaskExecutorPoolHostTimeoutMS server parameter.

db.adminCommand(
  {
    "dropConnections" : 1,
    "hostAndPort" : [
      "<hostname>:<port>"
    ]
  }
)

Replace <hostname> and <port> with those of the removed member.

Member Priority and Votes

• Members with priority greater than 0 cannot have 0 votes.

• Non-voting (i.e. votes is 0) members must have priority of 0.