Sharded Cluster Balancer

Adding and Removing Shards from the Cluster

Adding a shard to a cluster creates an imbalance, since the new shard has no data. While MongoDB begins migrating data to the new shard immediately, it can take some time before the cluster balances. See the Add Shards to a Cluster tutorial for instructions on adding a shard to a cluster.

Removing a shard from a cluster creates a similar imbalance, since data residing on that shard must be redistributed throughout the cluster. While MongoDB begins draining a removed shard immediately, it can take some time before the cluster balances. Do not shutdown the servers associated to the removed shard during this process.

When you remove a shard in a cluster with an uneven chunk distribution, the balancer first removes the chunks from the draining shard and then balances the remaining uneven chunk distribution.

See the Remove Shards from a Cluster tutorial for instructions on safely removing a shard from a cluster.

Migration Thresholds

To minimize the impact of balancing on the cluster, the balancer only begins balancing after the distribution of data for a sharded collection has reached certain thresholds.

A collection is considered balanced if the difference in data between shards (for that collection) is less than three times the configured range size for the collection. For the default range size of 128MB, two shards must have a data size difference for a given collection of at least 384MB for a migration to occur.

Asynchronous Range Migration Cleanup

To migrate data from a shard, the balancer migrates the data one range at a time. However, the balancer does not wait for the current migration's delete phase to complete before starting the next range migration. See Range Migration for the range migration process and the delete phase.

This queuing behavior allows shards to unload data more quickly in cases of heavily imbalanced cluster, such as when performing initial data loads without pre-splitting and when adding new shards.

This behavior also affects the moveRange command, and migration scripts that use the moveRange command may proceed more quickly.

In some cases, the delete phases may persist longer. Range migrations are enhanced to be more resilient in the event of a failover during the delete phase. Orphaned documents are cleaned up even if a replica set's primary crashes or restarts during this phase.

The _waitForDelete balancer setting can alter the behavior so that the delete phase of the current migration blocks the start of the next chunk migration. The _waitForDelete is generally for internal testing purposes. For more information, see Wait for Delete.

Range Migration and Replication

During range migration, the _secondaryThrottle value determines when the migration proceeds with next document in the range.

In the config.settings collection:

• If the _secondaryThrottle setting for the balancer is set to a write concern, each document moved during range migration must receive the requested acknowledgment before proceeding with the next document.

• If the _secondaryThrottle setting is unset, the migration process does not wait for replication to a secondary and instead continues with the next document.

To update the _secondaryThrottle parameter for the balancer, see Secondary Throttle for an example.

Independent of any _secondaryThrottle setting, certain phases of the range migration have the following replication policy:

• MongoDB briefly pauses all application reads and writes to the collection being migrated to on the source shard before updating the config servers with the range location. MongoDB resumes application reads and writes after the update. The range move requires all writes to be acknowledged by majority of the members of the replica set both before and after committing the range move to config servers.

• When an outgoing migration finishes and cleanup occurs, all writes must be replicated to a majority of servers before further cleanup (from other outgoing migrations) or new incoming migrations can proceed.

To update the _secondaryThrottle setting in the config.settings collection, see Secondary Throttle for an example.

Maximum Number of Documents Per Range to Migrate

By default, MongoDB cannot move a range if the number of documents in the range is greater than 2 times the result of dividing the configured range size by the average document size. If MongoDB can move a sub-range of a chunk and reduce the size to less than that, the balancer does so by migrating a range. db.collection.stats() includes the avgObjSize field, which represents the average document size in the collection.

For chunks that are too large to migrate:

• The balancer setting attemptToBalanceJumboChunks allows the balancer to migrate chunks too large to move as long as the chunks are not labeled jumbo. See Balance Ranges that Exceed Size Limit for details.

  When issuing moveRange and moveChunk commands, it's possible to specify the forceJumbo option to allow for the migration of ranges that are too large to move. The ranges may or may not be labeled jumbo.

Range Deletion Performance Tuning

You can tune the performance impact of range deletions with rangeDeleterBatchSize, rangeDeleterBatchDelayMS, and rangeDeleterHighPriority parameters. For example:

• To limit the number of documents deleted per batch, you can set rangeDeleterBatchSize to a small value such as 32.

• To add an additional delay between batch deletions, you can set rangeDeleterBatchDelayMS above the current default of 20 milliseconds.

• To prioritize range deletions, you can set rangeDeleterHighPriority to true. Range deletions are potentially long-running background tasks that might negatively impact the throughput of user operations when the system is under heavy load.

Change Streams and Orphan Documents

Starting in MongoDB 5.3, during range migration, change stream events are not generated for updates to orphaned documents.