Rolling Index Builds on Sharded Clusters

Considerations

Prerequisites

For building unique indexes

1. To create unique indexes using the following procedure, you must stop all writes to the collection during the index build. Otherwise, you may end up with inconsistent data across the replica set members. If you cannot stop all writes to the collection, do not use the following procedure to create unique indexes.

   Warning

   If you cannot stop all writes to the collection, do not use the following procedure to create unique indexes.

2. Before creating the index, validate that no documents in the collection violate the index constraints. If a collection is distributed across shards and a shard contains a chunk with duplicate documents, the create index operation may succeed on the shards without duplicates but not on the shard with duplicates. To avoid leaving inconsistent indexes across shards, you can issue the db.collection.dropIndex() from a mongos to drop the index from the collection.

Procedure

sh.stopBalancer()

sh.getBalancerState()

db.adminCommand( { flushRouterConfig: "test.records" } );
db.records.getShardDistribution();

Shard shardA at shardA/s1-mongo1.example.net:27018,s1-mongo2.example.net:27018,s1-mongo3.example.net:27018
 data : 1KiB docs : 50 chunks : 1
 estimated data per chunk : 1KiB
 estimated docs per chunk : 50
Shard shardC at shardC/s3-mongo1.example.net:27018,s3-mongo2.example.net:27018,s3-mongo3.example.net:27018
 data : 1KiB docs : 50 chunks : 1
 estimated data per chunk : 1KiB
 estimated docs per chunk : 50
Totals
 data : 3KiB docs : 100 chunks : 2
 Shard shardA contains 50% data, 50% docs in cluster, avg obj size on shard : 40B
 Shard shardC contains 50% data, 50% docs in cluster, avg obj size on shard : 40B

net:
   bindIp: localhost,<hostname(s)|ip address(es)>
   port: 27218
#   port: 27018
#replication:
#   replSetName: shardA
#sharding:
#   clusterRole: shardsvr
setParameter:
   skipShardingConfigurationChecks: true
   disableLogicalSessionCacheRefresh: true

mongod --config <path/To/ConfigFile>

mongod --port 27218 --setParameter skipShardingConfigurationChecks=true --setParameter disableLogicalSessionCacheRefresh=true

db.records.createIndex( { username: 1 } )

net:
   bindIp: localhost,<hostname(s)|ip address(es)>
   port: 27018
replication:
   replSetName: shardA
sharding:
   clusterRole: shardsvr

mongod --config <path/To/ConfigFile>

mongod --port 27018 --replSet shardA --shardsvr

sh.startBalancer()

Additional Information

A sharded collection has an inconsistent index if the collection does not have the exact same indexes (including the index options) on each shard that contains chunks for the collection. Although inconsistent indexes should not occur during normal operations, inconsistent indexes can occur, such as:

• When a user is creating an index with a unique key constraint and one shard contains a chunk with duplicate documents. In such cases, the create index operation may succeed on the shards without duplicates but not on the shard with duplicates.

• When a user is creating an index across the shards in a rolling manner but either fails to build the index for an associated shard or incorrectly builds an index with different specification.

Starting in MongoDB 4.4 (and 4.2.6), the config server primary periodically checks for index inconsistencies across the shards for sharded collections. To configure these periodic checks, see enableShardedIndexConsistencyCheck and shardedIndexConsistencyCheckIntervalMS.

The command serverStatus returns the field shardedIndexConsistency to report on index inconsistencies when run on the config server primary.

To check if a sharded collection has inconsistent indexes, see Find Inconsistent Indexes Across Shards.