Sharding

Sharded Cluster

A MongoDB sharded cluster consists of the following components:

• shard: Each shard contains a subset of the sharded data. Each shard must be deployed as a replica set.

• mongos: The mongos acts as a query router, providing an interface between client applications and the sharded cluster. Starting in MongoDB 4.4, mongos can support hedged reads to minimize latencies.

• config servers: Config servers store metadata and configuration settings for the cluster. As of MongoDB 3.4, config servers must be deployed as a replica set (CSRS).

The following graphic describes the interaction of components within a sharded cluster:

MongoDB shards data at the collection level, distributing the collection data across the shards in the cluster.

Shard Keys

MongoDB uses the shard key to distribute the collection's documents across shards. The shard key consists of a field or multiple fields in the documents.

• Starting in version 4.4, documents in sharded collections can be missing the shard key fields. Missing shard key fields are treated as having null values when distributing the documents across shards but not when routing queries. For more information, see Missing Shard Key.

• In version 4.2 and earlier, shard key fields must exist in every document for a sharded collection.

You select the shard key when sharding a collection.

• Starting in MongoDB 4.4, you can refine a collection's shard key by adding a suffix field or fields to the existing key. See refineCollectionShardKey for details.

• In MongoDB 4.2 and earlier, the choice of shard key cannot be changed after sharding.

A document's shard key value determines its distribution across the shards.

• Starting in MongoDB 4.2, you can update a document's shard key value unless your shard key field is the immutable _id field. See Change a Document's Shard Key Value for more information.

• In MongoDB 4.0 and earlier, a document's shard key field value is immutable.

Chunks

MongoDB partitions sharded data into chunks. Each chunk has an inclusive lower and exclusive upper range based on the shard key.

Balancer and Even Chunk Distribution

In an attempt to achieve an even distribution of chunks across all shards in the cluster, a balancer runs in the background to migrate chunks across the shards .

See Data Partitioning with Chunks for more information.

Advantages of Sharding

Considerations Before Sharding

Sharded cluster infrastructure requirements and complexity require careful planning, execution, and maintenance.

Once a collection has been sharded, MongoDB provides no method to unshard a sharded collection.

Careful consideration in choosing the shard key is necessary for ensuring cluster performance and efficiency. See Choosing a Shard Key.

Sharding has certain operational requirements and restrictions. See Operational Restrictions in Sharded Clusters for more information.

If queries do not include the shard key or the prefix of a compound shard key, mongos performs a broadcast operation, querying all shards in the sharded cluster. These scatter/gather queries can be long running operations.

Sharded and Non-Sharded Collections

A database can have a mixture of sharded and unsharded collections. Sharded collections are partitioned and distributed across the shards in the cluster. Unsharded collections are stored on a primary shard. Each database has its own primary shard.

Connecting to a Sharded Cluster

You must connect to a mongos router to interact with any collection in the sharded cluster. This includes sharded and unsharded collections. Clients should never connect to a single shard in order to perform read or write operations.

You can connect to a mongos the same way you connect to a mongod, such as via the mongo shell or a MongoDB driver.

Sharding Strategy

MongoDB supports two sharding strategies for distributing data across sharded clusters.

Hashed Sharding

Hashed Sharding involves computing a hash of the shard key field's value. Each chunk is then assigned a range based on the hashed shard key values.

Tip

MongoDB automatically computes the hashes when resolving queries using hashed indexes. Applications do not need to compute hashes.

While a range of shard keys may be "close", their hashed values are unlikely to be on the same chunk. Data distribution based on hashed values facilitates more even data distribution, especially in data sets where the shard key changes monotonically.

However, hashed distribution means that range-based queries on the shard key are less likely to target a single shard, resulting in more cluster wide broadcast operations

See Hashed Sharding for more information.

Ranged Sharding

Ranged sharding involves dividing data into ranges based on the shard key values. Each chunk is then assigned a range based on the shard key values.

A range of shard keys whose values are "close" are more likely to reside on the same chunk. This allows for targeted operations as a mongos can route the operations to only the shards that contain the required data.

The efficiency of ranged sharding depends on the shard key chosen. Poorly considered shard keys can result in uneven distribution of data, which can negate some benefits of sharding or can cause performance bottlenecks. See shard key selection for range-based sharding.

See Ranged Sharding for more information.

Zones in Sharded Clusters

Zones can help improve the locality of data for sharded clusters that span multiple data centers.

In sharded clusters, you can create zones of sharded data based on the shard key. You can associate each zone with one or more shards in the cluster. A shard can associate with any number of zones. In a balanced cluster, MongoDB migrates chunks covered by a zone only to those shards associated with the zone.

Each zone covers one or more ranges of shard key values. Each range a zone covers is always inclusive of its lower boundary and exclusive of its upper boundary.

click to enlarge

You must use fields contained in the shard key when defining a new range for a zone to cover. If using a compound shard key, the range must include the prefix of the shard key. See shard keys in zones for more information.

The possible use of zones in the future should be taken into consideration when choosing a shard key.

See zones for more information.

Collations in Sharding

Use the shardCollection command with the collation : { locale : "simple" } option to shard a collection which has a default collation. Successful sharding requires that:

• The collection must have an index whose prefix is the shard key

• The index must have the collation { locale: "simple" }

When creating new collections with a collation, ensure these conditions are met prior to sharding the collection.

See shardCollection for more information about sharding and collation.

Change Streams

Starting in MongoDB 3.6, change streams are available for replica sets and sharded clusters. Change streams allow applications to access real-time data changes without the complexity and risk of tailing the oplog. Applications can use change streams to subscribe to all data changes on a collection or collections.

Transactions

Starting in MongoDB 4.2, with the introduction of distributed transactions, multi-document transactions are available on sharded clusters.

Until a transaction commits, the data changes made in the transaction are not visible outside the transaction.

However, when a transaction writes to multiple shards, not all outside read operations need to wait for the result of the committed transaction to be visible across the shards. For example, if a transaction is committed and write 1 is visible on shard A but write 2 is not yet visible on shard B, an outside read at read concern "local" can read the results of write 1 without seeing write 2.