We’re on a migration from Cosmos to MongoDB and We’re looking for a use of M40 tier from Atlas. Actually We have 10TB of data into our database and this is divided into too many partitions (cosmos uses the partition key concept).
The questions is, is there a shard quantity limit into MongoDB M40 instance or any? We plan to have more than 1200 partitions, this looks too much? The Storage that shows on pricing tier is 64 GB, is It for each partition, for each chunk or all for the cluster? Is there a storage limit at chunks or clusters?
MongoDB doesn’t use the partition concept the same way Cosmos does, and the approach is overall very different.
In Cosmos you have physical partitions + logical partitions. Cosmos automatically creates new physical partitions based on the data set size OR RUs provisioned. At 10TB, you probably have a lot of physical partitions and I can only imagine your struggle.
In MongoDB, you have shards (which are like physical partitions in Cosmos) and chunks (which are like logical partitions in Cosmos). You don’t HAVE to run a sharded cluster at all actually. A single replica set might be just fine and much easier to manage. MongoDB engine is performant enough for a lot of use cases.
If you do want to consider a sharded cluster, you would choose the number of shards manually. I suggest to have the following key considerations in mind:
Throughput / scalability requirements
Operational requirements. For 10TB a backup and restore is going to be slow for a single Replica Set. Empirically, the sweet spot is to keep the data size below 4TB per shard.
In your case, you may opt for a single Replica Set or maybe a sharded cluster with 3 shards.
What migration approach are you using? We’re working on a Cosmos to Mongo migration tool (https://www.adiom.io/cosmos-to-mongodb-migration) - would love to see if/how we can help. Send us a ping on the website. I’m the CEO and used to work at MongoDB on migrations.