Build a Resilient Application with MongoDB Atlas

Improved Memory Management

To run your application safely in production, it's important to ensure that your memory utilization allows for headroom. If a node runs out of available memory, it can become susceptible to the Linux Out of Memory Killer that terminates the mongod process.

MongoDB 8.0 uses an upgraded TCMalloc for all deployments automatically, which reduces average memory fragmentation growth over time. This lower fragmentation improves operational stability during peak loads and results in overall improved memory utilization.

Operation Rejection Filters

An unintentional resource-intensive operation can cause problems in production if you don't handle it promptly.

MongoDB 8.0 allows you to minimize the impact of these operations by using operation rejection filters. Operation rejection filters allow you to configure MongoDB to reject queries from running until you re-enable queries with that query shape.

In other words, once you identify a slow query, you don't need to wait for application teams to fix their queries to contain the slow query's impact. Instead, once you notice a poorly performing query in either your Query Profiler, Real Time Performance Panel, or query logs, you can set a rejection filter on that query shape. MongoDB then prevents new instances of that incoming query shape from executing. Once you fix the query, you can re-enable the query shape.

You should use an operation rejection filter if you want to:

• Contain the impact of slow queries quickly while the fix is in progress.

• Prioritize more important workloads during times of overload by rejecting less important queries.

• Give the cluster time to recover if it's close to max resource utilization.

Cluster-level Timeouts for Read Operations

It's important to ensure that your development process carefully considers the efficiency of queries before they reach production. Exceptions may always occur, but having a proactive mitigation against inefficient queries can help prevent cluster performance issues.

With MongoDB 8.0, you can protect your queries from unindexed operations with the server-side defaultMaxTimeMS coming into the cluster. If an operation exceeds this timeout, MongoDB cancels the operation to prevent queries from running too long and holding on to resources. This allows you to:

• Shift the responsibility of setting timeouts from individual application teams to database focused teams.

• Minimize the impact of a collection scan if the query is missing an index.

• Have a last-round mitigation against expensive operations that make it to production.

If you have queries that require a different timeout, such as analytics queries, you can override them by setting the operation-level timeout with the maxTimeMS method.

Isolate the Impact of Busy, Unsharded Collections

Sharding allows you to scale your cluster horizontally. With MongoDB, you can shard some collections, while allowing other collections in the same cluster to remain unsharded. When you create a new database, the shard in the cluster with the least amount of data is picked as that database's primary shard by default. All of the unsharded collections of that database live in that primary shard by default. This can cause increased traffic to the primary shard as your workload grows, especially if the workload growth focuses on the unsharded collections on the primary shard.

To distribute this workload better, MongoDB 8.0 allows you to move an unsharded collection to other shards from the primary shard with the moveCollection command. This allows you to place active, busy collections onto shards with less expected resource usage. With this, you can:

• Optimize performance on larger, complex workloads.

• Achieve better resource utilization.

• Distribute date more evenly across shards.

We recommended to isolate your collection in the following circumstances:

• If your primary shard experiences significant workload due to the presence of multiple high-throughput unsharded collections.

• You anticipate an unsharded collection to experience future growth, which could become a bottleneck for other collections.

• You are running a one-collection-per-cluster deployment design and you want to isolate those customers based on priority or workloads.

• Your shards have more than a proportional amount of data due to the number of unsharded collections located on them.

To learn how to move an unsharded collection with mongosh, see Move a Collection.

Install the Latest Drivers

Install the latest drivers for your language from MongoDB Drivers. Drivers connect and relay queries from your application to your database. Using the latest drivers enables the latest MongoDB features.

Then, in your application, import the dependency:

<dependencies>
    <dependency>
        <groupId>org.mongodb</groupId>
        <artifactId>mongodb-driver-sync</artifactId>
        <version>4.0.1</version>
    </dependency>
</dependencies>

dependencies {
  compile 'org.mongodb:mongodb-driver-sync:4.0.1'
}

// Latest 'mongodb' version installed with npm
const MongoClient = require('mongodb').MongoClient;

# Install the latest 'pymongo' version with pip and
# import MongoClient from the package to establish a connection.
from pymongo import MongoClient

Connection Strings

Use a connection string that specifies all the nodes in your Atlas cluster to connect your application to your database. If your cluster performs a replica set election and a new primary is elected, a connection string that specifies all the nodes in your cluster discovers the new primary without application logic.

You can specify all the nodes in your cluster using either:

• the Standard Connection String Format, or

• the DNS Seedlist Connection Format (recommended with Atlas).

The connection string can also specify options, notably retryWrites and writeConcern.

Atlas can generate an optimized SRV connection string for sharded clusters using the load balancers from your private endpoint service. When you use an optimized connection string, Atlas limits the number of connections per mongos between your application and your sharded cluster. The limited connections per mongos improve performance during spikes in connection counts.

To learn more about optimized connection strings for sharded clusters behind a private endpoint, see Improve Connection Performance for Sharded Clusters Behind a Private Endpoint.

// Copy the connection string provided by Atlas
String uri = <your Atlas connection string>;
// Instantiate the MongoDB client with the URI
MongoClient client = MongoClients.create(uri);

// Copy the connection string provided by Atlas
const uri = <your Atlas connection string>;
// Instantiate the MongoDB client with the URI
const client = new MongoClient(uri, {
    useNewUrlParser: true,
    useUnifiedTopology: true
});

# Copy the connection string provided by Atlas
uri = <your Atlas connection string>
# Pass your connection string URI to the MongoClient constructor
client = MongoClient(uri)

Retryable Writes and Reads

Write and Read Concern

You can tune the consistency and availability of your application using write concerns and read concerns. Stricter concerns imply that database operations wait for stronger data consistency guarantees, whereas loosening consistency requirements provides higher availability.

Error Handling

Invalid commands, network outages, and network errors that are not handled by retryable writes return errors. Refer to your driver's API documentation for error details.

For example, if an application tries to insert a document that contains an _id value that is already used in the database's collection, your driver returns an error that includes:

Unable to insert due to an error: com.mongodb.MongoWriteException:
E11000 duplicate key error collection: <db>.<collection> ...

{
    "name": : "MongoError",
    "message": "E11000 duplicate key error collection on: <db>.<collection> ... ",
    ...
}

pymongo.errors.DuplicateKeyError: E11000 duplicate key error collection: <db>.<collection> ...

Without proper error handling, an error may block your application from processing requests until it is restarted.

Your application should handle errors without crashing or side effects. In the previous example of an application inserting a duplicate _id into a collection, that application could handle errors as follows:

// Declare a logger instance from java.util.logging.Logger
private static final Logger LOGGER = ...
...
try {
    InsertOneResult result = collection.insertOne(new Document()
        .append("_id", 1)
        .append("body", "I'm a goofball trying to insert a duplicate _id"));
    // Everything is OK
    LOGGER.info("Inserted document id: " + result.getInsertedId());
// Refer to the API documentation for specific exceptions to catch
} catch (MongoException me) {
    // Report the error
    LOGGER.severe("Failed due to an error: " + me);
}

...
collection.insertOne({
    _id: 1,
    body: "I'm a goofball trying to insert a duplicate _id"
})
.then(result => {
    response.sendStatus(200) // send "OK" message to the client
},
err => {
    response.sendStatus(400); // send "Bad Request" message to the client
});

...
try:
    collection.insert_one({
        "_id": 1,
        "body": "I'm a goofball trying to insert a duplicate _id"
    })
    return {"message": "User successfully added!"}
except pymongo.errors.DuplicateKeyError as e:
    print ("The insert operation failed:", e)

The insert operation in this example throws a "duplicate key" error the second time it's invoked because the _id field must be unique. The error is caught, the client is notified, and the app continues to run. The insert operation fails, however, and it is up to you to decide whether to show the user a message, retry the operation, or do something else.

You should always log errors. Common strategies for further processing errors include:

• Return the error to the client with an error message. This is a good strategy when you cannot resolve the error and need to inform a user that an action cannot be completed.

• Write to a backup database. This is a good strategy when you cannot resolve the error but don't want to risk losing the request data.

• Retry the operation beyond the single default retry. This is a good strategy when you can solve the cause of an error programmatically, then retry it.

You must select the best strategies for your application context.

Test Failover

In the spirit of chaos testing, Atlas will perform replica set elections automatically for periodic maintenance and certain configuration changes.

To check if your application is resilient to replica set elections, test the failover process by simulating a failover event.