Monitoring a Self-Managed MongoDB Deployment

Monitoring Strategies

MongoDB provides various methods for collecting data about the state of a running MongoDB instance:

• MongoDB distributes a set of utilities that provides real-time reporting of database activities.

• MongoDB provides various database commands that return statistics regarding the current database state with greater fidelity.

• MongoDB Atlas is a cloud-hosted database-as-a-service for running, monitoring, and maintaining MongoDB deployments.

• MongoDB Cloud Manager is a hosted service that monitors running MongoDB deployments to collect data and provide visualization and alerts based on that data.

• MongoDB Ops Manager is an on-premise solution available in MongoDB Enterprise Advanced that monitors running MongoDB deployments to collect data and provide visualization and alerts based on that data.

Each strategy can help answer different questions and is useful in different contexts. These methods are complementary.

MongoDB Reporting Tools

This section provides an overview of the reporting methods distributed with MongoDB. It also offers examples of the kinds of questions that each method is best suited to help you address.

Process Logging

During normal operation, mongod and mongos instances report a live account of all server activity and operations to either standard output or a log file. The following runtime settings control these options.

• quiet. Limits the amount of information written to the log or output.

• verbosity. Increases the amount of information written to the log or output. You can also modify the logging verbosity during runtime with the logLevel parameter or the db.setLogLevel() method in the shell.

• path. Enables logging to a file, rather than the standard output. You must specify the full path to the log file when adjusting this setting.

• logAppend. Adds information to a log file instead of overwriting the file.

mongod -v --logpath /var/log/mongodb/server1.log --logappend

The following database commands also affect logging:

• getLog. Displays recent messages from the mongod process log.

• logRotate. Rotates the log files for mongod processes only. See Rotate Log Files.

db.clients.insertOne( { "name" : "Joe", "PII" : "Sensitive Information" } )

{
   "t": { "$date": "2024-07-19T15:36:55.024-07:00" },
   "s": "I",
   "c": "COMMAND",
   ...
   "attr": {
      "type": "command",
      ...
      "appName": "mongosh 2.2.10",
      "command": {
         "insert": "clients",
         "documents": [
            {
               "name": "Joe",
               "PII": "Sensitive Information",
               "_id": { "$oid": "669aea8792c7fd822d3e1d8c" }
            }
         ],
         "ordered": true,
         ...
      }
      ...
   }
}

{
   "t": { "$date": "2024-07-19T15:36:55.024-07:00" },
   "s": "I",
   "c": "COMMAND",
   ...
   "attr": {
      "type": "command",
      ...
      "appName": "mongosh 2.2.10",
      "command": {
         "insert": "###",
         "documents": [
            {
               "name": "###",
               "PII": "###",
               "_id": "###"
            }
         ],
         "ordered": "###",
         ...
      }
      ...
   }
}

Diagnosing Performance Issues

As you develop and operate applications with MongoDB, you may want to analyze the performance of the database as the application. MongoDB Performance discusses some of the operational factors that can influence performance.

Replication and Monitoring

Beyond the basic monitoring requirements for any MongoDB instance, for replica sets, administrators must monitor replication lag. "Replication lag" refers to the amount of time that it takes to copy (i.e. replicate) a write operation on the primary to a secondary. Some small delay period may be acceptable, but significant problems emerge as replication lag grows, including:

• Growing cache pressure on the primary.

• Operations that occurred during the period of lag are not replicated to one or more secondaries. If you're using replication to ensure data persistence, exceptionally long delays may impact the integrity of your data set.

• If the replication lag exceeds the length of the operation log (oplog) then MongoDB will have to perform an initial sync on the secondary, copying all data from the primary and rebuilding all indexes. [1] This is uncommon under normal circumstances, but if you configure the oplog to be smaller than the default, the issue can arise.

  Note

  The size of the oplog is only configurable during the first run using the --oplogSize argument to the mongod command, or preferably, the oplogSizeMB setting in the MongoDB configuration file. If you do not specify this on the command line before running with the --replSet option, mongod will create a default sized oplog.

  By default, the oplog is 5 percent of total available disk space on 64-bit systems. For more information about changing the oplog size, see the Change the Oplog Size of Self-Managed Replica Set Members.

rs.status()

Sharding and Monitoring

In most cases, the components of sharded clusters benefit from the same monitoring and analysis as all other MongoDB instances. In addition, clusters require further monitoring to ensure that data is effectively distributed among nodes and that sharding operations are functioning appropriately.

use config
db.locks.find()

db.locks.find( { _id : "balancer" } )

Storage Node Watchdog

The Storage Node Watchdog monitors the following MongoDB directories to detect filesystem unresponsiveness:

• The --dbpath directory

• The journal directory inside the --dbpath directory if journaling is enabled

• The directory of --logpath file

• The directory of --auditPath file

By default, the Storage Node Watchdog is disabled. You can only enable the Storage Node Watchdog on a mongod at startup time by setting the watchdogPeriodSeconds parameter to an integer greater than or equal to 60. However, once enabled, you can pause the Storage Node Watchdog and restart during runtime. See watchdogPeriodSeconds parameter for details.

If any of the filesystems containing the monitored directories become unresponsive, the Storage Node Watchdog terminates the mongod and exits with a status code of 61. If the mongod is the primary of a replica set, the termination initiates a failover, allowing another member to become primary.

Once a mongod has terminated, it may not be possible to cleanly restart it on the same machine.

The maximum time the Storage Node Watchdog can take to detect an unresponsive filesystem and terminate is nearly twice the value of watchdogPeriodSeconds.