• Monitor Your Deployments >
• Configure and Resolve Alerts >
• Configure Alert Settings >
• Manage System Alerts

Manage System Alerts

System alerts are internal health checks that monitor the health of Ops Manager itself, including the health of backing databases, Backup Daemons, and backed-up deployments. Ops Manager runs health checks every five minutes.

To view the list of system alerts:

1. Click the Admin link at the top of the Ops Manager UI.
2. Click the Alerts tab.
3. Click the Open Alerts link under System Alerts.

Disabled system alerts are grayed out.

If you have the Global Owner or Global Monitoring Admin roles, you can modify notification settings or disable a system alert.

System Alert Components

Each system alert consists of three components:

Component	Examples
A condition that triggers the alert	A blockstore is using too much disk space. An Ops Manager backing database had startup warnings.
A list of recipients of the alert	Ops Manager user or role: Ops Manager Administrators Email address: admin@example.com Slack channel: #alerts
A method by which the alert is sent	Email Slack SMS

When the alert is enabled and its trigger condition is met, Ops Manager sends an alert to the specified recipients using the specified medium for that alert. For a list of the notification options, see the Select the alert recipients and delivery methods step in the Modify Notification Settings for a System Alert procedure on this page.

By default, Ops Manager enables all alerts and sends the alerts to the email address specified in the Admin Email Address field in the Ops Manager configuration options.

Available System Alerts

Ops Manager provides the following system alerts:

Alert Processing

Alert Type	Alert Message	Description
ORDINARY_ALERT_PROCESSING_DISABLED	Ordinary alert processing is disabled
ORDINARY_ALERT_PROCESSING_ENABLED	Ordinary alert processing is enabled

Backup

Alert Type	Alert Message	Description
OPLOG_TTL_RESIZE	Backup oplog TTL was resized	Sent when the Backup Daemon has fallen so far behind in applying oplog entries that Ops Manager has extended the period of time it stores the oplog entries. By default, Ops Manager stores oplog entries in the Oplog Store for 24 hours. If the Daemon has not yet applied an oplog entry an hour before its expiration, Ops Manager extends the storage period for another three hours. Ops Manager can continue to extend the storage period up to 14 days. If you receive this alert: Check that your Backup Daemon is running. Check that your Backup Daemon runs on sufficiently performant hardware to apply oplog entries in a timely manner.
THEFT_FAILED	Backup was not moved successfully	Sent when a backup job migration to a new Backup Daemon fails. The backup job continues to run on the original Backup Daemon. For more information on moving jobs, see Jobs.

Backup Daemon

Alert Type	Alert Message	Description
DAEMON_DOWN	Backup Daemon is down	Sent when the Backup Daemon has not pinged Ops Manager for more than 15 minutes.
DAEMON_UP	Backup Daemon is up
LOW_HEAD_FREE_SPACE	Backup Daemon has low free head space	Sent when the disk partition on which the local copy of a backed-up replica set is stored has less than 1 GB of free space remaining. Follow the Modify Notification Settings for a System Alert procedure to change this space limit.
LOW_HEAD_FREE_SPACE_PERCENT	Backup Daemon has low free head space percentage	Sent when the disk partition on which the local copy of a backed-up replica set is stored has less than 10 percent of free space remaining. Follow the Modify Notification Settings for a System Alert procedure to change this percentage.
SUFFICIENT_HEAD_FREE_SPACE	Backup Daemon has sufficient free head space

Blockstore

Alert Type	Alert Message	Description
BALANCER_OFF	Blockstore does not have a running balancer
BALANCER_ON	Blockstore has a running balancer	Sent when a sharded blockstore is running the sharded cluster balancer. You should disable the balancer on a sharded blockstore. To disable the balancer, see Disable the Balancer.
INSIDE_SPACE_USED_THRESHOLD	Blockstore space used within threshold
OUTSIDE_SPACE_USED_THRESHOLD	Blockstore space used exceeds threshold	Sent when the disk space the blockstore uses exceeds the configured threshold setting. The default threshold is 85% of the total disk capacity on which the blockstore is stored. You can change the mms.alerts.OutsideSpaceUsedThreshold.maximumSpaceUsedPercent value in the Ops Manager configuration.

Cron Job

Alert Type	Alert Message	Description
CRON_JOB_COMPLETED	Cron job has completed successfully
CRON_JOB_FAILED	Cron job has failed

Cron Job Status

Alert Type	Alert Message	Description
CRON_JOB_DISABLED	Cron job disabled
CRON_JOB_ENABLED	Cron job enabled

Database Process

Alert Type	Alert Message	Description
BACKING_DATABASE_PROCESS_DOWN	Detected a problem connecting to a database backing the system	Sent when Ops Manager cannot connect to a backing database and run the ping command.
BACKING_DATABASE_PROCESS_NO_STARTUP_WARNINGS	Detected no startup warnings on the databases backing the system
BACKING_DATABASE_PROCESS_STARTUP_WARNINGS	Detected startup warnings on a database backing the system	Sent when the MongoDB process hosting a backing database contains startupWarnings in its log files.
BACKING_DATABASE_PROCESS_UP	Successfully connected to a database backing the system

Log

Alert Type	Alert Message	Description
LOG_DEBUG_OVERRIDE_ACTIVE	Production logger overridden with a DEBUG level

Modify Notification Settings for a System Alert

1

Click the Admin link in the top right corner of Ops Manager.

Ops Manager displays the Admin link only if you have administrative privileges.

2

Select the Alerts tab and then the System Alert Settings page.

3

On the line for the alert, click the ellipsis icon and select Edit.

4

Set the desired alert trigger values (Optional).

You can edit the values that trigger certain system alerts. These values are found in the Alert If section of the following alerts:

Alerts for which you can change the Alert if trigger values

Alert	Field	Default
Backup daemon has low free head space	Backup daemon free head space is below	1 GB
Backup daemon has low free head space percentage	Backup daemon free head space percentage is below	10

5

Select the alert recipients and delivery methods.

In the Send to section, configure notifications. To add notifications or recipients, click Add and select from the options listed below. To test a notification, click the test link that appears after you configure the notification and ensure that the service you are testing receives the message.

Which alert notification methods can be set depend on the scope of the alert:

Project alerts

Apply to one or more individual Organizations and Projects only.

Global alerts

Apply to all Organizations and Projects.

System alerts

Apply to the health of Ops Manager and its backing databases.

The alert notifications methods are as follows:

Notification Method	Project	Global	System	Description
Ops Manager Project	check icon	check icon		Sends the alert by email or text message to users with specific roles in the Project. Select the Project roles that should receive the alerts from the Select Role(s) check boxes or select All Roles for all users in the Project to receive the alert. Select SMS to send these alerts to the mobile number configured for each Ops Manager Project user in their Account page. Select Email to send these alerts to the email address configured for each Ops Manager Project user in their Account page. Email is checked by default.
Ops Manager Organization	check icon	check icon		Sends the alert by email or text message to users with specific roles in the Organization. Select the Organization roles that should receive the alerts from the Select Role(s) check boxes or select All Roles for all users in the Organization to receive the alert. Select SMS to send these alerts to the mobile number configured for each Ops Manager Organization user in their Account page. Select Email to send these alerts to the email address configured for each Ops Manager Organization user in their Account page. Email is checked by default.
Ops Manager User	check icon			Sends the alert to a Ops Manager user, either by email or text message. Select SMS to send these alerts to the mobile number configured for the Ops Manager user’s account. Select Email to send these alerts to the email address configured for the Ops Manager user’s account. Email is checked by default.
Ops Manager Team	check icon			Sends the alert to a Ops Manager user, either by email or text message. Select SMS to send these alerts to the mobile number configured for the Ops Manager user’s account. Select Email to send these alerts to the email address configured for the Ops Manager user’s account. Email is checked by default.
SNMP Host	check icon	check icon	check icon	Specify the hostname that will receive the v2c trap on standard port 162. The MIB file for SNMP is available for download.
Email	check icon	check icon	check icon	Sends the alert to a specified email address.
SMS	check icon	check icon	check icon	Sends the alert to a phone number. Ops Manager removes all letters and punctuation except the +. For international (non-U.S.) phone numbers, format phone numbers using the E.164 standard. Ops Manager uses the U.S.-based Twilio to send text messages. If you want to use a non-U.S. phone number, try using a Google Voice phone number. Example For New Zealand, enter +64 before the phone number. Configure Ops Manager for Twilio integration to use SMS.
HipChat	check icon	check icon	check icon	Sends the alert to a HipChat room message stream. Enter the HipChat room name and API token.
Slack	check icon	check icon	check icon	Sends the alert to a Slack channel in the authorized Slack workplace for the Organization. Enter the channel name and either an API token or a Bot token. To create an API token, see the API page in your Slack account. To learn more about Bot users in Slack, see the Slack documentation.
FlowDock	check icon	check icon	check icon	Sends the alert to a Flowdock account. Enter the following: Org Name The Flowdock organization name in lower-case letters. This is the name that appears after www.flowdock.com/app/ in the URL string. Flow Name The flow name in lower-case letters. The flow name appears after the organization name in the URL string: www.flowdock.com/app/<organization-name>/<flow-name> User API Token Your Flowdock personal API token found on the Tokens page of your Flowdock account.
PagerDuty	check icon	check icon	check icon	Sends the alert to a PagerDuty account. Enter only the PagerDuty integration key. Define escalation rules and alert assignments directly in PagerDuty. This option is available only for alerts that require an acknowledgement. Information alerts, such as User joined the organization, can’t use this notification method. Acknowledge PagerDuty alerts from the PagerDuty dashboard. Note PagerDuty decommissioned their REST API v1 key in October 2018. If you have a v1 key, you can continue to use that key with Ops Manager. All new PagerDuty keys use their REST API v2, but Ops Manager does not support their v2 keys. If you don’t have their REST API v1 key, use the PagerDuty Events API v1 instead.
Webhook	check icon	check icon		Sends an HTTP POST request to an endpoint for programmatic processing. The request body contains a JSON document that uses the same format as the Ops Manager API Alerts resource. To configure this option, configure the Webhook settings on the Project Settings page. To use this method at the Global level: Navigate to the Ops Manager Config Miscellaneous tab of the Administration console. Update the Webhook URL and Webhook Secret settings. Ops Manager adds a request header called X-MMS-Event to distinguish between various alert states. The possible values for this header are: alert.open The alert was just opened. alert.close The alert was resolved. alert.update A previously opened alert is still open. alert.acknowledge The alert was acknowledged. alert.cancel The alert became invalid and was canceled. alert.inform Represents an informational alert, which is a point-in-time event, such as “Primary Elected.” If you specify a key in the Webhook Secret field, MongoDB Ops Manager adds the X-MMS-Signature request header. This header contains the base64-encoded HMAC-SHA-1 signature of the request body. MongoDB Ops Manager creates the signature using the provided secret.
Datadog	check icon	check icon	check icon	Sends the alert to a Datadog account as a Datadog event. When the alert first opens, Ops Manager sends the alert as an “error” event. Subsequent updates are sent as “info” events. When the alert closes, Ops Manager sends a “success” event. If prompted, enter your DataDog API key under API Key and click Validate Datadog API Key. Find your DataDog API Key in your Datadog account.
Administrators		check icon	check icon	Sends the alert to the email address specified in the Admin Email Address field in the Ops Manager configuration options.
Global Alerts Summary Email		check icon		Sends a summary email of all global alerts to the specified email address.

6

Click Save.

Disable a System Alert

1

Click the Admin link in the top right corner of Ops Manager.

Ops Manager displays the Admin link only if you have administrative privileges.

2

Select the Alerts tab and then the System Alert Settings page.

3

Disable the alert.

On the line for the system alert that you want to disable, click the ellipsis icon and select Disable.

←   Manage Global Alerts Resolve Alerts  →

© MongoDB, Inc 2008-present. MongoDB, Mongo, and the leaf logo are registered trademarks of MongoDB, Inc.