Monitoring

On this page

Overview

Use Cases
Sink Connector
Source Connector
Monitor the Connector
Enable Monitoring
Types of Metrics
JMX Paths
Example - Monitor the Quick Start
Download JConsole.
Start the Quick Start pipeline and add connectors.
Start JConsole.
Connect to the Kafka Connect JMX server.
Explore your connectors' metrics.
Stop and remove the Quick Start environment.
Available Metrics
Sink Connector JMX Metrics
Source Connector JMX Metrics

Overview

Learn how to observe the behavior of your MongoDB Kafka source connector or MongoDB Kafka sink connector through monitoring. Monitoring is the process of getting information about the activities a running program performs for use in an application or an application performance management library.

To learn how monitoring works in the connector and how to use it, see the Use Cases section.

To view an example that shows how to monitor a running connector, see the Example - Monitor the Quick Start section.

To view a list of all metrics produced by MongoDB source and sink connectors, see the Available Metrics section.

Use Cases

This section describes use cases for monitoring MongoDB source and sink connectors, and how you can use the metrics your connector provides to satisfy those use cases.

Tip

Computed Values

To learn what types of metrics the connector provides and when to implement logic to compute a value, see Types of Metrics.

Sink Connector

The following table describes some use cases for monitoring the MongoDB sink connector and the metrics the sink connector provides to satisfy those use cases:

Use Case	Metrics to Use
You want to know if a component of your pipeline is falling behind.	Use the `latest-kafka-time-difference-ms` metric. This metric indicates the interval of time between when a record arrived in a Kafka topic and when your connector received that record. If the value of this metric is increasing, it signals that there may be a problem with Apache Kafka or MongoDB.
You want to know the total number of records your connector wrote to MongoDB.	Use the `records` metric.
You want to know the total number of write errors your connector encountered when attempting to write to MongoDB.	Use the `batch-writes-failed` metric.
You want to know if your MongoDB performance is getting slower over time.	Use the `in-task-put-duration-ms` metric to initially diagnose a slowdown. Use the following metrics to further diagnose any issues: `batch-writes-successful-duration-over-<number>-ms` `batch-writes-failed-duration-over-<number>-ms` `processing-phase-duration-over-<number>-ms`
You want to find the time Kafka Connect and the MongoDB sink connector spend writing records to MongoDB.	Compare the values of the following metrics: `in-task-put-duration-ms` `in-connect-framework-duration-ms`

You can find descriptions of all MongoDB sink connector metrics in the Sink Connector JMX Metrics section.

Source Connector

The following table describes some use cases for monitoring the MongoDB source connector and the metrics the source connector provides to satisfy those use cases:

Use Case

Metrics to Use

You want to know if a component of your pipeline is falling behind.

Use the latest-mongodb-time-difference-secs metric. This metric indicates how old the most recent change stream event your connector processed is. If this metric is increasing, it signals that there may be a problem with Apache Kafka or MongoDB.

You want to know the total number of change stream events your source connector has processed.

Use the records metric.

You want to know the percentage of records your connector received but failed to write to Apache Kafka.

Perform the following calculation with the records, records-filtered, and records-acknowledged metrics:

(records - (records-acknowledged + records-filtered)) / records

You want to know the average size of the documents your connector has processed.

Perform the following calculation with the mongodb-bytes-read and records metrics:

mongodb-bytes-read / records

To learn how to calculate the average size of records over a span of time, see mongodb-bytes-read.

You want to find the time Kafka Connect and the MongoDB source connector spend writing records to Apache Kafka.

Compare the values of the following metrics:

in-task-poll-duration-ms
in-connect-framework-duration-ms

You want to know if your MongoDB performance is getting slower over time.

Use the in-task-poll-duration-ms metric to initially diagnose a slowdown.

Use the following metrics to further diagnose any issues:

initial-commands-successful-duration-over-<number>-ms
initial-commands-failed-duration-over-<number>-ms
getmore-commands-successful-duration-over-<number>-ms
getmore-commands-failed-duration-over-<number>-ms

You can find descriptions of all MongoDB source connector metrics in the Source Connector JMX Metrics section.

Monitor the Connector

The MongoDB Kafka Connector uses Java Management Extensions (JMX) to enable monitoring. JMX is a technology included in the Java Platform, Standard Edition that provides tools to monitor applications and devices. You can view the metrics produced by the connector with any JMX console, such as JConsole.

The MongoDB Kafka Connector provides metrics for individual tasks. Tasks are classes instantiated by Kafka Connect that copy data to and from datastores and Apache Kafka. The names and responsibilities of the two types of tasks in Kafka Connect are as follows:

A source task copies data from a data store to Apache Kafka.
A sink task copies data from Apache Kafka to a data store.

A sink connector configures one or more sink tasks. A source connector configures one or more source tasks.

To learn more about JMX, see the following resources from Oracle:

To learn more about tasks and connectors in Kafka Connect, see the following resources:

Enable Monitoring

The MongoDB Kafka Connector uses Kafka Connect's metrics infrastructure to serve metrics. To read the metrics produced by your connector, enable JMX in your Kafka Connect deployment.

To learn how to enable JMX for a Kafka Connect instance running on your host machine, see the Official Kafka Documentation.

To learn how to enable JMX for a containerized Kafka Connect deployment, see Kafka Monitoring and Metrics Using JMX with Docker.

Types of Metrics

The connector provides metrics related to the following types of quantities:

The number of times an event has occurred in total for a connector task
The value related to the most recent occurrence of an event

For some use cases, you must perform extra computations with the metrics the connector provides. For example, you can compute the following values from provided metrics:

The rate of change of a metric
The value of a metric over a span of time
The difference between one metric and another metric

To view some examples of computed metrics, see the Use Cases section.

JMX Paths

The MongoDB Kafka Connector and Kafka Connect both produce metrics for MongoDB connector tasks.

Both sets of metrics provide information about how your tasks interact with Kafka Connect, but only the MongoDB Kafka Connector metrics provide information about how your tasks interact with MongoDB.

The MongoDB Kafka Connector produces metrics under the following JMX paths:

com.mongodb.kafka.connect.sink-task-metrics.sink-task-<monitonically increasing number>
com.mongodb.kafka.connect.source-task-metrics.source-task-<monitonically increasing number>
com.mongodb.kafka.connect.source-task-metrics.source-task-change-stream-<monitonically increasing number>
com.mongodb.kafka.connect.source-task-metrics.source-task-copy-existing-<monitonically increasing number>

Kafka Connect produces metrics under the following JMX paths:

kafka.connect.sink-task-metrics.<connector-name>
kafka.connect.source-task-metrics.<connector-name>
kafka.connect.connector-task-metrics.<connector-name>

To relate Kafka Connect metrics to MongoDB Kafka Connector metrics, you must remember the order in which you added your connectors to Kafka Connect.

Note

Naming Conflicts

If the MongoDB Kafka Connector ever encounters a naming conflict when it attempts to register an MBean on a JMX path, the MongoDB Kafka Connector adds a version suffix to the MBean.

For example, if the connector tries to register an MBean under the path com.mongodb.kafka.connect.sink-task-metrics.sink-task-0 and is unable to do so, it attempts to register the MBean under com.mongodb.kafka.connect.sink-task-metrics.sink-task-0-v1.

Example

Assume you add a single MongoDB source connector named my-source-connector to your deployment.

The MongoDB source connector writes metrics to the following JMX path:

com.mongodb.kafka.connect.sink-task-metrics.sink-task-0

Kafka Connect writes metrics for this task under the following path:

kafka.connect.sink-task-metrics.my-source-connector

Example - Monitor the Quick Start

The sample environment provided in the Quick Start exposes metrics on your host machine at the URI localhost:35000.

To view these metrics with JConsole, perform the following actions:

Download JConsole.

JConsole is part of the Java Platform, Standard Edition. To download JConsole, download the Java SE Development Kit from Oracle.

Start the Quick Start pipeline and add connectors.

Follow the Quick Start until the Send the Contents of a Document through Your Connectors step.

Start JConsole.

Run the following command from your command line to start JConsole:

jconsole

Connect to the Kafka Connect JMX server.

Enter your JMX Server URI localhost:35000 into the Remote Process text input box in the JConsole interface.
Click Connect.
In the dialog box, click Insecure Connection.

Explore your connectors' metrics.

Navigate to the MBeans tab in JConsole.
Inspect connector metrics. Notice that the com.mongodb.kafka.connect.sink-task-metrics.sink-task-0.records attribute has a value of 0. This value indicates that your sink task has not received any records from Apache Kafka.
Continue the Quick Start until, but not through, the Remove the Sandbox step.
Navigate back to the MBeans tab in JConsole. The com.mongodb.kafka.connect.sink-task-metrics.sink-task-0.records attribute should now have a value of 1.

Stop and remove the Quick Start environment.

To stop and remove the Quick Start environment, follow the Remove the Sandbox step of the Quick Start.

Available Metrics

Use the attributes in the tables in this section to monitor the behavior of your source and sink connectors through Java Management Extensions (JMX).

Tip

JMX Attributes

JMX represents an individual metric as an attribute of an MBean. To learn more about attributes and MBeans, see the Standard MBeans Tutorial from Oracle.

Note

Poll and Put Methods

A MongoDB source connector task has a poll() method to retrieve documents from MongoDB and send them to Apache Kafka. A MongoDB sink connector task has a put() method to retrieve documents from Apache Kafka and send them to MongoDB.

To learn more about poll() and put() methods, see the following resources:

Sink Connector JMX Metrics

Attribute Name	Description
records	The total number of Kafka records a MongoDB sink task received.
records-successful	The total number of Kafka records a MongoDB sink task successfully wrote to MongoDB.
records-failed	The total number of Kafka records a MongoDB sink task failed to write to MongoDB.
latest-kafka-time-difference-ms	The number of milliseconds of the most recent time difference recorded between a MongoDB sink task and Kafka. This value is calculated by subtracting the current time of the connector's clock and the timestamp of the last record the task received.
in-task-put	The total number of times the Kafka Connect framework executed the `put()` method of the MongoDB sink task.
in-task-put-duration-ms	The total number of milliseconds the Kafka Connect framework spent executing the `put()` method of a MongoDB sink task.
in-task-put-duration-over-1-ms	The total number of MongoDB sink task `put()` method executions with a duration that exceeded 1 millisecond.
in-task-put-duration-over-10-ms	The total number of MongoDB sink task `put()` method executions with a duration that exceeded 10 milliseconds.
in-task-put-duration-over-100-ms	The total number of MongoDB sink task `put()` method executions with a duration that exceeded 100 milliseconds.
in-task-put-duration-over-1000-ms	The total number of MongoDB sink task `put()` method executions with a duration that exceeded 1000 milliseconds.
in-task-put-duration-over-10000-ms	The total number of MongoDB sink task `put()` method executions with a duration that exceeded 10000 milliseconds.
in-connect-framework	The total number of times code in the Kafka Connect framework executed after the first invocation of the `put()` method of the MongoDB sink task.
in-connect-framework-duration-ms	The total number of milliseconds spent executing code in the Kafka Connect framework since the framework first invoked the `put()` method of the MongoDB sink task. This metric does not count time executing code in the MongoDB sink task towards the total.
in-connect-framework-duration-over-1-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 1 millisecond.
in-connect-framework-duration-over-10-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 10 milliseconds.
in-connect-framework-duration-over-100-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 100 milliseconds.
in-connect-framework-duration-over-1000-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 1000 milliseconds.
in-connect-framework-duration-over-10000-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 10000 milliseconds.
processing-phases	The total number of times a MongoDB sink task executed the processing phase on a batch of records from Kafka. The processing phase of a MongoDB sink task is the set of actions that starts after records are obtained from Kafka and ends before records are written to MongoDB.
processing-phases-duration-ms	The total number of milliseconds a MongoDB sink task spent processing records before writing them to MongoDB.
processing-phases-duration-over-1-ms	The total number of MongoDB sink task processing phase executions with a duration that exceeded 1 millisecond.
processing-phases-duration-over-10-ms	The total number of MongoDB sink task processing phase executions with a duration that exceeded 10 milliseconds.
processing-phases-duration-over-100-ms	The total number of MongoDB sink task processing phase executions with a duration that exceeded 100 milliseconds.
processing-phases-duration-over-1000-ms	The total number of MongoDB sink task processing phase executions with a duration that exceeded 1000 milliseconds.
processing-phases-duration-over-10000-ms	The total number of MongoDB sink task processing phase executions with a duration that exceeded 10000 milliseconds.
batch-writes-successful	The total number of batches a MongoDB sink task successfully wrote to the MongoDB cluster.
batch-writes-successful-duration-ms	The total number of milliseconds a MongoDB sink task spent successfully writing to the MongoDB cluster.
batch-writes-successful-duration-over-1-ms	The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 1 millisecond.
batch-writes-successful-duration-over-10-ms	The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 10 milliseconds.
batch-writes-successful-duration-over-100-ms	The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 100 milliseconds.
batch-writes-successful-duration-over-1000-ms	The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 1000 milliseconds.
batch-writes-successful-duration-over-10000-ms	The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 10000 milliseconds.
batch-writes-failed	The total number of batches a MongoDB sink task failed to write to the MongoDB cluster.
batch-writes-failed-duration-ms	The total number of milliseconds a MongoDB sink task spent unsuccessfully attempting to write batches to the MongoDB cluster.
batch-writes-failed-duration-over-1-ms	The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 1 millisecond.
batch-writes-failed-duration-over-10-ms	The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 10 milliseconds.
batch-writes-failed-duration-over-100-ms	The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 100 milliseconds.
batch-writes-failed-duration-over-1000-ms	The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 1000 milliseconds.
batch-writes-failed-duration-over-10000-ms	The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 10000 milliseconds.

Source Connector JMX Metrics

Note

Initial Commands and getMore Commands

Some metrics for source connector tasks distinguish between initial commands and getMore commands. An initial command is a find or aggregate command sent to a MongoDB instance that retrieves the first set of documents in a client-side MongoDB cursor. The getMore command is the MongoDB command that fetches the subsequent sets of documents in a cursor.

To learn more about getMore commands, see the getMore page.

Attribute Name	Description
records	The total number of records a MongoDB source task passed to the Kafka Connect framework.
records-filtered	The number of records a MongoDB source task passed to the Kafka Connect framework that were then filtered by the framework. A filtered record is not written to Kafka.
records-acknowledged	The total number of records a MongoDB source task passed to the Kafka Connect framework that were then successfully written to Kafka.
mongodb-bytes-read	The total number of bytes a MongoDB source task read from the MongoDB cluster. To calculate the average size of the records your sink connector processed over a span of time, perform the following actions: Determine the change in the value of the `mongodb-bytes-read` attribute for a span of time. Determine the change in the value of the `records` attribute for the same span of time you used for the preceding step. Divide the change in the value of the `mongodb-bytes-read` attribute by the change in the value of the `records` attribute.
latest-mongodb-time-difference-secs	The number of seconds of the most recent time difference recorded between a MongoDB cluster and the post-batch resume token held by a MongoDB source task. This value is calculated by subtracting the timestamp of the task's post-batch resume token from the `operationTime` value of the most recent successful MongoDB command executed by the task.
in-task-poll	The total number of times the Kafka Connect framework executed the `poll()` method of a MongoDB source task.
in-task-poll-duration-ms	The total number of milliseconds the Kafka Connect framework spent executing the `poll()` method of a MongoDB source task.
in-task-poll-duration-over-1-ms	The total number of MongoDB source task `poll()` method executions with a duration that exceeded 1 millisecond.
in-task-poll-duration-over-10-ms	The total number of MongoDB source task `poll()` method executions with a duration that exceeded 10 milliseconds.
in-task-poll-duration-over-100-ms	The total number of MongoDB source task `poll()` method executions with a duration that exceeded 100 milliseconds.
in-task-poll-duration-over-1000-ms	The total number of MongoDB source task `poll()` method executions with a duration that exceeded 1000 milliseconds.
in-task-poll-duration-over-10000-ms	The total number of MongoDB source task `poll()` method executions with a duration that exceeded 10000 milliseconds.
in-connect-framework	The total number of times code in the Kafka Connect framework executed after the first invocation of the `poll()` method of the MongoDB source task.
in-connect-framework-duration-ms	The total number of milliseconds spent executing code in the Kafka Connect framework since the framework first invoked the `poll()` method of the MongoDB source task. This metric does not count time executing code in the MongoDB sink task towards the total.
in-connect-framework-duration-over-1-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 1 millisecond.
in-connect-framework-duration-over-10-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 10 milliseconds.
in-connect-framework-duration-over-100-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 100 milliseconds.
in-connect-framework-duration-over-1000-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 1000 milliseconds.
in-connect-framework-duration-over-10000-ms	The total number of times code in the Kafka Connect framework executed for a duration that exceeded 10000 milliseconds.
initial-commands-successful	The total number of initial commands issued by a MongoDB source task that succeeded. An initial command is a find or aggregate command sent to a MongoDB cluster that retrieves the first set of documents in a cursor. A `getMore` command is not an initial command.
initial-commands-successful-duration-ms	The total number of milliseconds a MongoDB source task spent executing initial commands that succeeded.
initial-commands-successful-duration-over-1-ms	The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 1 millisecond.
initial-commands-successful-duration-over-10-ms	The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 10 milliseconds.
initial-commands-successful-duration-over-100-ms	The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 100 milliseconds.
initial-commands-successful-duration-over-1000-ms	The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 1000 milliseconds.
initial-commands-successful-duration-over-10000-ms	The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 10000 milliseconds.
getmore-commands-successful	The total number of `getMore` commands issued by a MongoDB source task that succeeded.
getmore-commands-successful-duration-ms	The total number of milliseconds a MongoDB source task spent executing `getMore` commands that succeeded.
getmore-commands-successful-duration-over-1-ms	The total number of successful `getMore` commands issued by a MongoDB source task with a duration that exceeded 1 millisecond.
getmore-commands-successful-duration-over-10-ms	The total number of successful `getMore` commands issued by a MongoDB source task with a duration that exceeded 10 milliseconds.
getmore-commands-successful-duration-over-100-ms	The total number of successful `getMore` commands issued by a MongoDB source task with a duration that exceeded 100 milliseconds.
getmore-commands-successful-duration-over-1000-ms	The total number of successful `getMore` commands issued by a MongoDB source task with a duration that exceeded 1000 milliseconds.
getmore-commands-successful-duration-over-10000-ms	The total number of successful `getMore` commands issued by a MongoDB source task with a duration that exceeded 10000 milliseconds.
initial-commands-failed	The total number of initial commands issued by a MongoDB source task that failed. An initial command is a find or aggregate command sent to a MongoDB cluster that retrieves the first set of documents in a cursor. A `getMore` command is not an initial command.
initial-commands-failed-duration-ms	The total number of milliseconds a MongoDB source task spent unsuccessfully attempting to issue initial commands to the MongoDB server.
initial-commands-failed-duration-over-1-ms	The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 1 millisecond.
initial-commands-failed-duration-over-10-ms	The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 10 milliseconds.
initial-commands-failed-duration-over-100-ms	The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 100 milliseconds.
initial-commands-failed-duration-over-1000-ms	The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 1000 milliseconds.
initial-commands-failed-duration-over-10000-ms	The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 10000 milliseconds.
getmore-commands-failed	The total number of `getMore` commands issued by a MongoDB source task that failed.
getmore-commands-failed-duration-ms	The total number of milliseconds a MongoDB source task spent unsuccessfully attempting to issue `getMore` commands to the MongoDB server.
getmore-commands-failed-duration-over-1-ms	The total number of failed `getMore` commands issued by a MongoDB source task with a duration that exceeded 1 millisecond.
getmore-commands-failed-duration-over-10-ms	The total number of failed `getMore` commands issued by a MongoDB source task with a duration that exceeded 10 milliseconds.
getmore-commands-failed-duration-over-100-ms	The total number of failed `getMore` commands issued by a MongoDB source task with a duration that exceeded 100 milliseconds.
getmore-commands-failed-duration-over-1000-ms	The total number of failed `getMore` commands issued by a MongoDB source task with a duration that exceeded 1000 milliseconds.
getmore-commands-failed-duration-over-10000-ms	The total number of failed `getMore` commands issued by a MongoDB source task with a duration that exceeded 10000 milliseconds.

Back

Custom Authentication Provider

Migrate from Kafka Connect MongoDB