Monitoring
On this page
- Overview
- Use Cases
- Sink Connector
- Source Connector
- Monitor the Connector
- Enable Monitoring
- Types of Metrics
- JMX Paths
- Example - Monitor the Quick Start
- Download JConsole.
- Start the Quick Start pipeline and add connectors.
- Start JConsole.
- Connect to the Kafka Connect JMX server.
- Explore your connectors' metrics.
- Stop and remove the Quick Start environment.
- Available Metrics
- Sink Connector JMX Metrics
- Source Connector JMX Metrics
Overview
Learn how to observe the behavior of your MongoDB Kafka source connector or MongoDB Kafka sink connector through monitoring. Monitoring is the process of getting information about the activities a running program performs for use in an application or an application performance management library.
To learn how monitoring works in the connector and how to use it, see the Use Cases section.
To view an example that shows how to monitor a running connector, see the Example - Monitor the Quick Start section.
To view a list of all metrics produced by MongoDB source and sink connectors, see the Available Metrics section.
Use Cases
This section describes use cases for monitoring MongoDB source and sink connectors, and how you can use the metrics your connector provides to satisfy those use cases.
Tip
Computed Values
To learn what types of metrics the connector provides and when to implement logic to compute a value, see Types of Metrics.
Sink Connector
The following table describes some use cases for monitoring the MongoDB sink connector and the metrics the sink connector provides to satisfy those use cases:
Use Case | Metrics to Use |
---|---|
You want to know if a component of your pipeline is falling behind. | Use the latest-kafka-time-difference-ms
metric. This metric indicates the interval of time between
when a record arrived in a Kafka topic and when your connector
received that record. If the value of this metric is increasing,
it signals that there may be a problem with Apache Kafka or MongoDB. |
You want to know the total number of records your connector
wrote to MongoDB. | Use the records metric. |
You want to know the total number of write errors your connector
encountered when attempting to write to MongoDB. | Use the batch-writes-failed metric. |
You want to know if your MongoDB performance is getting slower
over time. | Use the Use the following metrics to further diagnose any issues:
|
You want to find the time Kafka Connect and the MongoDB sink
connector spend writing records to MongoDB. | Compare the values of the following metrics:
|
You can find descriptions of all MongoDB sink connector metrics in the Sink Connector JMX Metrics section.
Source Connector
The following table describes some use cases for monitoring the MongoDB source connector and the metrics the source connector provides to satisfy those use cases:
Use Case | Metrics to Use | |
---|---|---|
You want to know if a component of your pipeline is falling behind. | Use the latest-mongodb-time-difference-secs
metric. This metric indicates how old the most recent change
stream event your connector processed is. If this metric is increasing,
it signals that there may be a problem with Apache Kafka or MongoDB. | |
You want to know the total number of change stream events your source connector
has processed. | Use the records metric. | |
You want to know the percentage of records your connector
received but failed to write to Apache Kafka. | Perform the following calculation with the
| |
You want to know the average size of the documents your connector
has processed. | Perform the following calculation with the
To learn how to calculate the average size of records over a span of time, see mongodb-bytes-read. | |
You want to find the time Kafka Connect and the MongoDB
source connector spend writing records to Apache Kafka. | Compare the values of the following metrics:
| |
You want to know if your MongoDB performance is getting slower
over time. | Use the Use the following metrics to further diagnose any issues:
|
You can find descriptions of all MongoDB source connector metrics in the Source Connector JMX Metrics section.
Monitor the Connector
The MongoDB Kafka Connector uses Java Management Extensions (JMX) to enable monitoring. JMX is a technology included in the Java Platform, Standard Edition that provides tools to monitor applications and devices. You can view the metrics produced by the connector with any JMX console, such as JConsole.
The MongoDB Kafka Connector provides metrics for individual tasks. Tasks are classes instantiated by Kafka Connect that copy data to and from datastores and Apache Kafka. The names and responsibilities of the two types of tasks in Kafka Connect are as follows:
A source task copies data from a data store to Apache Kafka.
A sink task copies data from Apache Kafka to a data store.
A sink connector configures one or more sink tasks. A source connector configures one or more source tasks.
To learn more about JMX, see the following resources from Oracle:
To learn more about tasks and connectors in Kafka Connect, see the following resources:
Enable Monitoring
The MongoDB Kafka Connector uses Kafka Connect's metrics infrastructure to serve metrics. To read the metrics produced by your connector, enable JMX in your Kafka Connect deployment.
To learn how to enable JMX for a Kafka Connect instance running on your host machine, see the Official Kafka Documentation.
To learn how to enable JMX for a containerized Kafka Connect deployment, see Kafka Monitoring and Metrics Using JMX with Docker.
Types of Metrics
The connector provides metrics related to the following types of quantities:
The number of times an event has occurred in total for a connector task
The value related to the most recent occurrence of an event
For some use cases, you must perform extra computations with the metrics the connector provides. For example, you can compute the following values from provided metrics:
The rate of change of a metric
The value of a metric over a span of time
The difference between one metric and another metric
To view some examples of computed metrics, see the Use Cases section.
JMX Paths
The MongoDB Kafka Connector and Kafka Connect both produce metrics for MongoDB connector tasks.
Both sets of metrics provide information about how your tasks interact with Kafka Connect, but only the MongoDB Kafka Connector metrics provide information about how your tasks interact with MongoDB.
The MongoDB Kafka Connector produces metrics under the following JMX paths:
com.mongodb.kafka.connect.sink-task-metrics.sink-task-<monitonically increasing number>
com.mongodb.kafka.connect.source-task-metrics.source-task-<monitonically increasing number>
com.mongodb.kafka.connect.source-task-metrics.source-task-change-stream-<monitonically increasing number>
com.mongodb.kafka.connect.source-task-metrics.source-task-copy-existing-<monitonically increasing number>
Kafka Connect produces metrics under the following JMX paths:
kafka.connect.sink-task-metrics.<connector-name>
kafka.connect.source-task-metrics.<connector-name>
kafka.connect.connector-task-metrics.<connector-name>
To relate Kafka Connect metrics to MongoDB Kafka Connector metrics, you must remember the order in which you added your connectors to Kafka Connect.
Note
Naming Conflicts
If the MongoDB Kafka Connector ever encounters a naming conflict when it attempts
to register an MBean
on a JMX path, the MongoDB Kafka Connector adds a version suffix
to the MBean
.
For example, if the connector tries to register an MBean
under the path
com.mongodb.kafka.connect.sink-task-metrics.sink-task-0
and is unable to do
so, it attempts to register the MBean
under
com.mongodb.kafka.connect.sink-task-metrics.sink-task-0-v1
.
Example
Assume you add a single MongoDB source connector
named my-source-connector
to your deployment.
The MongoDB source connector writes metrics to the following JMX path:
com.mongodb.kafka.connect.sink-task-metrics.sink-task-0
Kafka Connect writes metrics for this task under the following path:
kafka.connect.sink-task-metrics.my-source-connector
Example - Monitor the Quick Start
The sample environment provided in the Quick Start exposes metrics
on your host machine at the URI localhost:35000
.
To view these metrics with JConsole, perform the following actions:
Download JConsole.
JConsole is part of the Java Platform, Standard Edition. To download JConsole, download the Java SE Development Kit from Oracle.
Start the Quick Start pipeline and add connectors.
Follow the Quick Start until the Send the Contents of a Document through Your Connectors step.
Explore your connectors' metrics.
Navigate to the MBeans tab in JConsole.
Inspect connector metrics. Notice that the
com.mongodb.kafka.connect.sink-task-metrics.sink-task-0.records
attribute has a value of0
. This value indicates that your sink task has not received any records from Apache Kafka.Continue the Quick Start until, but not through, the Remove the Sandbox step.
Navigate back to the MBeans tab in JConsole. The
com.mongodb.kafka.connect.sink-task-metrics.sink-task-0.records
attribute should now have a value of1
.
Stop and remove the Quick Start environment.
To stop and remove the Quick Start environment, follow the Remove the Sandbox step of the Quick Start.
Available Metrics
Use the attributes in the tables in this section to monitor the behavior of your source and sink connectors through Java Management Extensions (JMX).
Tip
JMX Attributes
JMX represents an individual metric as an attribute of an MBean
.
To learn more about attributes and MBeans
, see the
Standard MBeans Tutorial
from Oracle.
Note
Poll and Put Methods
A MongoDB source connector task has a poll()
method to retrieve
documents from MongoDB and send them to Apache Kafka. A MongoDB sink
connector task has a put()
method to retrieve documents from Apache Kafka
and send them to MongoDB.
To learn more about poll()
and put()
methods, see the following
resources:
Sink Connector JMX Metrics
Attribute Name | Description |
---|---|
records | The total number of Kafka records a MongoDB sink task received. |
records-successful | The total number of Kafka records a MongoDB sink task successfully
wrote to MongoDB. |
records-failed | The total number of Kafka records a MongoDB sink task failed to write
to MongoDB. |
latest-kafka-time-difference-ms | The number of milliseconds of the most recent time difference recorded
between a MongoDB sink task and Kafka. This value is calculated by
subtracting the current time of the connector's clock and the timestamp
of the last record the task received. |
in-task-put | The total number of times the Kafka Connect framework executed the
put() method of the MongoDB sink task. |
in-task-put-duration-ms | The total number of milliseconds the Kafka Connect framework spent
executing the put() method of a MongoDB sink task. |
in-task-put-duration-over-1-ms | The total number of MongoDB sink task put() method executions
with a duration that exceeded 1 millisecond. |
in-task-put-duration-over-10-ms | The total number of MongoDB sink task put() method executions
with a duration that exceeded 10 milliseconds. |
in-task-put-duration-over-100-ms | The total number of MongoDB sink task put() method executions
with a duration that exceeded 100 milliseconds. |
in-task-put-duration-over-1000-ms | The total number of MongoDB sink task put() method executions
with a duration that exceeded 1000 milliseconds. |
in-task-put-duration-over-10000-ms | The total number of MongoDB sink task put() method executions
with a duration that exceeded 10000 milliseconds. |
in-connect-framework | The total number of times code in the Kafka Connect framework
executed after the first invocation of the put() method of the
MongoDB sink task. |
in-connect-framework-duration-ms | The total number of milliseconds spent executing code in the Kafka
Connect framework since the framework first invoked the put()
method of the MongoDB sink task. This metric does not count time
executing code in the MongoDB sink task towards the total. |
in-connect-framework-duration-over-1-ms | The total number of times code in the Kafka Connect framework
executed for a duration that exceeded 1 millisecond. |
in-connect-framework-duration-over-10-ms | The total number of times code in the Kafka Connect framework
executed for a duration that exceeded 10 milliseconds. |
in-connect-framework-duration-over-100-ms | The total number of times code in the Kafka Connect framework
executed for a duration that exceeded 100 milliseconds. |
in-connect-framework-duration-over-1000-ms | The total number of times code in the Kafka Connect framework executed
for a duration that exceeded 1000 milliseconds. |
in-connect-framework-duration-over-10000-ms | The total number of times code in the Kafka Connect framework executed
for a duration that exceeded 10000 milliseconds. |
processing-phases | The total number of times a MongoDB sink task executed the processing
phase on a batch of records from Kafka. The processing phase of a
MongoDB sink task is the set of actions that starts after records are
obtained from Kafka and ends before records are written to MongoDB. |
processing-phases-duration-ms | The total number of milliseconds a MongoDB sink task spent processing
records before writing them to MongoDB. |
processing-phases-duration-over-1-ms | The total number of MongoDB sink task processing phase executions with
a duration that exceeded 1 millisecond. |
processing-phases-duration-over-10-ms | The total number of MongoDB sink task processing phase executions with
a duration that exceeded 10 milliseconds. |
processing-phases-duration-over-100-ms | The total number of MongoDB sink task processing phase executions with
a duration that exceeded 100 milliseconds. |
processing-phases-duration-over-1000-ms | The total number of MongoDB sink task processing phase executions with
a duration that exceeded 1000 milliseconds. |
processing-phases-duration-over-10000-ms | The total number of MongoDB sink task processing phase executions with
a duration that exceeded 10000 milliseconds. |
batch-writes-successful | The total number of batches a MongoDB sink task successfully wrote
to the MongoDB cluster. |
batch-writes-successful-duration-ms | The total number of milliseconds a MongoDB sink task spent successfully
writing to the MongoDB cluster. |
batch-writes-successful-duration-over-1-ms | The total number of successful batch writes performed by the MongoDB
sink task with a duration that exceeded 1 millisecond. |
batch-writes-successful-duration-over-10-ms | The total number of successful batch writes performed by the MongoDB
sink task with a duration that exceeded 10 milliseconds. |
batch-writes-successful-duration-over-100-ms | The total number of successful batch writes performed by the MongoDB
sink task with a duration that exceeded 100 milliseconds. |
batch-writes-successful-duration-over-1000-ms | The total number of successful batch writes performed by the MongoDB
sink task with a duration that exceeded 1000 milliseconds. |
batch-writes-successful-duration-over-10000-ms | The total number of successful batch writes performed by the MongoDB
sink task with a duration that exceeded 10000 milliseconds. |
batch-writes-failed | The total number of batches a MongoDB sink task failed to write
to the MongoDB cluster. |
batch-writes-failed-duration-ms | The total number of milliseconds a MongoDB sink task spent
unsuccessfully attempting to write batches to the MongoDB cluster. |
batch-writes-failed-duration-over-1-ms | The total number of failed batch writes attempted by the MongoDB
sink task with a duration that exceeded 1 millisecond. |
batch-writes-failed-duration-over-10-ms | The total number of failed batch writes attempted by the MongoDB
sink task with a duration that exceeded 10 milliseconds. |
batch-writes-failed-duration-over-100-ms | The total number of failed batch writes attempted by the MongoDB sink
task with a duration that exceeded 100 milliseconds. |
batch-writes-failed-duration-over-1000-ms | The total number of failed batch writes attempted by the MongoDB sink
task with a duration that exceeded 1000 milliseconds. |
batch-writes-failed-duration-over-10000-ms | The total number of failed batch writes attempted by the MongoDB sink
task with a duration that exceeded 10000 milliseconds. |
Source Connector JMX Metrics
Note
Initial Commands and getMore Commands
Some metrics for source connector tasks distinguish between
initial commands and getMore
commands. An initial
command is a find
or aggregate
command sent to a MongoDB
instance that retrieves the first set of documents in a client-side
MongoDB cursor. The getMore
command is the MongoDB command that fetches
the subsequent sets of documents in a cursor.
To learn more about getMore
commands, see the
getMore page.
Attribute Name | Description |
---|---|
records | The total number of records a MongoDB source task passed to the
Kafka Connect framework. |
records-filtered | The number of records a MongoDB source task passed to the
Kafka Connect framework that were then filtered by the framework.
A filtered record is not written to Kafka. |
records-acknowledged | The total number of records a MongoDB source task passed to the
Kafka Connect framework that were then successfully
written to Kafka. |
mongodb-bytes-read | The total number of bytes a MongoDB source task read from the MongoDB cluster. To calculate the average size of the records your sink connector processed over a span of time, perform the following actions:
|
latest-mongodb-time-difference-secs | The number of seconds of the most recent time difference recorded between
a MongoDB cluster and the post-batch resume token held by a MongoDB
source task. This value is calculated by subtracting the timestamp
of the task's post-batch resume token from the operationTime value of
the most recent successful MongoDB command executed by the task. |
in-task-poll | The total number of times the Kafka Connect framework executed
the poll() method of a MongoDB source task. |
in-task-poll-duration-ms | The total number of milliseconds the Kafka Connect framework
spent executing the poll() method of a MongoDB source task. |
in-task-poll-duration-over-1-ms | The total number of MongoDB source task poll() method executions
with a duration that exceeded 1 millisecond. |
in-task-poll-duration-over-10-ms | The total number of MongoDB source task poll() method executions
with a duration that exceeded 10 milliseconds. |
in-task-poll-duration-over-100-ms | The total number of MongoDB source task poll() method executions
with a duration that exceeded 100 milliseconds. |
in-task-poll-duration-over-1000-ms | The total number of MongoDB source task poll() method executions
with a duration that exceeded 1000 milliseconds. |
in-task-poll-duration-over-10000-ms | The total number of MongoDB source task poll() method executions
with a duration that exceeded 10000 milliseconds. |
in-connect-framework | The total number of times code in the Kafka Connect framework
executed after the first invocation of the poll() method of the
MongoDB source task. |
in-connect-framework-duration-ms | The total number of milliseconds spent executing code in the
Kafka Connect framework since the framework first invoked the
poll() method of the MongoDB source task. This metric does
not count time executing code in the MongoDB sink task towards the total. |
in-connect-framework-duration-over-1-ms | The total number of times code in the Kafka Connect framework
executed for a duration that exceeded 1 millisecond. |
in-connect-framework-duration-over-10-ms | The total number of times code in the Kafka Connect framework
executed for a duration that exceeded 10 milliseconds. |
in-connect-framework-duration-over-100-ms | The total number of times code in the Kafka Connect framework
executed for a duration that exceeded 100 milliseconds. |
in-connect-framework-duration-over-1000-ms | The total number of times code in the Kafka Connect framework
executed for a duration that exceeded 1000 milliseconds. |
in-connect-framework-duration-over-10000-ms | The total number of times code in the Kafka Connect framework
executed for a duration that exceeded 10000 milliseconds. |
initial-commands-successful | The total number of initial commands issued by a MongoDB source
task that succeeded. An initial command is a find or aggregate command
sent to a MongoDB cluster that retrieves the first set of documents in a
cursor. A getMore command is not an initial command. |
initial-commands-successful-duration-ms | The total number of milliseconds a MongoDB source task spent executing
initial commands that succeeded. |
initial-commands-successful-duration-over-1-ms | The total number of successful initial commands issued by a MongoDB
source task with a duration that exceeded 1 millisecond. |
initial-commands-successful-duration-over-10-ms | The total number of successful initial commands issued by a MongoDB
source task with a duration that exceeded 10 milliseconds. |
initial-commands-successful-duration-over-100-ms | The total number of successful initial commands issued by a MongoDB
source task with a duration that exceeded 100 milliseconds. |
initial-commands-successful-duration-over-1000-ms | The total number of successful initial commands issued by a MongoDB
source task with a duration that exceeded 1000 milliseconds. |
initial-commands-successful-duration-over-10000-ms | The total number of successful initial commands issued by a MongoDB
source task with a duration that exceeded 10000 milliseconds. |
getmore-commands-successful | The total number of getMore commands issued by a MongoDB source
task that succeeded. |
getmore-commands-successful-duration-ms | The total number of milliseconds a MongoDB source task spent executing
getMore commands that succeeded. |
getmore-commands-successful-duration-over-1-ms | The total number of successful getMore commands issued by a
MongoDB source task with a duration that exceeded 1 millisecond. |
getmore-commands-successful-duration-over-10-ms | The total number of successful getMore commands issued by a MongoDB
source task with a duration that exceeded 10 milliseconds. |
getmore-commands-successful-duration-over-100-ms | The total number of successful getMore commands issued by a MongoDB
source task with a duration that exceeded 100 milliseconds. |
getmore-commands-successful-duration-over-1000-ms | The total number of successful getMore commands issued by a
MongoDB source task with a duration that exceeded 1000 milliseconds. |
getmore-commands-successful-duration-over-10000-ms | The total number of successful getMore commands issued by a MongoDB
source task with a duration that exceeded 10000 milliseconds. |
initial-commands-failed | The total number of initial commands issued by a MongoDB source
task that failed. An initial command is a find or aggregate command
sent to a MongoDB cluster that retrieves the first set of documents in a
cursor. A getMore command is not an initial command. |
initial-commands-failed-duration-ms | The total number of milliseconds a MongoDB source task spent
unsuccessfully attempting to issue initial commands to the MongoDB
server. |
initial-commands-failed-duration-over-1-ms | The total number of failed initial commands issued by a MongoDB
source task with a duration that exceeded 1 millisecond. |
initial-commands-failed-duration-over-10-ms | The total number of failed initial commands issued by a
MongoDB source task with a duration that exceeded 10 milliseconds. |
initial-commands-failed-duration-over-100-ms | The total number of failed initial commands issued by a MongoDB source
task with a duration that exceeded 100 milliseconds. |
initial-commands-failed-duration-over-1000-ms | The total number of failed initial commands issued by a MongoDB source
task with a duration that exceeded 1000 milliseconds. |
initial-commands-failed-duration-over-10000-ms | The total number of failed initial commands issued by a MongoDB
source task with a duration that exceeded 10000 milliseconds. |
getmore-commands-failed | The total number of getMore commands issued by a MongoDB source
task that failed. |
getmore-commands-failed-duration-ms | The total number of milliseconds a MongoDB source task spent
unsuccessfully attempting to issue getMore commands to the MongoDB
server. |
getmore-commands-failed-duration-over-1-ms | The total number of failed getMore commands issued by a MongoDB source
task with a duration that exceeded 1 millisecond. |
getmore-commands-failed-duration-over-10-ms | The total number of failed getMore commands issued by a MongoDB source
task with a duration that exceeded 10 milliseconds. |
getmore-commands-failed-duration-over-100-ms | The total number of failed getMore commands issued by a MongoDB source
task with a duration that exceeded 100 milliseconds. |
getmore-commands-failed-duration-over-1000-ms | The total number of failed getMore commands issued by a MongoDB source
task with a duration that exceeded 1000 milliseconds. |
getmore-commands-failed-duration-over-10000-ms | The total number of failed getMore commands issued by a MongoDB
source task with a duration that exceeded 10000 milliseconds. |