Invalid Resume Token

On this page

Overview

Stack Trace
Cause
Solutions
Temporarily Tolerate Errors
Reset Stored Offsets
Prevention

Overview

Learn how to recover from an invalid resume token in a MongoDB Kafka source connector.

Stack Trace

The following stack trace indicates that the source connector has an invalid resume token:

...
org.apache.kafka.connect.errors.ConnectException: ResumeToken not found.
Cannot create a change stream cursor
...
Command failed with error 286 (ChangeStreamHistoryLost): 'PlanExecutor
error during aggregation :: caused by :: Resume of change stream was not
possible, as the resume point may no longer be in the oplog
...

Cause

When the ID of your source connector's resume token does not correspond to any entry in your MongoDB deployment's oplog, your connector has no way to determine where to begin to process your MongoDB change stream. Click the following tabs to see scenarios in which you can experience this issue:

In this scenario, you pause your source connector and you fill your MongoDB deployment's oplog:

You start a Kafka deployment with a MongoDB Kafka source connector.
You produce a change event in your source MongoDB namespace, and your connector stores a resume token corresponding to this event.
You pause your source connector.
While your connector pauses, you fill your MongoDB oplog such that MongoDB deletes the oplog entry corresponding to your resume token.
You restart your source connector and it is unable to resume processing as its resume token does not exist in your MongoDB oplog.

In this scenario, your source connector listens for changes on an infrequently updated MongoDB namespace and the heartbeat feature is not enabled:

You start a Kafka deployment with a MongoDB Kafka source connector.
You produce a change event in your source MongoDB namespace, and your connector stores a resume token corresponding to this event.
Your source MongoDB namespace is not updated in the time it takes for your MongoDB deployment to rotate the change event corresponding to your resume token out of its oplog.
You produce a change event in your source MongoDB namespace and your source connector is unable to resume processing as its resume token does not exist in your MongoDB oplog.

For more information on the oplog, see the MongoDB Manual.

For more information on change streams, see the Change Streams guide.

Solutions

You can recover from an invalid resume token using one of the following strategies:

Temporarily Tolerate Errors
Reset Stored Offsets

Temporarily Tolerate Errors

You can configure your source connector to tolerate errors while you produce a change stream event that updates the connector's resume token. This recovery strategy is the simplest, but there is a risk that your connector briefly ignores errors unrelated to the invalid resume token. If you aren't comfortable briefly tolerating errors in your deployment, you can delete stored offsets instead.

To configure your source connector to temporarily tolerate errors:

Set the errors.tolerance option to tolerate all errors:
```
errors.tolerance=all
```
Insert, update, or delete a document in the collection referenced by your source connector to produce a change stream event that updates your connector's resume token.
Once you produce a change stream event, set the errors.tolerance option to no longer tolerate errors:
```
errors.tolerance=none
```

For more information on the errors.tolerance option, see the Error Handling and Resuming from Interruption Properties page.

Reset Stored Offsets

You can reset your Kafka Connect offset data, which contains your resume token, to allow your connector to resume processing your change stream.

To reset your offset data, change the value of the offset.partition.name configuration property to a partition name that does not exist on your Kafka deployment. You can set your offset.partition.name property like this:

offset.partition.name=<a string>

Tip

Naming your Offset Partitions

Consider using the following pattern to name your offset partitions:

offset.partition.name=<source connector name>.<monotonically increasing number>

This pattern provides the following benefits:

Records the number of times you reset your connector
Documents to which connector an offset partition belongs

For example, if you named your source connector "source-values" and you are setting the offset.partition.name property for the first time, configure your connector as follows:

offset.partition.name=source-values.1

The next time you reset your connector's offset data, configure your connector as follows:

offset.partition.name=source-values.2

To learn more about the offset.partition.name configuration property, see the Error Handling and Resuming from Interruption Properties page.

To learn about naming your connector, see the official Apache Kafka documentation.

Prevention

To prevent invalid resume token errors caused by an infrequently updated namespace, enable heartbeats. Heartbeats is a feature of your source connector that causes your connector to update its resume token at regular intervals as well as when the contents of your source MongoDB namespace changes.

Specify the following option in your source connector configuration to enable heartbeats:

heartbeat.interval.ms=<a positive integer>

To learn more about heartbeats, see the Error Handling and Resuming from Interruption Properties guide.

Back

Troubleshooting

How to Contribute