Kafka Connector Tutorial Setup

The tutorials in this section run on a development environment using Docker to package the dependencies and configurations required to run the MongoDB Connector for Apache Kafka. Make sure you complete the development environment setup steps before proceeding to the tutorials.

Requirements

Create or sign into your Docker account to download the Docker containers you need for this tutorial. To learn how to sign up for an account and install Docker Desktop, read the signup and download steps in the Docker Hub Quickstart.
A terminal app and shell. For MacOS users, use Terminal or a similar app. For Windows users, use PowerShell.
Optional. Install git to download the setup files. To learn how to install git, read the Git Downloads page.

Set Up Your Development Environment with Docker

Clone or Download the Tutorial Repository

Next, clone the tutorial git repository with the following command:

git clone https://github.com/mongodb-university/kafka-edu.git

If you do not have git installed, you can download the zip archive instead.

Run the Development Environment

Select the tab that matches your OS for instructions on how to run the commands in this guide:

Navigate to the tutorial directory "mongodb-kafka-base" within the repository or unzipped archive using bash shell. If you cloned the repository with git, your command resembles the following:

cd kafka-edu/docs-examples/mongodb-kafka-base/

Navigate to the tutorial directory "mongodb-kafka-base" within the repository or unzipped archive using PowerShell. If you cloned the repository with git, your command resembles the following:

cd kafka-edu\docs-examples\mongodb-kafka-base\

Start the Docker image with the following command:

docker-compose -p mongo-kafka up -d --force-recreate

Note

Port Mappings

The sandbox maps the following services to ports on your host machine:

The sandbox MongoDB server maps to port 35001 on your host machine
The sandbox Kafka Connect JMX server maps to port 35000 on your host machine

These ports must be free to start the sandbox.

The "mongo-kafka-base" image creates a Docker container that includes all the services you need in the tutorial and runs them on a shared network called "mongodb-kafka-base_localnet" as shown in the following diagram:

Diagram that shows the Docker compose containers in mongo-kafka-base

When the command completes successfully, it outputs the following text:

...
Creating zookeeper ... done
Creating broker    ... done
Creating schema-registry ... done
Creating connect         ... done
Creating rest-proxy      ... done
Creating mongo1          ... done
Creating mongo1-setup    ... done

Verify the Successful Setup

Confirm that the development environment started successfully by running the following commands:

docker exec mongo1 status

This command should output the following information if the Docker development environment was set up successfully:

Kafka topics:
    "topic": "docker-connect-status",
    "topic": "docker-connect-offsets",
    "topic": "docker-connect-configs",
    "topic": "__consumer_offsets",
The status of the connectors:
Currently configured connectors
[]
Version of MongoDB Connector for Apache Kafka installed:
{"class":"com.mongodb.kafka.connect.MongoSinkConnector","type":"sink","version":"1.8.0"}
{"class":"com.mongodb.kafka.connect.MongoSourceConnector","type":"source","version":"1.8.0"}

Since you have not started the connectors, the status and configured list are empty.

Your development environment setup is complete and you can proceed to the next step of the tutorial.

Tip

Connect To Development Environment MongoDB

You can connect to the MongoDB cluster running in your development environment with the following connection string:

mongodb://localhost:35001/?directConnection=true

Back

Tutorials

Explore MongoDB Change Streams