Kafka Connector Tutorial Setup
The tutorials in this section run on a development environment using Docker to package the dependencies and configurations required to run the MongoDB Connector for Apache Kafka. Make sure you complete the development environment setup steps before proceeding to the tutorials.
Requirements
Create or sign into your Docker account to download the Docker containers you need for this tutorial. To learn how to sign up for an account and install Docker Desktop, read the signup and download steps in the Docker Hub Quickstart.
A terminal app and shell. For MacOS users, use Terminal or a similar app. For Windows users, use PowerShell.
Optional. Install git to download the setup files. To learn how to install git, read the Git Downloads page.
Set Up Your Development Environment with Docker
Clone or Download the Tutorial Repository
Next, clone the tutorial git repository with the following command:
git clone https://github.com/mongodb-university/kafka-edu.git
If you do not have git installed, you can download the zip archive instead.
Run the Development Environment
Select the tab that matches your OS for instructions on how to run the commands in this guide:
Navigate to the tutorial directory "mongodb-kafka-base" within the repository or unzipped archive using bash shell. If you cloned the repository with git, your command resembles the following:
cd kafka-edu/docs-examples/mongodb-kafka-base/
Navigate to the tutorial directory "mongodb-kafka-base" within the repository or unzipped archive using PowerShell. If you cloned the repository with git, your command resembles the following:
cd kafka-edu\docs-examples\mongodb-kafka-base\
Start the Docker image with the following command:
docker-compose -p mongo-kafka up -d --force-recreate
Note
Port Mappings
The sandbox maps the following services to ports on your host machine:
The sandbox MongoDB server maps to port
35001
on your host machineThe sandbox Kafka Connect JMX server maps to port
35000
on your host machine
These ports must be free to start the sandbox.
The "mongo-kafka-base" image creates a Docker container that includes all the services you need in the tutorial and runs them on a shared network called "mongodb-kafka-base_localnet" as shown in the following diagram:
When the command completes successfully, it outputs the following text:
... Creating zookeeper ... done Creating broker ... done Creating schema-registry ... done Creating connect ... done Creating rest-proxy ... done Creating mongo1 ... done Creating mongo1-setup ... done
Verify the Successful Setup
Confirm that the development environment started successfully by running the following commands:
docker exec mongo1 status
This command should output the following information if the Docker development environment was set up successfully:
Kafka topics: "topic": "docker-connect-status", "topic": "docker-connect-offsets", "topic": "docker-connect-configs", "topic": "__consumer_offsets", The status of the connectors: Currently configured connectors [] Version of MongoDB Connector for Apache Kafka installed: {"class":"com.mongodb.kafka.connect.MongoSinkConnector","type":"sink","version":"1.8.0"} {"class":"com.mongodb.kafka.connect.MongoSourceConnector","type":"source","version":"1.8.0"}
Since you have not started the connectors, the status and configured list are empty.
Your development environment setup is complete and you can proceed to the next step of the tutorial.
Tip
Connect To Development Environment MongoDB
You can connect to the MongoDB cluster running in your development environment with the following connection string:
mongodb://localhost:35001/?directConnection=true