Apache Kafka Connections

Configure your Confluent cluster.

You must configure your Confluent cluster to accept incoming connections from your Atlas project.

1. Call the Return Account ID and VPC ID for group and region Atlas Administration API endpoint. Note the value of awsAccountId, you will need this in a later step.

2. In your Confluent account, navigate to the cluster you want to connect to. In your cluster networking interface, navigate to your cluster networking details.

3. Add Private Link Access

For a Confluent dedicated cluster, provide a name of your choice. For the AWS account number, provide the value of the awsAccountId field you noted earlier.

Request a connection to your cloud provider.

The Atlas Administration API provides an endpoint for requesting a Private Link connection configured for Atlas Stream Processing.

Create One Private Link

For an AWS Confluent Private Link connection, you must set the following key-value pairs:

You can find these values in your Confluent cluster's networking details.

The following example command requests a connection to your Confluent cluster and illustrates a typical response:

curl --location 'https://cloud.mongodb.com/api/atlas/v2/groups/8358217d3abb5c76c3434648/streams/privateLinkConnections' \
--digest \
--user "slrntglrbn:933fb118-ac62-4991-db05-ee67a3481fde" \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.atlas.2023-02-01+json' \
--data '{ "vendor": "Confluent", "provider": "AWS",
  "region": "us_east_1", "serviceEndpointId":
  "com.amazonaws.vpce.us-east-1.vpce-svc-93da685022ee702a9",
  "dnsDomain": "sample.us-east-1.aws.confluent.cloud",
  "dnsSubDomain: [
    "use1-az1.sample.us-east-1.aws.confluent.cloud",
    "use1-az2.sample.us-east-1.aws.confluent.cloud",
    "use1-az4.sample.us-east-1.aws.confluent.cloud"
  ]
}'

{"_id":"6aa12e7ccd660d4b2380b1c1","dnsDomain":"sample.us-east-1.aws.confluent.cloud.","vendor":"Confluent","provider":"AWS","region":"us_east_1","serviceEndpointId":"com.amazonaws.vpce.us-east-1.vpce-svc-93da685022ee702a9"}

After you send the request, note the value of the _id field in the response body. You will need this in a later step.

Provide the interface endpoint ID to Confluent.

Call the Return All Private Link Connections endpoint. Note the value of interfaceEndpointId.

In your Confluent account, navigate to the cluster you want to connect to. In your cluster networking interface, navigate to your cluster networking details. Navigate to the access points interface, and add a new access point. When Confluent prompts you for an interface endpoint, provide the value of interfaceEndpointId that you noted previously.

Create the Atlas-side connection.

Add a connection with the following key-value pairs:

Set all other values as necessary.

The following example command creates a Apache Kafka connection in Atlas:

curl --location 'https://cloud.mongodb.com/api/atlas/v2/groups/8358217d3abb5c76c3434648/streams/spinstance/connections' \
--digest \
--user "slrntglrbn:933fb118-ac62-4991-db05-ee67a3481fde" \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.atlas.2023-02-01+json' \
--data '{
  "name": "confluent_demo",
  "bootstrapServers": "slr-ntgrbn.sample.us-east-1.aws.confluent.cloud:9092",
  "security": {
    "protocol": "SASL_SSL"
  },
  "authentication": {
    "mechanism": "PLAIN",
    "password": "apiSecretDemo",
    "username": "apiUserDemo"
  },
  "type": "Kafka",
  "networking": {
    "access": {
    "type": "PRIVATE_LINK",
    "connectionId": "38972b0cbe9c2aa40a30a246"
    }
  }
}'

Configure your AWS MSK cluster.

You must configure your AWS MSK cluster to accept incoming connections from your Atlas project.

1. Use the Get Account Details endpoint to retrieve the AWS Principal identity. You will need this value for your AWS MSK cluster policy.

2. Sign in to the AWS Management Console and navigate to the AWS MSK console. Ensure that multi-VPC connectivity is enabled on the cluster to which you want to connect.

3. Click Properties, Security Settings, and Edit cluster policy.

   Provide the full ARN form of the Principal identity you retrieved earlier as the value of Statement.Principal.Aws.[] and ensure the policy takes the following form:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": [
                       "arn:aws:iam:123456789012:root"
                   ]
               },
               "Action": [
                   "kafka:CreateVpcConnection",
                   "kafka:GetBootstrapBrokers",
                   "kafka:DescribeCluster",
                   "kafka:DescribeClusterV2"
               ],
               "Resource": "arn:aws:kafka:us-east-1:123456789012:cluster/testing/de8982fa-8222-4e87-8b20-9bf3cdfa1521-2"
           }
       ]
   }

Request a connection to your cloud provider.

The Atlas Administration API provides an endpoint for requesting a Private Link connection configured for Atlas Stream Processing.

Create One Private Link

For an AWS MSK Private Link connection, you must set the following key-value pairs:

You can find the ARN in your AWS MSK cluster's networking details.

The following example command requests a connection to your AWS MSK cluster and illustrates a typical response:

curl --location 'https://cloud.mongodb.com/api/atlas/v2/groups/8358217d3abb5c76c3434648/streams/privateLinkConnections' \
--digest \
--user "slrntglrbn:933fb118-ac62-4991-db05-ee67a3481fde" \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.atlas.2023-02-01+json' \
--data '{ "vendor": "msk",
  "provider": "AWS",
  "arn": "1235711"}'

{"_id":"6aa12e7ccd660d4b2380b1c1","dnsDomain":"scram.sample.us-east-1.amazonaws.com","vendor":"msk","provider":"AWS","region":"us_east_1","serviceEndpointId":"com.amazonaws.vpce.us-east-1.vpce-svc-93da685022ee702a9"}

After you send the request, note the value of the _id field in the response body. You will need this in a later step.

Create the Atlas-side connection.

Add a connection with the following key-value pairs:

Set all other values as necessary.

The following example command creates a Apache Kafka connection in Atlas:

curl --location 'https://cloud.mongodb.com/api/atlas/v2/groups/8358217d3abb5c76c3434648/streams/spinstance/connections' \
--digest \
--user "slrntglrbn:933fb118-ac62-4991-db05-ee67a3481fde" \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.atlas.2023-02-01+json' \
--data '{
  "name": "msk_demo",
  "bootstrapServers": "slr-ntgrbn.sample.us-east-1.amazonaws.com:9092",
  "security": {
    "protocol": "SASL_SSL"
  },
  "authentication": {
    "mechanism": "PLAIN",
    "password": "scramSecretDemo",
    "username": "scramUserDemo"
  },
  "type": "Kafka",
  "networking": {
    "access": {
    "type": "PRIVATE_LINK",
    "connectionId": "38972b0cbe9c2aa40a30a246"
    }
  }
}'