Manage Connection Registries

On this page

Prerequisites

View Connections in the Connection Registry
Add a Connection to the Connection Registry
Modify a Connection from the Connection Registry
Delete a Connection from the Connection Registry
List Private Link Connections
View One Private Link Connection
Create A Private Link Connection
Delete a Private Link Connection

Each Stream Processing Instance stores every configured connection to a streaming data source or sink in its connection registry. The procedures on this page walk you through creating and configuring your Atlas Stream Processing connections.

Prerequisites

To create and configure connections, you must have the following:

An Atlas project
An Atlas user with the Project Owner or the Project Stream Processing Owner role to manage a connection registry
Note
The Project Owner role allows you to create database deployments, manage project access and project settings, manage IP Access List entries, and more.
The Project Stream Processing Owner role enables Atlas Stream Processing actions such as viewing, creating, deleting, and editing stream processing instances, and viewing, adding, modifying, and deleting connections in the connection registry.
See Project Roles to learn more about the differences between the two roles.
An Atlas cluster

View Connections in the Connection Registry

To view a connection to your stream processing instance:

To list all connections from one stream processing instance you specify using the Atlas CLI, run the following command:

atlas streams connections list [options]

To return the details for one stream processing connection you specify using the Atlas CLI, run the following command:

atlas streams connections describe <streamConnectionName> [options]

To learn more about the syntax and parameters for the previous commands, see the Atlas CLI documentation for atlas streams connections list and atlas streams connections describe.

Tip

See: Related Links

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.

Go to the Connection Registry page.

Locate the overview panel of the stream processing instance containing the connection registry you want to view.
Click Configure.
Select the Connection Registry tab.

Atlas Stream Processing displays all of the connections available in your instance connection registry.

The Atlas Administration API provides different endpoints for retrieving one or all connections in a connection registry.

View One Connection in a Connection Registry

List All Connections in a Connection Registry

To list all available connections using mongosh, use the sp.listConnections() method. It returns a list of documents describing each connection in the stream processing instance's connection registry. It has the following syntax:

sp.listConnections()

Add a Connection to the Connection Registry

Atlas Stream Processing supports the following connection types:

Apache Kafka brokers
Atlas databases
HTTPS connections

Each type has distinct configuration requirements.

Add an Apache Kafka Connection

Important

VPC peering for Atlas Stream Processing is currently in private preview. To learn more, contact your account team.

To add a Kafka connection to your stream processing instance:

To create one connection for the stream processing instance you specify using the Atlas CLI, run the following command:

atlas streams connections create [connectionName] [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections create.

Tip

See: Related Links

When you create a stream processing instance using the Atlas CLI, you must provide a .json configuration file to define the new connection's parameters. The format of this file depends on the type of connection you define.

For an Apache Kafka connection, provide a configuration file with the following syntax:

{
  "name": "<name>",
  "type": "Kafka",
  "bootstrapServers": "<address:port>, ...",
  "security": {
    "protocol": "<protocol>"
  },
  "authentication": {
    "mechanism": "<auth-mechanism>",
    "username": "<username>",
    "password": "<password>"
  }
}

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.

Go to the Connection Registry.

Locate the overview panel of the stream processing instance you want to modify and click Configure.
Select the Connection Registry tab.

Click + Add connection.

Add a new connection.

Select a Kafka connection.
Provide a Connection Name. Each connection name must be unique within a stream processing instance. This is the name used to reference the connection in Atlas Stream Processing aggregations.
Select a Network Access type. Atlas Stream Processing supports Public IP or VPC Peering connections.
Click the Public IP button. No further configuration is needed for this network access type.
Click the VPC Peering button.
Toggle Enable VPC Peering on. Atlas Stream Processing automatically selects the appropriate VPC peering connection from your configured connections.
If you do not have a VPC peering connection, Configure an Atlas Network Peering Connection.
Specify an IP address for one or more bootstrap servers for your Apache Kafka system.
From the dropdown menu, select a Security Protocol Method.
Atlas Stream Processing supports SASL_PLAINTEXT or SASL_SSL.
SASL_PLAINTEXT is incompatible with VPC peering. To use VPC peering, you must select the SASL_SSL method.
From the dropdown menu, select a SASL Mechanism.
Atlas Stream Processing supports:
PLAIN
SCRAM-SHA-256
SCRAM-SHA-512
Provide a Username for authentication.
Provide a password for authentication.
Click Add connection.
From the dropdown menu, select a SASL Mechanism.
Atlas Stream Processing supports:
PLAIN
SCRAM-SHA-256
SCRAM-SHA-512
Click Upload to upload your Certificate Authority PEM file
Provide a Username for authentication.
Provide a password for authentication.
Click Add connection.

The Atlas Administration API provides an endpoint for adding a connection to a connection registry.

Add a Connection to the Connection Registry

Important

After adding an external connection such as an Apache Kafka cluster to your connection registry, you must add Atlas IP addresses to an access list for that external connection. For more information, see Allow Access to or from the Atlas Control Plane.

Add an Atlas Connection

Important

VPC peering for Atlas Stream Processing is currently in private preview. To learn more, contact your account team.

To add an Atlas database connection to your stream processing instance:

To create one connection for the stream processing instance you specify using the Atlas CLI, run the following command:

atlas streams connections create [connectionName] [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections create.

Tip

See: Related Links

For an Atlas database connection, provide a configuration file with the following syntax:

{
  "name": "<name>",
  "type": "Cluster",
  "clusterName": "<clusterName>"
}

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.

Go to the Connection Registry.

Locate the overview panel of the stream processing instance you want to modify and click Configure.
Select the Connection Registry tab.

Click + Add connection.

Add a new connection.

To create a new connection to an Atlas change stream:

Select an Atlas Database connection.
Provide a Connection Name. Each connection name must be unique within an stream processing instance. This is the name used to reference the connection in Atlas Stream Processing aggregations.
From the dropdown menu, select an Atlas Cluster. Atlas Stream Processing is only available on dedicated-tier clusters.
Click Add connection.

The Atlas Administration API provides an endpoint for adding a connection to a connection registry.

Add a Connection to the Connection Registry

Add an HTTPS Connection

To add an HTTPS connection to your Stream Processing Instance:

Creating an HTTPS connection to an Stream Processing Instance using the Atlas CLI is not currently supported.

Creating an HTTPS connection to an Stream Processing Instance using the Atlas UI is not currently supported.

The Atlas Administration API provides an endpoint for adding a connection to a connection registry.

Create One Connection

If the API endpoint requires authentication, such as an API key or Bearer Access Token authentication, you should add authentication details as headers when you define the connection to prevent providing these as plaintext as part of the $https operator.

Other authentication schemes, such as Digest Auth or OAuth, are not currently supported.

To learn how to use HTTPS connections with Atlas Stream Processing, see $https.

Example:

curl --user "<publicApiKey>:<privateApiKey>" --digest \
  --header "Content-Type: application/json" \
  --header "Accept: application/vnd.atlas.2023-02-01+json" \
  --include \
  --data '{"name": "HTTPSConnection","type": "Https","url": "<apiBasePath>"}' \
  --request POST "https://cloud.mongodb.com/api/atlas/v2/groups/<projectID>/streams/<tenantName>/connections"

Modify a Connection from the Connection Registry

To modify a connection from a connection registry:

To modify the details of one connection on the stream processing instance you specify using the Atlas CLI, run the following command:

atlas streams connections update <connectionName> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections update.

Tip

See: Related Links

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.

Go to the Connection Registry.

Locate the overview panel of the stream processing instance that contains the connection registry that you want to modify and click Configure in the upper right.

Select the Connection Registry tab.

Modify the connection.

For the connection that you want to modify in the table, click the pencil icon in the Actions column.
You can edit a connection only if it is not currently in use by a stream processor. If you want to edit a connection in use, stop all stream processors that use the connection first.
Modify the fields you want to modify.
Click Save changes.

The Atlas Administration API provides an endpoint for editing a stream processing instance.

Delete One Stream Instance

Delete a Connection from the Connection Registry

To delete a connection from a connection registry:

To remove one connection from the stream processing instance you specify using the Atlas CLI, run the following command:

atlas streams connections delete <connectionName> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections delete.

Tip

See: Related Links

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.

Go to the Connection Registry.

Locate the overview panel of the stream processing instance that contains the connection registry that you want to modify
Click Configure in the upper right.
Select the Connection Registry tab.

Delete the connection.

For the connection you want to modify in the table, click the trash can icon in the Actions column of that entry.
When prompted, click Delete.

The Atlas Administration API provides an endpoint for editing a stream processing instance.

Delete One Stream Instance

List Private Link Connections

The Atlas Administration API provides an endpoint for viewing all Private Link connections configured for Atlas Stream Processing within a specified project.

Return All Private Links

View One Private Link Connection

The Atlas Administration API provides an endpoint for viewing the details of one Private Link connection configured for Atlas Stream Processing within a specified project.

Return One Private Link

Create A Private Link Connection

Atlas Stream Processing currently supports creating Private Link connections to Azure and Confluent clusters hosted on AWS.

To create a Private Link connection to use in your Atlas Stream Processing project:

Important

You can't have more than one Private Link connection to a given Confluent cluster per Atlas project. Before you begin this procedure, call the Return All Private Link Connections endpoint. If you have an existing Private Link connection to your Confluent cluster within Atlas but not within your Confluent account, only perform those steps that configure your Confluent-side networking.

Configure your Confluent cluster.

You must configure your Confluent cluster to accept incoming connections from your Atlas project.

Important

Confluent accepts incoming connections only from AWS. To use a Confluent Private Link connection, you must host your stream processing instances on AWS.

Call the Return Account ID and VPC ID for group and region Atlas Administration API endpoint. Note the value of awsAccountId, you will need this in a later step.
In your Confluent account, navigate to the cluster you want to connect to. In your cluster networking interface, navigate to your cluster networking details.
Add Private Link Access

For a Confluent dedicated cluster, provide a name of your choice. For the AWS account number, provide the value of the awsAccountId field you noted earlier.

Note

This step is not required for Confluent serverless clusters.

Request a connection to your cloud provider.

The Atlas Administration API provides an endpoint for requesting a Private Link connection configured for Atlas Stream Processing.

Create One Private Link

For an AWS Private Link connection, you must set the following key-value pairs:

Key	Value
`serviceEndpointId`	Your Confluent cluster's VPC Endpoint service name.
`dnsDomain`	Fully qualified domain name of the bootstrap server on your Confluent cluster.
`dnsSubDomain`	If your cluster doesn't use subdomains, you must set this to the empty array `[]`. If your cluster uses subdomains, you must set this to an array containing one fully qualified subdomain name for each of your cluster's subdomains.

You can find these values in your Confluent cluster's networking details.

The following example command requests a connection to your Confluent cluster and illustrates a typical response:

curl --location 'https://cloud.mongodb.com/api/atlas/v2/groups/8358217d3abb5c76c3434648/streams/privateLinkConnections' \
--digest \
--user "slrntglrbn:933fb118-ac62-4991-db05-ee67a3481fde" \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.atlas.2023-02-01+json' \
--data '{ "vendor": "Confluent", "provider": "AWS",
  "region": "us_east_1", "serviceEndpointId":
  "com.amazonaws.vpce.us-east-1.vpce-svc-93da685022ee702a9",
  "dnsDomain": "sample.us-east-1.aws.confluent.cloud",
  "dnsSubDomain: [
    "use1-az1.sample.us-east-1.aws.confluent.cloud",
    "use1-az2.sample.us-east-1.aws.confluent.cloud",
    "use1-az4.sample.us-east-1.aws.confluent.cloud"
  ]
}'

{"_id":"6aa12e7ccd660d4b2380b1c1","dnsDomain":"sample.us-east-1.aws.confluent.cloud.","vendor":"Confluent","provider":"AWS","region":"us_east_1","serviceEndpointId":"com.amazonaws.vpce.us-east-1.vpce-svc-93da685022ee702a9"}

After you send the request, note the value of the _id field in the response body. You will need this in a later step.

Provide the interface endpoint ID to Confluent.

Note

This step applies only to Confluent serverless clusters.

Call the Return All Private Link Connections endpoint. Note the value of interfaceEndpointId.

In your Confluent account, navigate to the cluster you want to connect to. In your cluster networking interface, navigate to your cluster networking details. Navigate to the access points interface, and add a new access point. When Confluent prompts you for an interface endpoint, provide the value of interfaceEndpointId that you noted previously.

Create the Atlas-side connection.

Add a connection with the following key-value pairs:

Key	Value
`bootstrapServers`	IP address of your cloud provider's Kafka bootstrap server.
`security.protocol`	`SASL_SSL`
`authentication.mechanism`	`"PLAIN"`
`authentication.password`	The password associated with your Confluent API key
`authentication.username`	The username associated with your Confluent API key
`type`	`"Kafka"`
`networking.access.type`	`"PRIVATE_LINK"`
`networking.access.connectionId`	`_id` value from your Private Link request response

Set all other values as necessary.

The following example command creates a Apache Kafka connection in Atlas:

curl --location 'https://cloud.mongodb.com/api/atlas/v2/groups/8358217d3abb5c76c3434648/streams/spinstance/connections' \
--digest \
--user "slrntglrbn:933fb118-ac62-4991-db05-ee67a3481fde" \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.atlas.2023-02-01+json' \
--data '{
  "name": "confluent_demo",
  "bootstrapServers": "slr-ntgrbn.sample.us-east-1.aws.confluent.cloud:9092",
  "security": {
    "protocol": "SASL_SSL"
  },
  "authentication": {
    "mechanism": "PLAIN",
    "password": "apiSecretDemo",
    "username": "apiUserDemo"
  },
  "type": "Kafka",
  "networking": {
    "access": {
    "type": "PRIVATE_LINK",
    "connectionId": "38972b0cbe9c2aa40a30a246"
    }
  }
}'

Request a connection to your cloud provider.

The Atlas Administration API provides an endpoint for requesting a Private Link connection configured for Atlas Stream Processing.

Create One Private Link

For an Azure Private Link connection, you must set the following key-value pairs:

Key	Value
`serviceEndpointId`	Your EventHub namespace endpoint
`dnsDomain`	Fully qualified domain name, with port number, of the bootstrap server in your Azure Event Hub namespace. This domain name conforms to the format described here.

The following example command requests a connection to your Azure Event Hub and illustrates a typical response:

curl --location 'https://cloud.mongodb.com/api/atlas/v2/groups/8358217d3abb5c76c3434648/streams/privateLinkConnections' \
--digest \
--user "slrntglrbn:933fb118-ac62-4991-db05-ee67a3481fde" \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.atlas.2023-02-01+json' \
--data '{ "provider": "AZURE", "region": "US_EAST_2", "serviceEndpointId": "/subscriptions/b82d6aa0-0b0a-ffa3-7c22-e167dc44f5b0/resourceGroups/asp/providers/Microsoft.EventHub/namespaces/sample", "dnsDomain": "sample.servicebus.windows.net" }'

{"_id":"6aa12e7ccd660d4b2380b1c1","dnsDomain":"sample.servicebus.windows.net","provider":"AZURE","region":"US_EAST_2","serviceEndpointId":"/subscriptions/b82d6aa0-0b0a-ffa3-7c22-e167dc44f5b0/resourceGroups/asp/providers/Microsoft.EventHub/namespaces/sample"}

After you send the request, note the value of the _id field in the response body. You will need this in a later step.

Accept the requested connection within your cloud provider account.

For Private Link connections to Azure, navigate to your Event Hub networking page and select the Private endpoint connections tab. In the table of connections, identify your newly requested connection and approve it.

Create the Atlas-side connection.

Add a connection with the following key-value pairs:

Key	Value
`bootstrapServers`	IP address of your cloud provider's Kafka bootstrap server.
`security.protocol`	`SASL_SSL`
`authentication.mechanism`	`"PLAIN"`
`authentication.password`	Your Event Hub connection string
`authentication.username`	`"$ConnectionString"`
`type`	`"Kafka"`
`networking.access.type`	`"PRIVATE_LINK"`
`networking.access.connectionId`	`_id` value from your Private Link request response

Set all other values as necessary.

The following example command creates a Apache Kafka connection in Atlas:

curl --location 'https://cloud.mongodb.com/api/atlas/v2/groups/8358217d3abb5c76c3434648/streams/spinstance/connections' \
--digest \
--user "slrntglrbn:933fb118-ac62-4991-db05-ee67a3481fde" \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.atlas.2023-02-01+json' \
--data '{ "name": "eventhubpl33333", "bootstrapServers": "sample.servicebus.windows.net:9093", "security": { "protocol": "SASL_SSL" }, "authentication": { "mechanism": "PLAIN", "password": "Endpoint=sb://sample.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=Irlo3OoRkc27T3ZoGOlbhEOqXQRXzb12+Q2hNXm0lc=", "username": "$ConnectionString" }, "type": "Kafka", "networking": { "access": { "type": "PRIVATE_LINK", "connectionId": "38972b0cbe9c2aa40a30a246" } } }'

Delete a Private Link Connection

The Atlas Administration API provides an endpoint for deleting a Private Link connection configured for Atlas Stream Processing within a specified project.

Delete One Private Link

Back

Manage Stream Processing Instances

Manage Stream Processors

Prerequisites

Note

View Connections in the Connection Registry

Tip

See: Related Links

In Atlas, go to the .css-h15tq0{font-style:normal;font-weight:700;}Stream Processing page for your project.

Go to the Connection Registry page.

Add a Connection to the Connection Registry

Add an Apache Kafka Connection

Important

Tip

See: Related Links

In Atlas, go to the Stream Processing page for your project.

Go to the Connection Registry.

Click + Add connection.

Add a new connection.

Important

Add an Atlas Connection

Important

Tip

See: Related Links

In Atlas, go to the Stream Processing page for your project.

Go to the Connection Registry.

Click + Add connection.

Add a new connection.

Add an HTTPS Connection

Modify a Connection from the Connection Registry

Tip

See: Related Links

In Atlas, go to the Stream Processing page for your project.

Go to the Connection Registry.

Select the Connection Registry tab.

Modify the connection.

Delete a Connection from the Connection Registry

Tip

See: Related Links

In Atlas, go to the Stream Processing page for your project.

Go to the Connection Registry.

Delete the connection.

List Private Link Connections

View One Private Link Connection

Create A Private Link Connection

Important

Configure your Confluent cluster.

Important

Note

Request a connection to your cloud provider.

Provide the interface endpoint ID to Confluent.

Note

Create the Atlas-side connection.

Request a connection to your cloud provider.

Create the Atlas-side connection.

Delete a Private Link Connection

In Atlas, go to the Stream Processing page for your project.