Manage Atlas Stream Processing
You can use Atlas Kubernetes Operator to manage stream processing instances and connections for Atlas Stream Processing. Atlas Stream Processing enables you to process streams of complex data using the same MongoDB Query API that Atlas databases use. Atlas Stream Processing allows you to do the following tasks:
Build aggregation pipelines to continuously operate on streaming data without the delays inherent in batch processing.
Perform continuous schema validation to check that messages are properly formed, detect message corruption, and detect late-arriving data.
Continuously publish results to Atlas collections or Apache Kafka clusters, ensuring up-to-date views and analysis of data.
Atlas Stream Processing components belong directly to Atlas projects and operate independent of Atlas clusters. To learn more, see Atlas Stream Processing Overview.
Stream Processing Instances and Connections
Atlas Stream Processing instances provide the context for all of your operations on streaming data. You can configure a sample connection, a connection to an Atlas change stream, or a connection to an Apache Kafka system. Then you can add the connection to the Connection Registry for your stream processing instance. To learn more, see Manage Stream Processing Instances.
Prerequisites
To use Atlas Kubernetes Operator to manage stream processing instances and connections, you must:
Have an Atlas user with the
Project Owneror theProject Stream Processing Ownerrole to manage a Connection Registry and Stream Processing Instance. To learn more about the differences between the two roles, see Project Roles.Have a running Kubernetes cluster with Atlas Kubernetes Operator deployed.
Ensure your IP address is in the organization's API access list.
Procedure
To use Atlas Kubernetes Operator to manage a stream processing instance and its connections, do the following steps:
Configure the AtlasStreamInstance Custom Resource.
Example:
cat <<EOF | kubectl apply -f - apiVersion: atlas.mongodb.com/v1 kind: AtlasStreamInstance metadata: name: my-stream-instance spec: name: my-stream-instance clusterConfig: provider: AWS region: VIRGINIA_USA tier: SP30 projectRef: name: my-project EOF
To learn more about the available parameters, see the
AtlasStreamInstance Custom Resource.
Note
Currently, Atlas Kubernetes Operator supports only the AWS provider and the
VIRGINIA_USA region for this custom resource.
Configure the AtlasStreamConnection Custom Resource.
You can configure a sample connection, a connection to an Atlas change stream, or a connection to an Apache Kafka system.
Examples:
apiVersion: atlas.mongodb.com/v1 kind: AtlasStreamConnection metadata: name: my-stream-connection spec: name: sample_stream_solar type: Sample
Note
If you specify Sample for the spec.type
parameter, the spec.name parameter must match the
sample collection name. Currently, Atlas Kubernetes Operator supports only
the sample_stream_solar sample collection for this
custom resource.
apiVersion: atlas.mongodb.com/v1 kind: AtlasStreamConnection metadata: name: my-stream-connection spec: name: my-stream-connection type: Cluster clusterConfig: name: my-cluster role: name: my-db-role type: CUSTOM
apiVersion: atlas.mongodb.com/v1 kind: AtlasStreamConnection metadata: name: my-stream-connection spec: name: my-stream-connection type: Kafka kafkaConfig: bootstrapServers: "comma,separated,list,of,server,addresses" authentication: mechanism: SCRAM-512 credentials: name: ref-to-creds-secret namespace: default security: protocol: SSL certificate: name: ref-to-certificate-secret namespace: default
To learn more about the available parameters, see the
AtlasStreamConnection Custom Resource.
Add your connections to the Connection Registry in the AtlasStreamInstance Custom Resource.
Example:
cat <<EOF | kubectl apply -f - apiVersion: atlas.mongodb.com/v1 kind: AtlasStreamInstance metadata: name: my-stream-instance spec: name: my-stream-instance clusterConfig: provider: AWS region: VIRGINIA_USA tier: SP30 projectRef: name: my-project connectionRegistry: - name: ref-my-connection-1 namespace: my-namespace1 - name: ref-my-connection-2 namespace: my-namespace2 - name: ref-my-connection-3 namespace: my-namespace1 EOF
Note
Currently, Atlas Kubernetes Operator supports only the AWS provider and the
VIRGINIA_USA region for this custom resource.
To learn more about the available parameters, see the
AtlasStreamInstance Custom Resource.