Multi-Cluster Architecture Diagram: Ops Manager and the Application Database
The following diagram shows the Ops Manager Application, the Application Database, the Backup Daemon, and the corresponding Persistent Volumes deployed on multiple Kubernetes clusters.
In this diagram:
The
Member Cluster 0
is also an "operator cluster" because you install the Kubernetes Operator on it. It is also a "member cluster" and can host any multi-cluster custom resource.The
Member Cluster 0
stores the kubeconfig files, which describe the Kubernetes configuration for member clusters, users, and contexts. When you configure the Kubernetes Operator for multi-cluster deployments using thekubectl mongodb
plugin, it creates the following resource:The
mongodb-enterprise-operator-multi-cluster-kubeconfig
secret, which contains the credentials to all Kubernetes clusters that the Kubernetes Operator is going to manage. If you plan to use the operator cluster as a member cluster, this secret might contain the credentials to the same cluster where you install the Kubernetes Operator.
When the Kubernetes Operator runs in the multi-cluster mode, it stores the resources it needs, such as ConfigMaps and secrets about the clusters that it is going to manage. These resources belong to the same namespace as the Kubernetes Operator. The Kubernetes Operator uses these resources to deploy the Ops Manager Application and the Application Database on multiple Kubernetes clusters.
The Kubernetes Operator also creates and maintains some additional multi-cluster deployment state ConfigMaps for each Ops Manager Application and Application Database deployment it manages. The
Member Cluster 0
stores this configuration, which includes the following ConfigMaps:The
<om_resource_name>-cluster-mapping
ConfigMap contains the mapping of member cluster names listed in thespec.clusterSpecList
to cluster indexes, referenced in this documentation ascluster_index
, such asCluster 0
, orCluster 1
. The Kubernetes Operator assigns these indexes to each cluster name.The
<om_resource_name>-db-cluster-mapping
ConfigMap contains the mapping of member cluster names listed in thespec.applicationDatabase.clusterSpecList
to cluster indexes.The
<om_resource_name>-db-member-spec
ConfigMap contains the number of Application Database replicas configured for each member cluster. Having this information allows the Kubernetes Operator to correctly scale or reconfigure the replica set as part of disaster recovery, such as after losing the entire member cluster.
The
MongoDBOpsManager
resource's configuration is a file that you create that describes a multi-cluster Ops Manager deployment. The Kubernetes Operator uses this file to deploy the Ops Manager components.The following example shows the configuration that leads to the Kubernetes Operator deploying the Ops Manager components described in this diagram. This example omits some settings that aren't relevant for this diagram, such as TLS configuration.
1 apiVersion: mongodb.com/v1 2 kind: MongoDBOpsManager 3 metadata: 4 name: om 5 namespace: om-ns 6 spec: 7 replicas: 1 # You can set this value and use it as a global or default 8 # setting for all clusters. The spec.clusterSpecList.members 9 # setting overrides this setting. 10 topology: MultiCluster 11 version: 6.0.22 12 adminCredentials: om-admin-secret 13 clusterSpecList: 14 - clusterName: "Member Cluster 1" # Ops Manager settings for "Member Cluster 1" 15 members: 2 16 backup: # Backup settings for "Member Cluster 1" 17 members: 2 # Overrides spec.backup.members 18 - clusterName: "Member Cluster 2" # Ops Manager settings for "Member Cluster 2" 19 members: 1 20 backup: # Backup settings for "Member Cluster 2" 21 members: 2 # Overrides spec.backup.members 22 applicationDatabase: # Global {+appdb+} settings 23 topology: MultiCluster 24 version: 6.0.5-ent 25 members: 3 # In multi-cluster mode, the Operator ignores this field. 26 # The Operator sets the number of members for the Application 27 # Database in spec.applicationDatabase.clusterSpecList.members. 28 clusterSpecList: 29 - clusterName: "Member Cluster 1" 30 members: 3 31 - clusterName: "Member Cluster 2" 32 members: 2 33 backup: # Global settings for the Backup Daemon 34 enabled: true 35 members: 1 # Set this value and use it as a global or default setting. 36 # To override this value, set the value for 37 # spec.clusterSpecList.backup.members. 38 # The Backup Daemon's configuration for each cluster isn't 39 # stored here. Use the Ops Manager's spec.clusterSpecList.backup to 40 # specify the Backup Daemon configuration for each member cluster. The Kubernetes Operator connects to the Ops Manager instances referencing either:
The default FQDN of the service it creates for the Ops Manager resource,
<om_resource_name>-svc.<namespace>.svc.cluster.local
, orThe URL that you specify in
spec.opsManagerURL
. In some deployments, such as when the cluster where you installed the Kubernetes Operator isn't attached to the service mesh, the default service FQDN might be unreachable. In this case, the Kubernetes Operator reports theMongoDBOpsManager
resource status asFailed
indicating a connection error. To account for such cases, provide the URL to Ops Manager in thespec.opsManagerURL
. This URL might be a hostname of an externally exposed Ops Manager instance. To learn more, see Networking Overview.
Two member clusters host the Ops Manager Application. In each cluster, the Kubernetes Operator deploys a StatefulSet named
<om_resource_name>-<cluster_index>
.The StatefulSet deploys two instances of the Ops Manager Application in
Member Cluster 1
and one instance inMember Cluster 2
.You define the number of instances in
spec.clusterSpecList.members
. You might set the number of instances to zero so that this cluster doesn't deploy any Ops Manager Application instances. This is useful, if, for example, you would like to use this cluster for hosting only Backup Daemon instances.If you remove a cluster from
spec.clusterSpecList
, this is equivalent to specifying zero members inspec.clusterSpecList.members
andspec.clusterSpecList[*].backup.members
.For each StatefulSet in each cluster, the Kubernetes Operator configures a service of type
ClusterIP
, named<om_resource_name>-svc
, that contains all Pods on the cluster's endpoints list. This service's FQDN,<om_resource_name>-svc.<namespace>.svc.cluster.local
, is a default hostname that the Kubernetes Operator uses to access the deployed endpoint for the Ops Manager Application.If you specify
spec.externalConnectivity
, the Kubernetes Operator also creates an external KubernetesLoadBalancer
-type service, named<om_resource_name>-svc-ext
, for each cluster. In each cluster, you can specify its own configuration for this external service usingspec.clusterSpecList.externalConnectivity
. For example, you can change the service's type or define annotations.
Application Database. The Kubernetes Operator deploys the Application Database on two clusters.
The
Member Cluster 1
contains threemongod
processes for the Application Database, and theMember Cluster 2
contains twomongod
processes..You define the Application Database configuration using the
spec.applicationDatabase
settings. On each member cluster, the Kubernetes Operator creates a StatefulSet named<om_resource_name>-db-<cluster_index>
with the number of member clusters defined inspec.applicationDatabase.clusterSpecList.members
. In multi-cluster mode, the Kubernetes Operator ignores values that you set for thespec.applicationDatabase.members
field. The Kubernetes Operator configures one replica set formed frommongod
processes deployed across all member clusters.For each Pod in
<statefulset_name>-<pod_index>
hosting a MongoDB process named<om_resource_name>-db-<cluster_index>-<pod_index>
, the Kubernetes Operator creates a KubernetesClusterIP
-type service for accessing the individualmongod
processes by its FQDN,<om_resource_name>-db-<cluster_index>-<pod_index>-svc
. Eachmongod
process in the replica set must be uniquely addressable.The processes in the replica set configuration must have their process hostnames configured to that Pod service's FQDN:
<om_resource_name>-db-<cluster_index>-<pod_index>-svc.<namespace>.svc.cluster.local
.Each Pod has its persistent volume attached via a Persistent Volume Claim that the Kubernetes Operator creates.
To form a replica set from all
mongod
processes, each process must connect to each other process for replication purposes. To achieve this, include all member clusters on which you deploy the Application Database into the same service mesh configuration.The service mesh handles cross-cluster DNS queries and routes the traffic accordingly. The service mesh assists in resolving each Pod service's FQDN
<om_resource_name>-db-<cluster_index>-<pod-index>-svc.<namespace>.svc.cluster.local
across all clusters and allows connectivity on the exposedmongod
port (27017 by default).For example, when a
mongod
process running in theom-db-1-0
Pod inMember Cluster 1
connects to amongod
running in theom-db-2-1
Pod inMember Cluster 2
, the firstmongod
process uses its hostname from the Automation Configuration,om-db-2-1-svc.om-ns.svc.cluster.local:27017
, and the service mesh routes this request toMember Cluster 2
to theom-db-2-1-svc
service. Without the service mesh, the KubernetesMember Cluster 1
has no information about theom-db-2-1-svc
service deployed in theMember Cluster 2
and the DNS resolution ofom-db-2-1-svc.om-ns.svc.cluster.local
would fail.When the Application Database and the Ops Manager Application instances are in a
Running
state, the Kubernetes Operator adds an additional monitoring container to the Application Database StatefulSets. This results in a rolling restart of all Application Database Pods in all clusters. The Kubernetes Operator updates the StatefulSets across all clusters sequentially, so that during the rolling restart process, in each cluster, only one replica set's member becomes temporarily unavailable.The Monitoring Agent connects to the Ops Manager Application instances using the Ops Manager service's FQDN,
<om_resource_name>-svc.<namespace>.svc.cluster.local
, or the value inspec.opsManagerURL
if you specify it.The Ops Manager Application and the Backup Daemon always use the connection string to the Application Database that contains all replica set members in it. The connection string is always constructed using the service-per-pod FQDNs.
The Kubernetes Operator deploys the Backup Daemon StatefulSets if you set the
spec.backup.enabled
totrue
.On each member cluster listed in the
spec.clusterSpecList
, the Kubernetes Operator creates one Backup Daemon StatefulSet, named<om_resource_name>-backup-daemon-<cluster_index>
with the number of Backup Daemon instances set tospec.backup.members
.Alternatively, you can configure the number of Backup Daemon instances for each cluster in
spec.clusterSpecList[*].backup.members
.The Backup Daemon instances connect only to the Application Database replica set using the same connection string as the Ops Manager Application instances.
In addition, in this diagram, you can observe the service mesh and the networking connections between the components:
The dotted lines surrounding the diagram show the single service mesh that includes the networking configuration for all clusters.
The dotted lines surrounding the Ops Manager Application across member clusters indicate that these instances are stateless and the traffic can be equally distributed to all instances, for example by using using a round-robin load balancer.
The dotted lines surrounding the Application Database across member clusters indicate that these instances communicate with each other and form a single MongoDB replica set.