Multi-Cluster Sharded Cluster Without Service Mesh

Example Details and Considerations

The above example is not production ready. Please make note of the following details and considerations:

• The domain in the example (kind-e2e-cluster-3.interconnected) is artificial and should be replaced with the proper externally accessible domain.

• The configuration uses TLS (it's not possible to securely expose externally MongoDB processes without TLS) and x509 authentication. This can be configured arbitrarily according to the needs. See the appropriate Encryption and Authentication to learn more.

• The TLS certificates in sharded cluster must be issued for the following components:

  • Each shard replica set

  • Config Server replica set

  • Mongos

• Each TLS certificate (-cert secrets of TLS type, provided manually or issued by cert-manager) must be provided in the Kubernetes cluster where the Kubernetes Operator is running. From there, the Kubernetes Operator automatically replicates necessary resources (generated -cert-pem secrets) to the other Kubernetes clusters.

• Each TLS certificate for each component must contain all the hostnames of all processes of that component's replica set. For example, the certificate issued for the first shard (index 0) must have in the cert's SANs field the following hostnames (for each cluster it's deployed on; the pod name follows the convention mdb-sharded-<shardIdx>-<clusterIdx>-<podIdxInThisCluster>).

  • mdb-sharded-0-0-0.kind-e2e-cluster-1.interconnected

  • mdb-sharded-0-1-0.kind-e2e-cluster-2.interconnected

  • mdb-sharded-0-2-0.kind-e2e-cluster-3.interconnected

• For reference, all the processes of the sharded cluster deployed with the provided example are configured with the following hostnames:

  mdb-sharded-0-0-0.kind-e2e-cluster-1.interconnected (shard-0)
  mdb-sharded-0-1-0.kind-e2e-cluster-2.interconnected (shard-0)
  mdb-sharded-0-2-0.kind-e2e-cluster-3.interconnected (shard-0)
  mdb-sharded-config-0-0.kind-e2e-cluster-1.interconnected (cs)
  mdb-sharded-config-1-0.kind-e2e-cluster-2.interconnected (cs)
  mdb-sharded-config-2-0.kind-e2e-cluster-3.interconnected (cs)
  mdb-sharded-mongos-0-0.kind-e2e-cluster-1.interconnected (mongos)
  mdb-sharded-mongos-1-0.kind-e2e-cluster-2.interconnected (mongos)
  mdb-sharded-mongos-2-0.kind-e2e-cluster-3.interconnected (mongos)

• Each externalService: is defined with a default object {}.

  This means that the Kubernetes Operator creates the External Services (services created for each pod which should be used to route the external traffic to) with the default values:

  • type: LoadBalancer

  • ports: 27017, 27018 for backup purposes

• Depending on the cluster configuration and load balancer controller in the Kubernetes cluster, using LoadBalancer type for external services will cause provisioning of load balancer resources for each of the processes in the sharded clusters. This might lead to the large number of LoadBalancer resources allocated. In the minimal deployment (1 mongos, 3-node config server replica set, 1 shard of 3-node replica set) this creates 7 LoadBalancer services.

  In order to minimize the number of LoadBalancer resources that are created, we recommend that you configure external access with ClusterIP services, as shown in the following example:

  - clusterName: kind-e2e-cluster-3
    externalAccess:
      externalDomain: kind-e2e-cluster-3.interconnected
      externalService:
        type: ClusterIP

• With each process (pod) having one associated ClusterIP external service, you need to create one external "entry point" proxy component to the Kubernetes cluster, which might be any proxy (e.g. HAProxy, Nginx) that supports TLS Passthrough routing.

  • The proxy component must be deployed in each Kubernetes cluster and exposed externally on a hostname defined for the components in each cluster:

    • In the example above, each component (mongos, config server, shards) defined for kind-e2e-cluster-3 use externalDomain: externalDomain: kind-e2e-cluster-3.interconnected

      This means, the proxy component should be configured to receive all connections to *.kind-e2e-cluster-3.interconnected.

      Each connection is be a hostname following the naming convention <pod-name>.<externalDomain>, for example:

      Shard pod: mdb-sharded-0-2-0.kind-e2e-cluster-2.interconnected (mdb-sharded-<shardIdx>-<clusterIdx>-<podIdx>

  • The proxy component should be configured such that any traffic (on ports 27017 or 27018) is proxied (on a TCP level, without re-encrypting the traffic) to the respective external service.

    • For example, for mdb-sharded-0-2-0-svc-external a generic rule might be used when configuring the dispatching as <pod-name>.kind-e2e-cluster-2.interconnected forwarded to <pod-name>-mdb-sharded-0-2-0-svc-external.

• Note that the externalDomain used in the example is an artificial one. It must be replaced with a proper domain name.

• The operator deploys all processes across the Kubernetes clusters along with all the necessary external services.

• Each external service in each cluster must receive external traffic on the defined external domains. Until the networking is fully configured, the cluster is not configured properly, because for the the replicaset to be configured, the mongod processes need to be able to connect with each other for replication purposes.

• When using external domains, all the networking (for replication and MongoDB Agent to MongoDB connectivity) is by default routed via external domain, which might not be efficient. One of the ways to improve it is to configure networking or the DNS resolution inside the cluster so that the external hostnames of the pods deployed in the same Kubernetes clusters are resolved to the cluster local IP address of the external services.