MongoDB Ops Manager在多个Kubernetes 集群上部署 资源

创建 Kubernetes 集群。

如果您已经使用服务网格安装并配置了自己的 Kubernetes 集群，则可以跳过此步骤。

1. 创建三个 GKE (Google Kubernetes Engine) 集群：

   1
   gcloud container clusters create "${K8S_CLUSTER_0}" \
   2
         --zone="${K8S_CLUSTER_0_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_0_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_0_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

   1
   gcloud container clusters create "${K8S_CLUSTER_1}" \
   2
         --zone="${K8S_CLUSTER_1_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_1_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_1_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

   1
   gcloud container clusters create "${K8S_CLUSTER_2}" \
   2
         --zone="${K8S_CLUSTER_2_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_2_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_2_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

2. 获取凭证并将上下文保存到当前的kubeconfig文件。 默认情况下，此文件位于~/.kube/config目录中，并由$KUBECONFIG环境变量引用。

   1
   gcloud container clusters get-credentials "${K8S_CLUSTER_0}" --zone="${K8S_CLUSTER_0_ZONE}"
   2
   gcloud container clusters get-credentials "${K8S_CLUSTER_1}" --zone="${K8S_CLUSTER_1_ZONE}"
   3
   gcloud container clusters get-credentials "${K8S_CLUSTER_2}" --zone="${K8S_CLUSTER_2_ZONE}"

   所有kubectl命令都使用以下变量引用这些上下文：

   • $K8S_CLUSTER_0_CONTEXT_NAME

   • $K8S_CLUSTER_1_CONTEXT_NAME

   • $K8S_CLUSTER_2_CONTEXT_NAME

3. 验证kubectl是否有权访问 Kubernetes 集群：

   1
   echo "Nodes in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" get nodes
   3
   echo; echo "Nodes in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   4
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" get nodes
   5
   echo; echo "Nodes in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"
   6
   kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" get nodes

   查看输出

   1
   Nodes in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   2
   NAME                                       STATUS   ROLES    AGE   VERSION
   3
   gke-k8s-mdb-0-default-pool-d0f98a43-dtlc   Ready    <none>   65s   v1.28.7-gke.1026000
   4
   gke-k8s-mdb-0-default-pool-d0f98a43-q9sf   Ready    <none>   65s   v1.28.7-gke.1026000
   5
   gke-k8s-mdb-0-default-pool-d0f98a43-zn8x   Ready    <none>   64s   v1.28.7-gke.1026000
   6
   7
   Nodes in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   8
   NAME                                       STATUS   ROLES    AGE    VERSION
   9
   gke-k8s-mdb-1-default-pool-37ea602a-0qgw   Ready    <none>   111s   v1.28.7-gke.1026000
   10
   gke-k8s-mdb-1-default-pool-37ea602a-k4qk   Ready    <none>   114s   v1.28.7-gke.1026000
   11
   gke-k8s-mdb-1-default-pool-37ea602a-p2g7   Ready    <none>   113s   v1.28.7-gke.1026000
   12
   13
   Nodes in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
   14
   NAME                                       STATUS   ROLES    AGE   VERSION
   15
   gke-k8s-mdb-2-default-pool-4b459a09-t1v9   Ready    <none>   29s   v1.28.7-gke.1026000

4. 安装 Istio 服务网格以允许跨集群 DNS 解析和 Kubernetes 集群之间的网络连接：

   1
   CTX_CLUSTER1=${K8S_CLUSTER_0_CONTEXT_NAME} \
   2
   CTX_CLUSTER2=${K8S_CLUSTER_1_CONTEXT_NAME} \
   3
   CTX_CLUSTER3=${K8S_CLUSTER_2_CONTEXT_NAME} \
   4
   ISTIO_VERSION="1.20.2" \
   5
   ../multi-cluster/install_istio_separate_network.sh

创建命名空间。

• 为 Kubernetes Operator 部署创建一个单独的命名空间mongodb-operator ，由$OPERATOR_NAMESPACE环境变量引用。

• 在每个成员 Kubernetes 集群上创建相同的$OPERATOR_NAMESPACE 。 这是必要的，以便kubectl MongoDB插件可以在每个成员集群上为Kubernetes Operator 创建服务帐户。 Kubernetes Operator 使用 Operator 集群上的这些服务帐户对每个成员集群执行操作。

  1
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  2
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
  3
  4
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  5
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
  6
  7
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  8
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite

• 在每个成员集群（包括用作操作符集群的成员集群）上，创建另一个独立的命名空间mongodb 。 Kubernetes Operator 将此命名空间用于MongoDB Ops Manager资源和组件。

  1
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  2
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
  3
  4
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  5
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
  6
  7
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  8
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite

可选。 授权集群从私有映像注册表中提取密钥。

如果您使用官方 HelmAtlas Charts 和来自 Quay 的图像，则此步骤是可选的 注册表。

1
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" create secret generic "image-registries-secret" \
2
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
3
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
4
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
5
kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
6
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
7
kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
8
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson

可选。 检查集群连接。

以下可选脚本验证是否为跨集群 DNS 解析和连接正确配置了服务网格。

1. 在集群0上运行此脚本：

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver0
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver0
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver0
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver0
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

2. 在集群1上运行此脚本：

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver1
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver1
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver1
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver1
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

3. 在集群2上运行此脚本：

   1
   kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver2
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver2
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver2
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver2
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

4. 运行此脚本以等待创建 StatefulSet：

   1
   kubectl wait --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver0-0 --timeout=60s
   2
   kubectl wait --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver1-0 --timeout=60s
   3
   kubectl wait --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver2-0 --timeout=60s

5. 在集群0上创建 Pod 服务：

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver0-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver0-0"
   13
   EOF

6. 在集群1上创建 Pod 服务：

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver1-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver1-0"
   13
   EOF

7. 在集群2上创建 Pod 服务：

   1
   kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver2-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver2-0"
   13
   EOF

8. 在集群0上创建循环服务：

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       app: echoserver0
   13
   EOF

9. 在集群1上创建循环服务：

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       app: echoserver1
   13
   EOF

10. 在集群2上创建循环服务：

    1
    kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2
    apiVersion: v1
    3
    kind: Service
    4
    metadata:
    5
      name: echoserver
    6
    spec:
    7
      ports:
    8
        - port: 8080
    9
          targetPort: 8080
    10
          protocol: TCP
    11
      selector:
    12
        app: echoserver2
    13
    EOF

11. 验证集群0 中的 Pod1 ：

    1
    source_cluster=${K8S_CLUSTER_1_CONTEXT_NAME}
    2
    target_pod="echoserver0-0"
    3
    source_pod="echoserver1-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    查看输出

    1
    Checking cross-cluster DNS resolution and connectivity from echoserver1-0 in gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1 to echoserver0-0
    2
    SUCCESS

12. 验证集群1 中的 Pod0 ：

    1
    source_cluster=${K8S_CLUSTER_0_CONTEXT_NAME}
    2
    target_pod="echoserver1-0"
    3
    source_pod="echoserver0-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    查看输出

    1
    Checking cross-cluster DNS resolution and connectivity from echoserver0-0 in gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0 to echoserver1-0
    2
    SUCCESS

13. 验证集群1 中的 Pod2 ：

    1
    source_cluster=${K8S_CLUSTER_2_CONTEXT_NAME}
    2
    target_pod="echoserver1-0"
    3
    source_pod="echoserver2-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    查看输出

    1
    Checking cross-cluster DNS resolution and connectivity from echoserver2-0 in gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2 to echoserver1-0
    2
    SUCCESS

15. 验证集群2 中的 Pod0 ：

    1
    source_cluster=${K8S_CLUSTER_0_CONTEXT_NAME}
    2
    target_pod="echoserver2-0"
    3
    source_pod="echoserver0-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    查看输出

    1
    Checking cross-cluster DNS resolution and connectivity from echoserver0-0 in gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0 to echoserver2-0
    2
    SUCCESS

16. 运行清理脚本：

    1
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver0
    2
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver1
    3
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver2
    4
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    5
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    6
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    7
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver0-0
    8
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver1-0
    9
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver2-0

部署多集群配置。

在此步骤中，您将使用kubectl mongodb插件自动执行 Kubernetes 集群配置，这是 Kubernetes Operator 管理多个 Kubernetes 集群上的工作负载所必需的。

由于您在安装 Kubernetes Operator 之前配置 Kubernetes 集群，因此当您为多 Kubernetes 集群操作部署 Kubernetes Operator 时，所有必要的多集群配置都已就位。

如概述中所述， Kubernetes Operator 具有三个成员集群的配置，可用于部署MongoDB Ops Manager MongoDB数据库。 第一个集群还用作 Operator 集群，您可以在其中安装 Kubernetes Operator 并部署自定义资源。

1
kubectl mongodb multicluster setup \
2
  --central-cluster="${K8S_CLUSTER_0_CONTEXT_NAME}" \
3
  --member-clusters="${K8S_CLUSTER_0_CONTEXT_NAME},${K8S_CLUSTER_1_CONTEXT_NAME},${K8S_CLUSTER_2_CONTEXT_NAME}" \
4
  --member-cluster-namespace="${NAMESPACE}" \
5
  --central-cluster-namespace="${OPERATOR_NAMESPACE}" \
6
  --create-service-account-secrets \
7
  --install-database-roles=true \
8
  --image-pull-secrets=image-registries-secret

1
Build: 1f23ae48c41d208f14c860356e483ba386a3aab8, 2024-04-26T12:19:36Z
2
Ensured namespaces exist in all clusters.
3
creating central cluster roles in cluster: gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
4
creating member roles in cluster: gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
5
creating member roles in cluster: gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
6
Ensured ServiceAccounts and Roles.
7
Creating KubeConfig secret mongodb-operator/mongodb-enterprise-operator-multi-cluster-kubeconfig in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
8
Ensured database Roles in member clusters.
9
Creating Member list Configmap mongodb-operator/mongodb-enterprise-operator-member-list in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0

使用 Helm Chart 安装 Kubernetes Operator。

1. 将 Kubernetes Operator 安装到$OPERATOR_NAMESPACE中，并配置为监视$NAMESPACE并管理三个成员 Kubernetes 集群。 此时， ServiceAccounts 和 角色 已由kubectl mongodb 插件部署。因此，以下脚本会跳过对它们的配置并设置operator.createOperatorServiceAccount=false和operator.createResourcesServiceAccountsAndRoles=false 。 这些脚本指定multiCluster.clusters设置来指示 Helm Chart 在多集群模式下部署 Kubernetes Operator。

   1
   helm upgrade --install \
   2
     --debug \
   3
     --kube-context "${K8S_CLUSTER_0_CONTEXT_NAME}" \
   4
     mongodb-enterprise-operator-multi-cluster \
   5
     "${OPERATOR_HELM_CHART}" \
   6
     --namespace="${OPERATOR_NAMESPACE}" \
   7
     --set namespace="${OPERATOR_NAMESPACE}" \
   8
     --set operator.namespace="${OPERATOR_NAMESPACE}" \
   9
     --set operator.watchNamespace="${NAMESPACE}" \
   10
     --set operator.name=mongodb-enterprise-operator-multi-cluster \
   11
     --set operator.createOperatorServiceAccount=false \
   12
     --set operator.createResourcesServiceAccountsAndRoles=false \
   13
     --set "multiCluster.clusters={${K8S_CLUSTER_0_CONTEXT_NAME},${K8S_CLUSTER_1_CONTEXT_NAME},${K8S_CLUSTER_2_CONTEXT_NAME}}" \
   14
     --set "${OPERATOR_ADDITIONAL_HELM_VALUES:-"dummy=value"}"

   查看输出

   1
   Release "mongodb-enterprise-operator-multi-cluster" does not exist. Installing it now.
   2
   NAME: mongodb-enterprise-operator-multi-cluster
   3
   LAST DEPLOYED: Tue Apr 30 19:40:26 2024
   4
   NAMESPACE: mongodb-operator
   5
   STATUS: deployed
   6
   REVISION: 1
   7
   TEST SUITE: None
   8
   USER-SUPPLIED VALUES:
   9
   dummy: value
   10
   multiCluster:
   11
     clusters:
   12
     - gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   13
     - gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   14
     - gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
   15
   namespace: mongodb-operator
   16
   operator:
   17
     createOperatorServiceAccount: false
   18
     createResourcesServiceAccountsAndRoles: false
   19
     name: mongodb-enterprise-operator-multi-cluster
   20
     namespace: mongodb-operator
   21
     watchNamespace: mongodb
   22
   23
   COMPUTED VALUES:
   24
   agent:
   25
     name: mongodb-agent-ubi
   26
     version: 107.0.0.8502-1
   27
   database:
   28
     name: mongodb-enterprise-database-ubi
   29
     version: 1.25.0
   30
   dummy: value
   31
   initAppDb:
   32
     name: mongodb-enterprise-init-appdb-ubi
   33
     version: 1.25.0
   34
   initDatabase:
   35
     name: mongodb-enterprise-init-database-ubi
   36
     version: 1.25.0
   37
   initOpsManager:
   38
     name: mongodb-enterprise-init-ops-manager-ubi
   39
     version: 1.25.0
   40
   managedSecurityContext: false
   41
   mongodb:
   42
     appdbAssumeOldFormat: false
   43
     imageType: ubi8
   44
     name: mongodb-enterprise-server
   45
     repo: quay.io/mongodb
   46
   mongodbLegacyAppDb:
   47
     name: mongodb-enterprise-appdb-database-ubi
   48
     repo: quay.io/mongodb
   49
   multiCluster:
   50
     clusterClientTimeout: 10
   51
     clusters:
   52
     - gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   53
     - gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   54
     - gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
   55
     kubeConfigSecretName: mongodb-enterprise-operator-multi-cluster-kubeconfig
   56
     performFailOver: true
   57
   namespace: mongodb-operator
   58
   operator:
   59
     additionalArguments: []
   60
     affinity: {}
   61
     createOperatorServiceAccount: false
   62
     createResourcesServiceAccountsAndRoles: false
   63
     deployment_name: mongodb-enterprise-operator
   64
     env: prod
   65
     mdbDefaultArchitecture: non-static
   66
     name: mongodb-enterprise-operator-multi-cluster
   67
     namespace: mongodb-operator
   68
     nodeSelector: {}
   69
     operator_image_name: mongodb-enterprise-operator-ubi
   70
     replicas: 1
   71
     resources:
   72
       limits:
   73
         cpu: 1100m
   74
         memory: 1Gi
   75
       requests:
   76
         cpu: 500m
   77
         memory: 200Mi
   78
     tolerations: []
   79
     vaultSecretBackend:
   80
       enabled: false
   81
       tlsSecretRef: ""
   82
     version: 1.25.0
   83
     watchNamespace: mongodb
   84
     watchedResources:
   85
     - mongodb
   86
     - opsmanagers
   87
     - mongodbusers
   88
     webhook:
   89
       registerConfiguration: true
   90
   opsManager:
   91
     name: mongodb-enterprise-ops-manager-ubi
   92
   registry:
   93
     agent: quay.io/mongodb
   94
     appDb: quay.io/mongodb
   95
     database: quay.io/mongodb
   96
     imagePullSecrets: null
   97
     initAppDb: quay.io/mongodb
   98
     initDatabase: quay.io/mongodb
   99
     initOpsManager: quay.io/mongodb
   100
     operator: quay.io/mongodb
   101
     opsManager: quay.io/mongodb
   102
     pullPolicy: Always
   103
   subresourceEnabled: true
   104
   105
   HOOKS:
   106
   MANIFEST:
   107
   ---
   108
   # Source: enterprise-operator/templates/operator-roles.yaml
   109
   kind: ClusterRole
   110
   apiVersion: rbac.authorization.k8s.io/v1
   111
   metadata:
   112
     name: mongodb-enterprise-operator-mongodb-webhook
   113
   rules:
   114
     - apiGroups:
   115
         - "admissionregistration.k8s.io"
   116
       resources:
   117
         - validatingwebhookconfigurations
   118
       verbs:
   119
         - get
   120
         - create
   121
         - update
   122
         - delete
   123
     - apiGroups:
   124
         - ""
   125
       resources:
   126
         - services
   127
       verbs:
   128
         - get
   129
         - list
   130
         - watch
   131
         - create
   132
         - update
   133
         - delete
   134
   ---
   135
   # Source: enterprise-operator/templates/operator-roles.yaml
   136
   kind: ClusterRoleBinding
   137
   apiVersion: rbac.authorization.k8s.io/v1
   138
   metadata:
   139
     name: mongodb-enterprise-operator-multi-cluster-mongodb-operator-webhook-binding
   140
   roleRef:
   141
     apiGroup: rbac.authorization.k8s.io
   142
     kind: ClusterRole
   143
     name: mongodb-enterprise-operator-mongodb-webhook
   144
   subjects:
   145
     - kind: ServiceAccount
   146
       name: mongodb-enterprise-operator-multi-cluster
   147
       namespace: mongodb-operator
   148
   ---
   149
   # Source: enterprise-operator/templates/operator.yaml
   150
   apiVersion: apps/v1
   151
   kind: Deployment
   152
   metadata:
   153
     name: mongodb-enterprise-operator-multi-cluster
   154
     namespace: mongodb-operator
   155
   spec:
   156
     replicas: 1
   157
     selector:
   158
       matchLabels:
   159
         app.kubernetes.io/component: controller
   160
         app.kubernetes.io/name: mongodb-enterprise-operator-multi-cluster
   161
         app.kubernetes.io/instance: mongodb-enterprise-operator-multi-cluster
   162
     template:
   163
       metadata:
   164
         labels:
   165
           app.kubernetes.io/component: controller
   166
           app.kubernetes.io/name: mongodb-enterprise-operator-multi-cluster
   167
           app.kubernetes.io/instance: mongodb-enterprise-operator-multi-cluster
   168
       spec:
   169
         serviceAccountName: mongodb-enterprise-operator-multi-cluster
   170
         securityContext:
   171
           runAsNonRoot: true
   172
           runAsUser: 2000
   173
         containers:
   174
           - name: mongodb-enterprise-operator-multi-cluster
   175
             image: "quay.io/mongodb/mongodb-enterprise-operator-ubi:1.25.0"
   176
             imagePullPolicy: Always
   177
             args:
   178
               - -watch-resource=mongodb
   179
               - -watch-resource=opsmanagers
   180
               - -watch-resource=mongodbusers
   181
               - -watch-resource=mongodbmulticluster
   182
             command:
   183
               - /usr/local/bin/mongodb-enterprise-operator
   184
             volumeMounts:
   185
               - mountPath: /etc/config/kubeconfig
   186
                 name: kube-config-volume
   187
             resources:
   188
               limits:
   189
                 cpu: 1100m
   190
                 memory: 1Gi
   191
               requests:
   192
                 cpu: 500m
   193
                 memory: 200Mi
   194
             env:
   195
               - name: OPERATOR_ENV
   196
                 value: prod
   197
               - name: MDB_DEFAULT_ARCHITECTURE
   198
                 value: non-static
   199
               - name: WATCH_NAMESPACE
   200
                 value: "mongodb"
   201
               - name: NAMESPACE
   202
                 valueFrom:
   203
                   fieldRef:
   204
                     fieldPath: metadata.namespace
   205
               - name: CLUSTER_CLIENT_TIMEOUT
   206
                 value: "10"
   207
               - name: IMAGE_PULL_POLICY
   208
                 value: Always
   209
               # Database
   210
               - name: MONGODB_ENTERPRISE_DATABASE_IMAGE
   211
                 value: quay.io/mongodb/mongodb-enterprise-database-ubi
   212
               - name: INIT_DATABASE_IMAGE_REPOSITORY
   213
                 value: quay.io/mongodb/mongodb-enterprise-init-database-ubi
   214
               - name: INIT_DATABASE_VERSION
   215
                 value: 1.25.0
   216
               - name: DATABASE_VERSION
   217
                 value: 1.25.0
   218
               # Ops Manager
   219
               - name: OPS_MANAGER_IMAGE_REPOSITORY
   220
                 value: quay.io/mongodb/mongodb-enterprise-ops-manager-ubi
   221
               - name: INIT_OPS_MANAGER_IMAGE_REPOSITORY
   222
                 value: quay.io/mongodb/mongodb-enterprise-init-ops-manager-ubi
   223
               - name: INIT_OPS_MANAGER_VERSION
   224
                 value: 1.25.0
   225
               # AppDB
   226
               - name: INIT_APPDB_IMAGE_REPOSITORY
   227
                 value: quay.io/mongodb/mongodb-enterprise-init-appdb-ubi
   228
               - name: INIT_APPDB_VERSION
   229
                 value: 1.25.0
   230
               - name: OPS_MANAGER_IMAGE_PULL_POLICY
   231
                 value: Always
   232
               - name: AGENT_IMAGE
   233
                 value: "quay.io/mongodb/mongodb-agent-ubi:107.0.0.8502-1"
   234
               - name: MDB_AGENT_IMAGE_REPOSITORY
   235
                 value: "quay.io/mongodb/mongodb-agent-ubi"
   236
               - name: MONGODB_IMAGE
   237
                 value: mongodb-enterprise-server
   238
               - name: MONGODB_REPO_URL
   239
                 value: quay.io/mongodb
   240
               - name: MDB_IMAGE_TYPE
   241
                 value: ubi8
   242
               - name: PERFORM_FAILOVER
   243
                 value: 'true'
   244
         volumes:
   245
           - name: kube-config-volume
   246
             secret:
   247
               defaultMode: 420
   248
               secretName: mongodb-enterprise-operator-multi-cluster-kubeconfig

2. 检查 Kubernetes Operator 部署：

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" rollout status deployment/mongodb-enterprise-operator-multi-cluster
   2
   echo "Operator deployment in ${OPERATOR_NAMESPACE} namespace"
   3
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" get deployments
   4
   echo; echo "Operator pod in ${OPERATOR_NAMESPACE} namespace"
   5
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" get pods

   查看输出

   1
   Waiting for deployment "mongodb-enterprise-operator-multi-cluster" rollout to finish: 0 of 1 updated replicas are available...
   2
   deployment "mongodb-enterprise-operator-multi-cluster" successfully rolled out
   3
   Operator deployment in mongodb-operator namespace
   4
   NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
   5
   mongodb-enterprise-operator-multi-cluster   1/1     1            1           12s
   6
   7
   Operator pod in mongodb-operator namespace
   8
   NAME                                                         READY   STATUS    RESTARTS     AGE
   9
   mongodb-enterprise-operator-multi-cluster-78cc97547d-nlgds   2/2     Running   1 (3s ago)   12s

准备 TLS 证书。

在此步骤中，您将为应用程序数据库和 应用程序启用 TLSMongoDB Ops Manager 。如果您不想使用 TLS，请从MongoDBOpsManager资源中删除以下字段：

• spec.security.certsSecretPrefix

• spec.security.tls.ca

• spec.applicationDatabase.security.certsSecretPrefix

• spec.applicationDatabase.security.tls.ca

1. 可选。 生成密钥和证书：

   使用openssl命令行工具生成自签名 CA 和证书以进行测试。

   1
   mkdir certs || true
   2
   3
   cat <<EOF >certs/ca.cnf
   4
   [ req ]
   5
   default_bits       = 2048
   6
   prompt             = no
   7
   default_md         = sha256
   8
   distinguished_name = dn
   9
   10
   [ dn ]
   11
   C=US
   12
   ST=New York
   13
   L=New York
   14
   O=Example Company
   15
   OU=IT Department
   16
   CN=exampleCA
   17
   EOF
   18
   19
   cat <<EOF >certs/om.cnf
   20
   [ req ]
   21
   default_bits       = 2048
   22
   prompt             = no
   23
   default_md         = sha256
   24
   distinguished_name = dn
   25
   req_extensions     = req_ext
   26
   27
   [ dn ]
   28
   C=US
   29
   ST=New York
   30
   L=New York
   31
   O=Example Company
   32
   OU=IT Department
   33
   CN=${OPS_MANAGER_EXTERNAL_DOMAIN}
   34
   35
   [ req_ext ]
   36
   subjectAltName = @alt_names
   37
   keyUsage = critical, digitalSignature, keyEncipherment
   38
   extendedKeyUsage = serverAuth, clientAuth
   39
   40
   [ alt_names ]
   41
   DNS.1 = ${OPS_MANAGER_EXTERNAL_DOMAIN}
   42
   DNS.2 = om-svc.${NAMESPACE}.svc.cluster.local
   43
   EOF
   44
   45
   cat <<EOF >certs/appdb.cnf
   46
   [ req ]
   47
   default_bits       = 2048
   48
   prompt             = no
   49
   default_md         = sha256
   50
   distinguished_name = dn
   51
   req_extensions     = req_ext
   52
   53
   [ dn ]
   54
   C=US
   55
   ST=New York
   56
   L=New York
   57
   O=Example Company
   58
   OU=IT Department
   59
   CN=AppDB
   60
   61
   [ req_ext ]
   62
   subjectAltName = @alt_names
   63
   keyUsage = critical, digitalSignature, keyEncipherment
   64
   extendedKeyUsage = serverAuth, clientAuth
   65
   66
   [ alt_names ]
   67
   # multi-cluster mongod hostnames from service for each pod
   68
   DNS.1 = *.${NAMESPACE}.svc.cluster.local
   69
   # single-cluster mongod hostnames from headless service
   70
   DNS.2 = *.om-db-svc.${NAMESPACE}.svc.cluster.local
   71
   EOF
   72
   73
   # generate CA keypair and certificate
   74
   openssl genrsa -out certs/ca.key 2048
   75
   openssl req -x509 -new -nodes -key certs/ca.key -days 1024 -out certs/ca.crt -config certs/ca.cnf
   76
   77
   # generate OpsManager's keypair and certificate
   78
   openssl genrsa -out certs/om.key 2048
   79
   openssl req -new -key certs/om.key -out certs/om.csr -config certs/om.cnf
   80
   81
   # generate AppDB's keypair and certificate
   82
   openssl genrsa -out certs/appdb.key 2048
   83
   openssl req -new -key certs/appdb.key -out certs/appdb.csr -config certs/appdb.cnf
   84
   85
   # generate certificates signed by CA for OpsManager and AppDB
   86
   openssl x509 -req -in certs/om.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/om.crt -days 365 -sha256 -extfile certs/om.cnf -extensions req_ext
   87
   openssl x509 -req -in certs/appdb.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/appdb.crt -days 365 -sha256 -extfile certs/appdb.cnf -extensions req_ext

2. 使用 TLS 密钥创建密钥：

   如果您希望使用自己的密钥和证书，请跳过上一步的生成步骤，并将密钥和证书放入以下文件中：

   • certs/ca.crt — CA证书。 使用受信任证书时，这些都不是必需的。

   • certs/appdb.key — 应用程序数据库的私钥。

   • certs/appdb.crt — 应用程序数据库的证书。

   • certs/om.key - MongoDB Ops Manager的私钥。

   • certs/om.crt - MongoDB Ops Manager证书。

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret tls cert-prefix-om-cert \
   2
     --cert=certs/om.crt \
   3
     --key=certs/om.key
   4
   5
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret tls cert-prefix-om-db-cert \
   6
     --cert=certs/appdb.crt \
   7
     --key=certs/appdb.key
   8
   9
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create configmap om-cert-ca --from-file="mms-ca.crt=certs/ca.crt"
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create configmap appdb-cert-ca --from-file="ca-pem=certs/ca.crt"

安装 Ops Manager

此时，您已准备好用于部署 资源的环境和Kubernetes MongoDB Ops ManagerOperator。

1. 为MongoDB Ops Manager管理员用户创建必要的档案， Kubernetes Operator 在部署MongoDB Ops Manager应用程序实例后将创建该用户：

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${NAMESPACE}" create secret generic om-admin-user-credentials \
   2
     --from-literal=Username="admin" \
   3
     --from-literal=Password="Passw0rd@" \
   4
     --from-literal=FirstName="Jane" \
   5
     --from-literal=LastName="Doe"

2. 在单个成员集群（也称为 Operator 集群）上部署尽可能简单的MongoDBOpsManager自定义资源（启用TLS ）。

   此部署与单集群模式几乎相同，但将spec.topology和spec.applicationDatabase.topology设立为MultiCluster 。

   这种部署方式表明，单个 Kubernetes 集群部署是单个 Kubernetes 成员集群上的多 Kubernetes 集群部署的特例。 MongoDB Ops ManagerKubernetes您可以从一开始就在所需数量的 集群上部署 应用程序和应用程序数据库，而不必从只有单个成员Kubernetes 集群的部署开始。

   此时，您已准备好MongoDB Ops Manager部署以跨越多个Kubernetes集群，本过程稍后将执行此操作。

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
     applicationDatabase:
   18
       version: "${APPDB_VERSION}"
   19
       topology: MultiCluster
   20
       security:
   21
         certsSecretPrefix: cert-prefix
   22
         tls:
   23
           ca: appdb-cert-ca
   24
       clusterSpecList:
   25
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   26
           members: 3
   27
     backup:
   28
       enabled: false
   29
   EOF

3. 等待 Kubernetes Operator 接手工作并进入status.applicationDatabase.phase=Pending状态。 等待应用程序数据库和MongoDB Ops Manager部署完成。

   1
   echo "Waiting for Application Database to reach Pending phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s

   查看输出

   1
   Waiting for Application Database to reach Pending phase...
   2
   mongodbopsmanager.mongodb.com/om condition met

4. 部署MongoDB Ops Manager 。 Kubernetes Operator 通过执行以下步骤来部署MongoDB Ops Manager 。 它：

   • 部署应用程序数据库的副本集节点，并等待副本集中的 MongoDB 进程开始运行。

   • 使用应用程序数据库的连接 部署MongoDB Ops Managerstring 应用程序实例，并等待其准备就绪。

   • 将监控MongoDB Agent容器添加到每个应用程序数据库的 Pod。

   • 等待MongoDB Ops Manager应用程序和应用程序数据库 Pod 开始运行。

   1
   echo "Waiting for Application Database to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   3
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s
   5
   echo; echo "Waiting for Application Database to reach Pending phase (enabling monitoring)..."
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   7
   echo "Waiting for Application Database to reach Running phase..."
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   9
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s
   11
   echo; echo "MongoDBOpsManager resource"
   12
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   13
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   14
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   15
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   16
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   查看输出

   1
   Waiting for Application Database to reach Running phase...
   2
   mongodbopsmanager.mongodb.com/om condition met
   3
   4
   Waiting for Ops Manager to reach Running phase...
   5
   mongodbopsmanager.mongodb.com/om condition met
   6
   7
   Waiting for Application Database to reach Pending phase (enabling monitoring)...
   8
   mongodbopsmanager.mongodb.com/om condition met
   9
   Waiting for Application Database to reach Running phase...
   10
   mongodbopsmanager.mongodb.com/om condition met
   11
   12
   Waiting for Ops Manager to reach Running phase...
   13
   mongodbopsmanager.mongodb.com/om condition met
   14
   15
   MongoDBOpsManager resource
   16
   NAME   REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGS
   17
   om                7.0.4     Running              Running         Disabled         11m   
   18
   19
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   20
   NAME        READY   STATUS    RESTARTS   AGE
   21
   om-0-0      2/2     Running   0          8m39s
   22
   om-db-0-0   4/4     Running   0          44s
   23
   om-db-0-1   4/4     Running   0          2m6s
   24
   om-db-0-2   4/4     Running   0          3m19s
   25
   26
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1

   现在您已经以多集群模式部署了一个单成员集群，您可以重新配置此部署以跨越多个 Kubernetes 集群。

5. 在第二个成员集群上，再部署两个应用程序数据库副本集成员和一个MongoDB Ops Manager应用程序实例：

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
       - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   18
         members: 1
   19
     applicationDatabase:
   20
       version: "${APPDB_VERSION}"
   21
       topology: MultiCluster
   22
       security:
   23
         certsSecretPrefix: cert-prefix
   24
         tls:
   25
           ca: appdb-cert-ca
   26
       clusterSpecList:
   27
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   28
           members: 3
   29
         - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   30
           members: 2
   31
     backup:
   32
       enabled: false
   33
   EOF

6. 等待 Kubernetes Operator 接手工作（待处理阶段）：

   1
   echo "Waiting for Application Database to reach Pending phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s

   查看输出

   1
   Waiting for Application Database to reach Pending phase...
   2
   mongodbopsmanager.mongodb.com/om condition met

7. 等待 Kubernetes Operator 完成所有组件的部署：

   1
   echo "Waiting for Application Database to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s
   3
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s
   5
   echo; echo "MongoDBOpsManager resource"
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   7
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   9
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   10
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   查看输出

   1
   Waiting for Application Database to reach Running phase...
   2
   mongodbopsmanager.mongodb.com/om condition met
   3
   4
   Waiting for Ops Manager to reach Running phase...
   5
   mongodbopsmanager.mongodb.com/om condition met
   6
   7
   MongoDBOpsManager resource
   8
   NAME   REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGS
   9
   om                7.0.4     Pending              Running         Disabled         14m   
   10
   11
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   12
   NAME        READY   STATUS        RESTARTS   AGE
   13
   om-0-0      2/2     Terminating   0          12m
   14
   om-db-0-0   4/4     Running       0          4m12s
   15
   om-db-0-1   4/4     Running       0          5m34s
   16
   om-db-0-2   4/4     Running       0          6m47s
   17
   18
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   19
   NAME        READY   STATUS     RESTARTS   AGE
   20
   om-1-0      0/2     Init:0/2   0          0s
   21
   om-db-1-0   4/4     Running    0          3m24s
   22
   om-db-1-1   4/4     Running    0          104s

启用备份。

在MongoDB Ops Manager应用程序的多 Kubernetes 集群部署中，您只能配置基于S3的备份存储。 此过程引用env_variables.sh 中定义的S3_* 。

1. 可选。 安装 MinIO Operator 。

   此过程使用 MinIO Operator 为备份部署 S3 兼容存储 。如果您有可用的Amazon Web Services S3或其他S3兼容存储桶，则可以跳过此步骤。 在这种情况下，请在env_variables.sh中相应地调整S3_*变量。

   1
   kustomize build "github.com/minio/operator/resources/?timeout=120&ref=v5.0.12" | \
   2
     kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
   3
   4
   kustomize build "github.com/minio/operator/examples/kustomization/tenant-tiny?timeout=120&ref=v5.0.12" | \
   5
     kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
   6
   7
   # add two buckets to the tenant config
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "tenant-tiny" patch tenant/myminio \
   9
     --type='json' \
   10
     -p="[{\"op\": \"add\", \"path\": \"/spec/buckets\", \"value\": [{\"name\": \"${S3_OPLOG_BUCKET_NAME}\"}, {\"name\": \"${S3_SNAPSHOT_BUCKET_NAME}\"}]}]"

2. 在配置和启用备份之前，请创建密钥：

   • s3-access-secret — 包含S 3档案。

   • s3-ca-cert - 包含颁发存储桶服务器证书的CA证书。 在此过程中使用的示例 MinIO 部署中，将使用默认的 Kubernetes 根CA证书对证书进行签名。 由于它不是公众信任的CA证书，因此您必须提供该证书， MongoDB Ops Manager才能信任该连接。

   如果您使用公开信任的证书，则可以跳过此步骤并删除spec.backup.s3Stores.customCertificateSecretRefs和spec.backup.s3OpLogStores.customCertificateSecretRefs设置中的值。

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic s3-access-secret \
   2
     --from-literal=accessKey="${S3_ACCESS_KEY}" \
   3
     --from-literal=secretKey="${S3_SECRET_KEY}"
   4
   5
   # minio TLS secrets are signed with the default k8s root CA
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic s3-ca-cert \
   7
     --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n kube-system get configmap kube-root-ca.crt -o jsonpath="{.data.ca\.crt}")"

在启用备份的情况下重新部署MongoDB Ops Manager 。

1. KubernetesOperatorMongoDB Ops Manager 可以在为其配置Kubernetes Operator 的任何成员集群上以任意组合配置和部署所有组件、 应用程序、备份守护程序实例和应用程序数据库的副本集节点。

   为了说明多 Kubernetes 集群部署配置的灵活性，在第三个成员集群上仅部署一个备份守护进程实例，并为第一个和第二个集群指定零个备份守护进程成员。

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
         backup:
   18
           members: 0
   19
       - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   20
         members: 1
   21
         backup:
   22
           members: 0
   23
       - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}"
   24
         members: 0
   25
         backup:
   26
           members: 1
   27
     configuration: # to avoid configuration wizard on first login
   28
         mms.adminEmailAddr: email@example.com
   29
         mms.fromEmailAddr: email@example.com
   30
         mms.ignoreInitialUiSetup: "true"
   31
         mms.mail.hostname: smtp@example.com
   32
         mms.mail.port: "465"
   33
         mms.mail.ssl: "true"
   34
         mms.mail.transport: smtp
   35
         mms.minimumTLSVersion: TLSv1.2
   36
         mms.replyToEmailAddr: email@example.com
   37
     applicationDatabase:
   38
       version: "${APPDB_VERSION}"
   39
       topology: MultiCluster
   40
       security:
   41
         certsSecretPrefix: cert-prefix
   42
         tls:
   43
           ca: appdb-cert-ca
   44
       clusterSpecList:
   45
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   46
           members: 3
   47
         - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   48
           members: 2
   49
     backup:
   50
       enabled: true
   51
       s3Stores:
   52
         - name: my-s3-block-store
   53
           s3SecretRef:
   54
             name: "s3-access-secret"
   55
           pathStyleAccessEnabled: true
   56
           s3BucketEndpoint: "${S3_ENDPOINT}"
   57
           s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}"
   58
           customCertificateSecretRefs:
   59
             - name: s3-ca-cert
   60
               key: ca.crt
   61
       s3OpLogStores:
   62
         - name: my-s3-oplog-store
   63
           s3SecretRef:
   64
             name: "s3-access-secret"
   65
           s3BucketEndpoint: "${S3_ENDPOINT}"
   66
           s3BucketName: "${S3_OPLOG_BUCKET_NAME}"
   67
           pathStyleAccessEnabled: true
   68
           customCertificateSecretRefs:
   69
             - name: s3-ca-cert
   70
               key: ca.crt
   71
   EOF

2. 等待 Kubernetes Operator 完成配置：

   1
   echo; echo "Waiting for Backup to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200s
   3
   echo "Waiting for Application Database to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s
   5
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s
   7
   echo; echo "MongoDBOpsManager resource"
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   9
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   11
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   12
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   13
   echo; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"
   14
   kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   查看输出

   1
   Waiting for Backup to reach Running phase...
   2
   mongodbopsmanager.mongodb.com/om condition met
   3
   Waiting for Application Database to reach Running phase...
   4
   mongodbopsmanager.mongodb.com/om condition met
   5
   6
   Waiting for Ops Manager to reach Running phase...
   7
   mongodbopsmanager.mongodb.com/om condition met
   8
   9
   MongoDBOpsManager resource
   10
   NAME   REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGS
   11
   om                7.0.4     Running              Running         Running          22m   
   12
   13
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   14
   NAME        READY   STATUS    RESTARTS   AGE
   15
   om-0-0      2/2     Running   0          7m10s
   16
   om-db-0-0   4/4     Running   0          11m
   17
   om-db-0-1   4/4     Running   0          13m
   18
   om-db-0-2   4/4     Running   0          14m
   19
   20
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   21
   NAME        READY   STATUS    RESTARTS   AGE
   22
   om-1-0      2/2     Running   0          4m8s
   23
   om-db-1-0   4/4     Running   0          11m
   24
   om-db-1-1   4/4     Running   0          9m25s
   25
   26
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
   27
   NAME                   READY   STATUS    RESTARTS   AGE
   28
   om-2-backup-daemon-0   2/2     Running   0          2m5s

可选。删除 GKE (Google Kubernetes Engine) 集群及其所有相关资源 (VM)。

运行以下脚本以删除 GKE 集群并清理环境。

1
yes | gcloud container clusters delete "${K8S_CLUSTER_0}" --zone="${K8S_CLUSTER_0_ZONE}" &
2
yes | gcloud container clusters delete "${K8S_CLUSTER_1}" --zone="${K8S_CLUSTER_1_ZONE}" &
3
yes | gcloud container clusters delete "${K8S_CLUSTER_2}" --zone="${K8S_CLUSTER_2_ZONE}" &
4
wait