Docs Menu
Docs Home
/
MongoDB Atlas
/
MongoDB Atlas Kubernetes Operator
/
Atlas Access

Configure Secret Storage

On this page

  • Considerations
  • Prerequisites
  • Procedure

You can choose where you store sensitive information for the components that Atlas Kubernetes Operator manages, but Atlas Kubernetes Operator must find the Kubernetes secrets it expects. You can store secrets for Atlas Kubernetes Operator in many ways, including the following methods:

  • Put sensitive information directly into Kubernetes secrets. All tutorials in the Atlas Kubernetes Operator documentation use Kubernetes secrets by default. To use Kubernetes secrets, follow the steps in the tutorials.

  • Put sensitive information in a Github repository following a GitOps flow. To store sensitive data in git securely you can use tools, such as Sealed Secrets, which encrypts secrets for the intended target cluster.

  • Put sensitive information in an external secret storage tool, such as HashiCorp Vault or Hyperscalers native secret management solutions. An intermediary secret provisioning tool fetches sensitive info from the external secret storage tool and creates Kubernetes secrets from the sensitive information. To learn more about the secret provisioning tool, see Considerations.

This tutorial sets up an external secret storage tool for use with Atlas Kubernetes Operator. This tutorial focuses on "secret-less" setups that don't require Atlas Kubernetes Operator to create and store a secret to provision secrets to their Kubernetes cluster.

Considerations

The following tutorial installs or configures the following tools and options:

  • A secret provisioning tool. The secret provisioning tool uses one or more authentication mechanisms to retrieve the credentials from the secret management service and create secrets that Atlas Kubernetes Operator can use. This tutorial installs one of the following open-source secret provisioning tools:

  • Authentication to access secrets. You can use different methods to authenticate the service accounts and namespaces that can access secrets in HashiCorp Vault:

    • For External Secrets Operator, this tutorial uses OIDC JWT authentication. To learn more, see JWT/OIDC authentication.

    • For Secrets Store CSI Driver, this tutorial uses Kubernetes authentication.

    Alternatively, your cloud provider's KMS can use native IAM systems to provide this authentication, which isn't covered in this tutorial. To learn how to configure your cloud provider's KMS for authentication, see the following resources in the External Secrets Operator documentation:

Prerequisites

Before you complete this tutorial, you need the following tools and configurations:

  • Running service accounts for Kubernetes, Atlas Kubernetes Operator, and Atlas and sufficient privileges to configure them.

    You need a running Kubernetes cluster with nodes running processors with the x86-64, AMD64, or ARM64 architecture. For this tutorial, the Kubernetes cluster is https://kube01.internal.io listening on the default port (443).

    You can access the Atlas Kubernetes Operator project on GitHub:

    To install the Atlas Kubernetes operator using the Atlas CLI, run the following command:

    atlas kubernetes operator install [options]

    To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas kubernetes operator install.

    Tip

    See: Related Links

    To deploy the Atlas Kubernetes Operator, run the following command. Replace <version> with the latest release number.

    kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-atlas-kubernetes/<version>/deploy/all-in-one.yaml

    To register for an Atlas account, see Create an Atlas Account.

  • API keys. You must create an API key and configure the API Access List.

    You need the following public API key, private API key, and the organization ID information to configure Atlas Kubernetes Operator access to Atlas.

    • If you want Atlas Kubernetes Operator to create a new Atlas project, Create an API (Application Programming Interface) Key in an Organization. If your organization requires an IP access list for the Atlas Administration API, you must also configure the API access list.

      Important

      You must assign the API key the Organization Project Creator organization role or higher.

    • If you want to work with an existing Atlas project, Create an API (Application Programming Interface) Key for a Project. If your organization requires an IP access list for the Atlas Administration API, you must also configure the API access list.

      Important

      You must assign the API key the Project Owner project role.

  • A secret storage vault. This tutorial uses HashiCorp Vault, which is a third-party service for secret storage, running at https://vault.internal.io.

    You can use other secret storage vaults with Atlas Kubernetes Operator as needed, including Cloud KMS from AWS, Azure, and Google.

  • Internal access only. To prevent exposing sensitive information over the public internet, the following components of the secret storage solution allow internal access only:

    • The HashiCorp Vault or KMS service.

    • The Kubernetes Cluster APIs service.

    • The internal network. This tutorial uses internal.io.

    While the previous components allow internal access only, they allow access to each other and allow access to anyone within your team or organization. This is a best practice for security.

  • Public Certificate Authorities (CAs). You can use public CAs to avoid managing and distributing custom CA root certificates.

    You can automate CA cert management and renewal by using any of the following tools:

    In this tutorial:

    • All internal.io HTTPs services are internal addresses, but their HTTPS sites hold automatically renewed certificates signed by a public CA.

    • No mutual TLS (mTLS) is required for this integration because it performs only server-side HTTPS validation.

    • Clients can trust these service certificates without extra certificate provisioning.

Procedure

Follow these steps to configure secret storage for Atlas Kubernetes Operator.

1

Install the secret provisioning tool in the target cluster.

Select a secret provisioning tool to install it.

To use External Secrets Operator as the secret provisioning tool:

  1. Run the following commands to install External Secrets Operator with Helm Charts and start the service:

    helm repo add external-secrets https://charts.external-secrets.io
    helm upgrade -i --atomic \
    -n external-secrets --create-namespace --set extraArgs.loglevel=debug \
    external-secrets external-secrets/external-secrets`

  2. Ensure External Secrets runs successfully:

    kubectl get pod -n external-secrets -l app.kubernetes.io/name=external-secrets
    NAME                                READY   STATUS    RESTARTS   AGE
    external-secrets-5779d5d6f6-2lhgd   1/1     Running   0          70s

To use Secrets Store CSI Driver as the secret provisioning tool, follow these steps:

  1. Run the following command to install Secrets Store CSI Driver with Helm Charts and start the service:

    helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
    helm upgrade -i --atomic --set syncSecret.enabled=true \
      -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

  2. Run the following command to install the Secrets Store CSI HashiCorp Vault plugin with Helm Charts. You don't need to install the HashiCorp Vault server or secrets injector.

    helm install vault hashicorp/vault \
    --set "server.enabled=false" --set "injector.enabled=false" \
    --set "csi.enabled=true"

  3. Ensure Secrets Store CSI runs successfully:

    kubectl get pod -n kube-system -l app.kubernetes.io/name=secrets-store-csi-driver
    NAME                                               READY   STATUS    RESTARTS   AGE
    csi-secrets-store-secrets-store-csi-driver-6dcm8   3/3     Running   0          2m2s

  4. Ensure the HashiCorp Vault CSI provider runs successfully:

    kubectl get pods -l app.kubernetes.io/name=vault-csi-provider
    NAME                       READY   STATUS    RESTARTS   AGE
    vault-csi-provider-j7xbr   2/2     Running   0          5m39s
2

Set up authentication to access secrets.

To set up OIDC JWT and Kubernetes authentication:

  1. Run the following command to enable OIDC JWT authentication for the mount path. If you set up several Kubernetes clusters, you must enable OIDC JWT authentication for each cluster's mount path.

    vault auth enable -path=jwt-kube01 jwt

  2. Run the following command to allow unauthenticated access to the OIDC discovery URLs on the kube01.internal.io cluster:

    $ kubectl create clusterrolebinding oidc-reviewer  \
      --clusterrole=system:service-account-issuer-discovery \
      --group=system:unauthenticated

  3. Run the following command to direct HashiCorp Vault to trust the cluster. The issuer for the cluster at https://kube01.internal.io/ must match the URL in the OIDC discovery document at .well-known/openid-configuration.

    curl https://kube01.internal.io/.well-known/openid-configuration | jq .issuer
    https://kube01.internal.io/
    vault write auth/jwt-kube01/config jwks_url="https://kube01.internal.io/openid/v1/jwks"

  4. Create the policies that allow access to the secrets you want to expose to the cluster.

    The following example creates the external-secrets policy that you specify in later steps.

    echo external-secrets-policy.hcl
    path "secret/data/kube01/external-secrets/*" {
      capabilities = ["read"]
    }

  5. Run the command to create a HashiCorp Vault role and bind a Kubernetes service account to the role, which restricts the access of the service account within the HashiCorp Vault.

    The following command creates the jwt-kube01-system role of the JWT (OIDC) type for the vault audience. The command specifies the sub user claim, and the default Kubernetes service account in the mongodb-atlas-system namespace as the bound subject. This command ties the role to the external-secrets policy set of permissions within HashiCorp Vault.

    vault write auth/jwt-kube01/role/jwt-kube01-system role_type="jwt" bound_audiences=vault \
      user_claim="sub" bound_subject="system:serviceaccount:mongodb-atlas-system:default" \
      policies="external-secrets"

  6. Run the command to create another role for External Secrets Operator to access the default namespace.

    The following command creates the jwt-kube01-default role of the JWT (OIDC) type for the vault audience. This command specifies the sub user claim, and the default Kubernetes service account in the default namespace as the bound subject . This command ties the role to the external-secrets policy set of permissions within HashiCorp Vault.

    vault write auth/jwt-kube01/role/jwt-kube01-default role_type="jwt" bound_audiences=vault \
      user_claim="sub" bound_subject="system:serviceaccount:default:default" \
      policies="external-secrets"

  7. Ensure OIDC authentication runs successfully for the system service account:

    export TEST_JWT_TOKEN=$(kubectl -n mongodb-atlas-system create token default --audience "vault")
    vault write auth/jwt-kube01/login role=jwt-kube01-system jwt=$(TEST_JWT_TOKEN)

    Atlas Kubernetes Operator returns your OIDC JWT credentials for the system service account.

  8. Ensure OIDC authentication runs successfully for the default service account:

    export TEST_JWT_TOKEN=$(kubectl -n default create token default --audience "vault")
    vault write auth/jwt-kube01/login role=jwt-kube01-default jwt=$(TEST_JWT_TOKEN)

    Atlas Kubernetes Operator returns your OIDC JWT credentials for the default service account.

To set up Kubernetes authentication:

  1. Run the following command to enable Kubernetes authentication for the mount path. If you set up several Kubernetes clusters, you must enable Kubernetes authentication for each cluster's mount path.

    vault auth enable -path=jwt-kube01 kubernetes

    When you install with Helm, it automatically configures cluster role binding.

  2. Create policies that allow access to the secrets you want to expose to the cluster.

    The following example creates the policy vault-secret used in later steps. It uses vault, a Kubernetes secret bound to the HashiCorp Vault service account that the Helm Chart CSI provider sets up.

    echo vault-secret.yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: vault
      annotations:
        kubernetes.io/service-account.name: vault
    type: kubernetes.io/service-account-token
    $ kubectl apply -f vault-secret.yaml

  3. Run the commands to trust the Kubernetes cluster and use Kubernetes authentication:

    export VAULT_JWT ?= $(shell kubectl get secret/vault -o jsonpath='{.data.token}' |base64 -d)
    vault write auth/k8s-kube01/config kubernetes_host="kube01.internal.io" token_reviewer_jwt=$(VAULT_JWT)

  4. Run the command to create a HashiCorp Vault role and bind a Kubernetes service account to the role, which restricts the access of the service account within the HashiCorp Vault.

    The following command creates the k8s-kube01 role at auth/k8s-kube01 bound to the Kubernetes service account in the default or mongodb-atlas-system namespaces. This role ties to the secrets-store permissions policy within HashiCorp Vault.

    vault write auth/k8s-kube01/role/k8s-kube01-role \
      bound_service_account_names=default,mongodb-atlas-operator \
      bound_service_account_namespaces=default,mongodb-atlas-system \
      policies=secrets-store

    The policies must exist and allow access to the secrets. The following example policy allows access to KV v2 secrets under the kube01/secrets-store path:

    path "secret/data/kube01/secrets-store/*" {
      capabilities = ["read"]
    }

  5. Ensure Kubernetes authentication runs successfully for the system service account:

    export TEST_JWT_TOKEN=$( kubectl create -n mongodb-atlas-system token mongodb-atlas-operator)
    vault write auth/jwt-kube01/login role=jwt-kube01-system jwt=$(TEST_JWT_TOKEN)

  6. Ensure Kubernetes authentication runs successfully for the default account:

    export TEST_JWT_TOKEN=$(kubectl -n default create token default)
    vault write auth/jwt-kube01/login role=jwt-kube01-default jwt=$(TEST_JWT_TOKEN)
3

Set up automatic secret provisioning.

To provision secrets with External Secrets Operator:

  1. Deploy the SecretStore custom resource for the default service account in the mongodb-atlas-system namespace:

    $ cat external-secrets/vault-system.yaml
    apiVersion: external-secrets.io/v1beta1
    kind: SecretStore
    metadata:
      name: vault-store
      namespace: mongodb-atlas-system
    spec:
      provider:
        vault:
          server: "https://vault.internal.io"
          path: "secret"
          version: "v2"
          auth:
            jwt:
              path: "jwt-kube01"
              role: "jwt-kube01-system"
              kubernetesServiceAccountToken:
                expirationSeconds: 600
                serviceAccountRef:
                  name: "default"
                  audiences:
                  - vault
    $ kubectl apply -f external-secrets/vault-system.yaml

  2. Deploy the SecretStore custom resource for the default service account in the default namespace:

    $ cat external-secrets/vault-default.yaml
    apiVersion: external-secrets.io/v1beta1
    kind: SecretStore
    metadata:
      name: vault-store
      namespace: default
    spec:
      provider:
        vault:
          server: "https://vault.internal.io"
          path: "secret"
          version: "v2"
          auth:
            jwt:
              path: "jwt-kube01"
              role: "jwt-role"
              kubernetesServiceAccountToken:
                expirationSeconds: 600
                serviceAccountRef:
                  name: "default"
                  audiences:
                  - vault
    $ kubectl apply -f external-secrets/vault-default.yaml

  3. Deploy the ExternalSecret custom resource that references the secret that contains the API key. You must set spec.target.template.metadata.labels to atlas.mongodb.com/type with the value credentials for Atlas Kubernetes Operator to find the secret that External Secrets Operator creates.

    Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path secret/data/kube01/external-secrets/atlas-account with the following properties:

    • orgId

    • publicApiKey

    • privateApiKey

    $ cat external-secrets/atlas.yaml
    apiVersion: external-secrets.io/v1beta1
    kind: ExternalSecret
    metadata:
      name: atlas
      namespace: mongodb-atlas-system
    spec:
        refreshInterval: "15s"
        secretStoreRef:
          name: vault-store
          kind: SecretStore
        target:
          name: mongodb-atlas-operator-api-key
          template:
            metadata:
              labels:
                atlas.mongodb.com/type: credentials
        data:
        - secretKey: orgId
          remoteRef:
            key: secret/data/kube01/external-secrets/atlas-account
            property: orgId
        - secretKey: publicApiKey
          remoteRef:
            key: secret/data/kube01/external-secrets/atlas-account
            property: publicApiKey
        - secretKey: privateApiKey
          remoteRef:
            key: secret/data/kube01/external-secrets/atlas-account
            property: privateApiKey
      $ kubectl apply -f external-secrets/atlas.yaml

    This command creates the Kubernetes secret mongodb-atlas-operator-api-key in the mongodb-atlas-system namespace.

  4. Deploy the ExternalSecret custom resource that references the secret that contains database user credentials. You must set spec.target.template.metadata.labels to atlas.mongodb.com/type with the value credentials for Atlas Kubernetes Operator to find the secret that External Secrets Operator creates.

    Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path secret/data/kube01/external-secrets/db-user with the password property.

    $ cat external-secrets/dbuser.yaml
    apiVersion: external-secrets.io/v1beta1
    kind: ExternalSecret
    metadata:
      name: dbuser
      namespace: default
    spec:
        refreshInterval: "15s"
        secretStoreRef:
          name: vault-store
          kind: SecretStore
        target:
          name: dbuser-password
          template:
            metadata:
              labels:
                atlas.mongodb.com/type: credentials
        data:
        - secretKey: password
          remoteRef:
            key: secret/data/kube01/external-secrets/db-user
            property: password
      $ kubectl apply -f external-secrets/atlas.yaml

  5. Ensure the secrets return as expected when you run the following commands:

    $ kubectl get -n mongodb-atlas-system secrets/mongodb-atlas-operator-api-key
    $ kubectl get -n default secrets/dbuser-password

To provision secrets with Secrets Store CSI:

  1. Deploy the SecretProviderClass custom resource in the mongodb-atlas-system namespace. You must set spec.secretObjects.labels to atlas.mongodb.com/type with the value credentials for Atlas Kubernetes Operator to find the secret that Secrets Store CSI creates.

    Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path secret/data/kube01/external-secrets/atlas-account with the following properties:

    • orgId

    • publicApiKey

    • privateApiKey

    $ cat secrets-store/atlas.yaml
    apiVersion: secrets-store.csi.x-k8s.io/v1
    kind: SecretProviderClass
    metadata:
      name: atlas
      namespace: mongodb-atlas-system
    spec:
      provider: vault
      secretObjects:
      - data:
        - key: orgId
          objectName: atlas-org
        - key: publicApiKey
          objectName: atlas-pub-key
        - key: privateApiKey
          objectName: atlas-secret-key
        secretName: mongodb-atlas-operator-api-key
        type: Opaque
        labels:
          atlas.mongodb.com/type: credentials
      parameters:
        vaultAddress: https://vault.internal.io
        vaultKubernetesMountPath: k8s-kube01
        roleName: k8s-kube01-role
        objects: |
          - objectName: atlas-org
            secretPath: secret/data/kube01/secrets-store/atlas-account
            secretKey: orgId
          - objectName: atlas-pub-key
            secretPath: secret/data/kube01/secrets-store/atlas-account
            secretKey: publicApiKey
          - objectName: atlas-secret-key
            secretPath: secret/data/kube01/secrets-store/atlas-account
            secretKey: privateApiKey
    $ kubectl apply -f secrets-store/atlas.yaml

    This command creates the Kubernetes secret mongodb-atlas-operator-api-key in the mongodb-atlas-system namespace.

  2. Run the following command to add a pod that mounts the required secrets:

    $ cat secrets-store/ako-patch.yaml
      template:
        spec:
          containers:
          - name: system-secret-placeholder
            image: mongodb/atlas
            command: ["sleep", "infinity"]
            volumeMounts:
            - name: secrets-store-mount
              mountPath: "/mnt/secrets-store"
              readOnly: true
          volumes:
            - name: secrets-store-mount
              csi:
                driver: secrets-store.csi.k8s.io
                readOnly: true
                volumeAttributes:
                  secretProviderClass: atlas
    $ kubectl apply -f secrets-store/ako-patch.yaml

  3. Deploy the SecretProviderClass custom resource that references the secret that contains database user credentials. You must set spec.target.template.metadata.labels to atlas.mongodb.com/type with the value credentials for Atlas Kubernetes Operator to find the secret that Secrets Store CSI creates.

    Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path secret/data/kube01/external-secrets/db-user with the password property.

    $ cat external-secrets/dbuser.yaml
    apiVersion: external-secrets.io/v1beta1
    kind: SecretProviderClass
    metadata:
      name: dbuser
      namespace: default
    spec:
      provider: vault
      secretObjects:
      - data:
        - key: password
          objectName: dbuser
        secretName: dbuser-password
        type: Opaque
        labels:
          atlas.mongodb.com/type: credentials
     parameters:
        vaultAddress: https://vault.internal.io
        vaultKubernetesMountPath: k8s-kube01
        roleName: k8s-kube01-role
        objects: |
          - objectName: "dbuser"
            secretPath: "secret/data/kube01/secrets-store/db-user"
            secretKey: "password"
    $ kubectl apply -f secrets-store/dbuser.yaml

  4. Run the following command to create the secret-placeholder sentinel pod, which ensures the Secrets Store CSI driver fetches the dbuser credentials and sync them to Kubernetes:

    $ cat secrets-store/placeholder.yaml
    kind: Pod
    apiVersion: v1
    metadata:
      name: secret-placeholder
    spec:
      containers:
      - image: mongodb/atlas
        command: ["sleep", "infinity"]
        name: secret-placeholder
        volumeMounts:
        - name: secrets-store-mount
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-mount
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: dbuser
    $ kubectl apply -f secrets-store/placeholder.yaml

  5. Ensure the secrets return as expected when you run the following commands:

    $ kubectl get -n mongodb-atlas-system secrets/mongodb-atlas-operator-api-key
    $ kubectl get -n default secrets/dbuser-password
4

Deploy Atlas Kubernetes Operator custom resources.

You can now deploy Atlas Kubernetes Operator custom resources. Atlas Kubernetes Operator authenticates with the Kubernetes secrets that reference your HashiCorp Vault. Adjust the timeout values as needed for your deployments.

kubectl apply -f ako/project.yaml
kubectl apply -f ako/deployment.yaml
kubectl apply -f ako/user.yaml
kubectl wait --for=condition=ready atlasdeployment/serverless-deployment --timeout=10m
kubectl wait --for=condition=ready atlasdatabaseuser/user --timeout=10m

To learn more about these custom resources, see Custom Resources.

5

Test your Atlas Kubernetes Operator deployment.

To test your Atlas Kubernetes Operator deployment, run the following command:

export ATLAS_DEPLOYMENT_CONN_STR=$(kubectl get secrets/test-atlas-operator-project-test-serverless-deployment-dbuser -o jsonpath='{.data.connectionStringStandardSrv}' |base64 -d)
mongosh $(ATLAS_DEPLOYMENT_CONN_STR) --apiVersion 1 --eval "show dbs"

Atlas Kubernetes Operator returns a list of your database deployments.

Back

Atlas Access