Configure Secret Storage

On this page

Considerations

Prerequisites
Procedure

You can choose where you store sensitive information for the components that Atlas Kubernetes Operator manages, but Atlas Kubernetes Operator must find the Kubernetes secrets it expects. You can store secrets for Atlas Kubernetes Operator in many ways, including the following methods:

Put sensitive information directly into Kubernetes secrets. All tutorials in the Atlas Kubernetes Operator documentation use Kubernetes secrets by default. To use Kubernetes secrets, follow the steps in the tutorials.
Put sensitive information in a Github repository following a GitOps flow. To store sensitive data in git securely you can use tools, such as Sealed Secrets, which encrypts secrets for the intended target cluster.
Put sensitive information in an external secret storage tool, such as HashiCorp Vault or Hyperscalers native secret management solutions. An intermediary secret provisioning tool fetches sensitive info from the external secret storage tool and creates Kubernetes secrets from the sensitive information. To learn more about the secret provisioning tool, see Considerations.

This tutorial sets up an external secret storage tool for use with Atlas Kubernetes Operator. This tutorial focuses on "secret-less" setups that don't require Atlas Kubernetes Operator to create and store a secret to provision secrets to their Kubernetes cluster.

Considerations

The following tutorial installs or configures the following tools and options:

A secret provisioning tool. The secret provisioning tool uses one or more authentication mechanisms to retrieve the credentials from the secret management service and create secrets that Atlas Kubernetes Operator can use. This tutorial installs one of the following open-source secret provisioning tools:
- External Secrets Operator
- Secrets Store CSI Driver
Authentication to access secrets. You can use different methods to authenticate the service accounts and namespaces that can access secrets in HashiCorp Vault:
- For External Secrets Operator, this tutorial uses OIDC JWT authentication. To learn more, see JWT/OIDC authentication.
- For Secrets Store CSI Driver, this tutorial uses Kubernetes authentication.
Alternatively, your cloud provider's KMS can use native IAM systems to provide this authentication, which isn't covered in this tutorial. To learn how to configure your cloud provider's KMS for authentication, see the following resources in the External Secrets Operator documentation:

Prerequisites

Before you complete this tutorial, you need the following tools and configurations:

Running service accounts for Kubernetes, Atlas Kubernetes Operator, and Atlas and sufficient privileges to configure them.
You need a running Kubernetes cluster with nodes running processors with the x86-64, AMD64, or ARM64 architecture. For this tutorial, the Kubernetes cluster is https://kube01.internal.io listening on the default port (443).
You can access the Atlas Kubernetes Operator project on GitHub:
- https://github.com/mongodb/mongodb-atlas-kubernetes
To install the Atlas Kubernetes operator using the Atlas CLI, run the following command:
```
atlas kubernetes operator install [options]
```
To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas kubernetes operator install.
Tip
See: Related Links
- Install the Atlas CLI
- Connect to the Atlas CLI
To deploy the Atlas Kubernetes Operator, run the following command. Replace <version> with the latest release number.
```
kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-atlas-kubernetes/<version>/deploy/all-in-one.yaml
```
To register for an Atlas account, see Create an Atlas Account.
API keys. You must create an API key and configure the API Access List.
You need the following public API key, private API key, and the organization ID information to configure Atlas Kubernetes Operator access to Atlas.
- If you want Atlas Kubernetes Operator to create a new Atlas project, Grant Programmatic Access to an Organization. If your organization requires an IP access list for the Atlas Administration API, you must also configure the API access list.
  Important
  You must assign the API key the Organization Project Creator organization role or higher.
- If you want to work with an existing Atlas project, Add Project Access from a Project. If your organization requires an IP access list for the Atlas Administration API, you must also configure the API access list.
  Important
  You must assign the API key the Project Owner project role.
A secret storage vault. This tutorial uses HashiCorp Vault, which is a third-party service for secret storage, running at https://vault.internal.io.
You can use other secret storage vaults with Atlas Kubernetes Operator as needed, including Cloud KMS from AWS, Azure, and Google.
Internal access only. To prevent exposing sensitive information over the public internet, the following components of the secret storage solution allow internal access only:
- The HashiCorp Vault or KMS service.
- The Kubernetes Cluster APIs service.
- The internal network. This tutorial uses internal.io.
While the previous components allow internal access only, they allow access to each other and allow access to anyone within your team or organization. This is a best practice for security.
Public Certificate Authorities (CAs). You can use public CAs to avoid managing and distributing custom CA root certificates.
You can automate CA cert management and renewal by using any of the following tools:
In this tutorial:
- All internal.io HTTPs services are internal addresses, but their HTTPS sites hold automatically renewed certificates signed by a public CA.
- No mutual TLS (mTLS) is required for this integration because it performs only server-side HTTPS validation.
- Clients can trust these service certificates without extra certificate provisioning.

Procedure

Follow these steps to configure secret storage for Atlas Kubernetes Operator.

Install the secret provisioning tool in the target cluster.

Select a secret provisioning tool to install it.

To use External Secrets Operator as the secret provisioning tool:

Run the following commands to install External Secrets Operator with Helm Charts and start the service:
```
helm repo add external-secrets https://charts.external-secrets.io
```
```
helm upgrade -i --atomic \
-n external-secrets --create-namespace --set extraArgs.loglevel=debug \
external-secrets external-secrets/external-secrets`
```
Ensure External Secrets runs successfully:
kubectl get pod -n external-secrets -l app.kubernetes.io/name=external-secrets
NAME READY STATUS RESTARTS AGE external-secrets-5779d5d6f6-2lhgd 1/1 Running 0 70s

To use Secrets Store CSI Driver as the secret provisioning tool, follow these steps:

Run the following command to install Secrets Store CSI Driver with Helm Charts and start the service:
```
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
```
```
helm upgrade -i --atomic --set syncSecret.enabled=true \
  -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
```
Run the following command to install the Secrets Store CSI HashiCorp Vault plugin with Helm Charts. You don't need to install the HashiCorp Vault server or secrets injector.
```
helm install vault hashicorp/vault \
--set "server.enabled=false" --set "injector.enabled=false" \
--set "csi.enabled=true"
```
Ensure Secrets Store CSI runs successfully:
kubectl get pod -n kube-system -l app.kubernetes.io/name=secrets-store-csi-driver
NAME READY STATUS RESTARTS AGE csi-secrets-store-secrets-store-csi-driver-6dcm8 3/3 Running 0 2m2s
Ensure the HashiCorp Vault CSI provider runs successfully:
kubectl get pods -l app.kubernetes.io/name=vault-csi-provider
NAME READY STATUS RESTARTS AGE vault-csi-provider-j7xbr 2/2 Running 0 5m39s

Set up authentication to access secrets.

To set up OIDC JWT and Kubernetes authentication:

Run the following command to enable OIDC JWT authentication for the mount path. If you set up several Kubernetes clusters, you must enable OIDC JWT authentication for each cluster's mount path.
```
vault auth enable -path=jwt-kube01 jwt
```
Run the following command to allow unauthenticated access to the OIDC discovery URLs on the kube01.internal.io cluster:
```
$ kubectl create clusterrolebinding oidc-reviewer  \
  --clusterrole=system:service-account-issuer-discovery \
  --group=system:unauthenticated
```
Run the following command to direct HashiCorp Vault to trust the cluster. The issuer for the cluster at https://kube01.internal.io/ must match the URL in the OIDC discovery document at .well-known/openid-configuration.
```
curl https://kube01.internal.io/.well-known/openid-configuration | jq .issuer
https://kube01.internal.io/
vault write auth/jwt-kube01/config jwks_url="https://kube01.internal.io/openid/v1/jwks"
```
Create the policies that allow access to the secrets you want to expose to the cluster.
The following example creates the external-secrets policy that you specify in later steps.
```
echo external-secrets-policy.hcl
path "secret/data/kube01/external-secrets/*" {
  capabilities = ["read"]
}
```

Run the command to create a HashiCorp Vault role and bind a Kubernetes service account to the role, which restricts the access of the service account within the HashiCorp Vault.

The following command creates the jwt-kube01-system role of the JWT (OIDC) type for the vault audience. The command specifies the sub user claim, and the default Kubernetes service account in the mongodb-atlas-system namespace as the bound subject. This command ties the role to the external-secrets policy set of permissions within HashiCorp Vault.

vault write auth/jwt-kube01/role/jwt-kube01-system role_type="jwt" bound_audiences=vault \
  user_claim="sub" bound_subject="system:serviceaccount:mongodb-atlas-system:default" \
  policies="external-secrets"

Run the command to create another role for External Secrets Operator to access the default namespace.
The following command creates the jwt-kube01-default role of the JWT (OIDC) type for the vault audience. This command specifies the sub user claim, and the default Kubernetes service account in the default namespace as the bound subject . This command ties the role to the external-secrets policy set of permissions within HashiCorp Vault.
```
vault write auth/jwt-kube01/role/jwt-kube01-default role_type="jwt" bound_audiences=vault \
  user_claim="sub" bound_subject="system:serviceaccount:default:default" \
  policies="external-secrets"
```
Ensure OIDC authentication runs successfully for the system service account:
```
export TEST_JWT_TOKEN=$(kubectl -n mongodb-atlas-system create token default --audience "vault")
vault write auth/jwt-kube01/login role=jwt-kube01-system jwt=$(TEST_JWT_TOKEN)
```
Atlas Kubernetes Operator returns your OIDC JWT credentials for the system service account.
Ensure OIDC authentication runs successfully for the default service account:
```
export TEST_JWT_TOKEN=$(kubectl -n default create token default --audience "vault")
vault write auth/jwt-kube01/login role=jwt-kube01-default jwt=$(TEST_JWT_TOKEN)
```
Atlas Kubernetes Operator returns your OIDC JWT credentials for the default service account.

To set up Kubernetes authentication:

Run the following command to enable Kubernetes authentication for the mount path. If you set up several Kubernetes clusters, you must enable Kubernetes authentication for each cluster's mount path.
```
vault auth enable -path=jwt-kube01 kubernetes
```
When you install with Helm, it automatically configures cluster role binding.

Create policies that allow access to the secrets you want to expose to the cluster.

The following example creates the policy vault-secret used in later steps. It uses vault, a Kubernetes secret bound to the HashiCorp Vault service account that the Helm Chart CSI provider sets up.

echo vault-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: vault
  annotations:
    kubernetes.io/service-account.name: vault
type: kubernetes.io/service-account-token
$ kubectl apply -f vault-secret.yaml

Run the commands to trust the Kubernetes cluster and use Kubernetes authentication:
```
export VAULT_JWT ?= $(shell kubectl get secret/vault -o jsonpath='{.data.token}' |base64 -d)
```
```
vault write auth/k8s-kube01/config kubernetes_host="kube01.internal.io" token_reviewer_jwt=$(VAULT_JWT)
```

Run the command to create a HashiCorp Vault role and bind a Kubernetes service account to the role, which restricts the access of the service account within the HashiCorp Vault.

The following command creates the k8s-kube01 role at auth/k8s-kube01 bound to the Kubernetes service account in the default or mongodb-atlas-system namespaces. This role ties to the secrets-store permissions policy within HashiCorp Vault.

vault write auth/k8s-kube01/role/k8s-kube01-role \
  bound_service_account_names=default,mongodb-atlas-operator \
  bound_service_account_namespaces=default,mongodb-atlas-system \
  policies=secrets-store

The policies must exist and allow access to the secrets. The following example policy allows access to KV v2 secrets under the kube01/secrets-store path:

path "secret/data/kube01/secrets-store/*" {
  capabilities = ["read"]
}

Ensure Kubernetes authentication runs successfully for the system service account:
```
export TEST_JWT_TOKEN=$( kubectl create -n mongodb-atlas-system token mongodb-atlas-operator)
vault write auth/jwt-kube01/login role=jwt-kube01-system jwt=$(TEST_JWT_TOKEN)
```
Ensure Kubernetes authentication runs successfully for the default account:
```
export TEST_JWT_TOKEN=$(kubectl -n default create token default)
vault write auth/jwt-kube01/login role=jwt-kube01-default jwt=$(TEST_JWT_TOKEN)
```

Set up automatic secret provisioning.

To provision secrets with External Secrets Operator:

Deploy the SecretStore custom resource for the default service account in the mongodb-atlas-system namespace:

$ cat external-secrets/vault-system.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-store
  namespace: mongodb-atlas-system
spec:
  provider:
    vault:
      server: "https://vault.internal.io"
      path: "secret"
      version: "v2"
      auth:
        jwt:
          path: "jwt-kube01"
          role: "jwt-kube01-system"
          kubernetesServiceAccountToken:
            expirationSeconds: 600
            serviceAccountRef:
              name: "default"
              audiences:
              - vault
$ kubectl apply -f external-secrets/vault-system.yaml

Deploy the SecretStore custom resource for the default service account in the default namespace:

$ cat external-secrets/vault-default.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-store
  namespace: default
spec:
  provider:
    vault:
      server: "https://vault.internal.io"
      path: "secret"
      version: "v2"
      auth:
        jwt:
          path: "jwt-kube01"
          role: "jwt-role"
          kubernetesServiceAccountToken:
            expirationSeconds: 600
            serviceAccountRef:
              name: "default"
              audiences:
              - vault
$ kubectl apply -f external-secrets/vault-default.yaml

Deploy the ExternalSecret custom resource that references the secret that contains the API key. You must set spec.target.template.metadata.labels to atlas.mongodb.com/type with the value credentials for Atlas Kubernetes Operator to find the secret that External Secrets Operator creates.

Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path secret/data/kube01/external-secrets/atlas-account with the following properties:

orgId
publicApiKey
privateApiKey

$ cat external-secrets/atlas.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: atlas
  namespace: mongodb-atlas-system
spec:
    refreshInterval: "15s"
    secretStoreRef:
      name: vault-store
      kind: SecretStore
    target:
      name: mongodb-atlas-operator-api-key
      template:
        metadata:
          labels:
            atlas.mongodb.com/type: credentials
    data:
    - secretKey: orgId
      remoteRef:
        key: secret/data/kube01/external-secrets/atlas-account
        property: orgId
    - secretKey: publicApiKey
      remoteRef:
        key: secret/data/kube01/external-secrets/atlas-account
        property: publicApiKey
    - secretKey: privateApiKey
      remoteRef:
        key: secret/data/kube01/external-secrets/atlas-account
        property: privateApiKey
  $ kubectl apply -f external-secrets/atlas.yaml

This command creates the Kubernetes secret mongodb-atlas-operator-api-key in the mongodb-atlas-system namespace.

Deploy the ExternalSecret custom resource that references the secret that contains database user credentials. You must set spec.target.template.metadata.labels to atlas.mongodb.com/type with the value credentials for Atlas Kubernetes Operator to find the secret that External Secrets Operator creates.

Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path secret/data/kube01/external-secrets/db-user with the password property.

$ cat external-secrets/dbuser.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: dbuser
  namespace: default
spec:
    refreshInterval: "15s"
    secretStoreRef:
      name: vault-store
      kind: SecretStore
    target:
      name: dbuser-password
      template:
        metadata:
          labels:
            atlas.mongodb.com/type: credentials
    data:
    - secretKey: password
      remoteRef:
        key: secret/data/kube01/external-secrets/db-user
        property: password
  $ kubectl apply -f external-secrets/atlas.yaml

Ensure the secrets return as expected when you run the following commands:
```
$ kubectl get -n mongodb-atlas-system secrets/mongodb-atlas-operator-api-key
```
```
$ kubectl get -n default secrets/dbuser-password
```

To provision secrets with Secrets Store CSI:

Deploy the SecretProviderClass custom resource in the mongodb-atlas-system namespace. You must set spec.secretObjects.labels to atlas.mongodb.com/type with the value credentials for Atlas Kubernetes Operator to find the secret that Secrets Store CSI creates.

Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path secret/data/kube01/external-secrets/atlas-account with the following properties:

orgId
publicApiKey
privateApiKey

$ cat secrets-store/atlas.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: atlas
  namespace: mongodb-atlas-system
spec:
  provider: vault
  secretObjects:
  - data:
    - key: orgId
      objectName: atlas-org
    - key: publicApiKey
      objectName: atlas-pub-key
    - key: privateApiKey
      objectName: atlas-secret-key
    secretName: mongodb-atlas-operator-api-key
    type: Opaque
    labels:
      atlas.mongodb.com/type: credentials
  parameters:
    vaultAddress: https://vault.internal.io
    vaultKubernetesMountPath: k8s-kube01
    roleName: k8s-kube01-role
    objects: |
      - objectName: atlas-org
        secretPath: secret/data/kube01/secrets-store/atlas-account
        secretKey: orgId
      - objectName: atlas-pub-key
        secretPath: secret/data/kube01/secrets-store/atlas-account
        secretKey: publicApiKey
      - objectName: atlas-secret-key
        secretPath: secret/data/kube01/secrets-store/atlas-account
        secretKey: privateApiKey
$ kubectl apply -f secrets-store/atlas.yaml

This command creates the Kubernetes secret mongodb-atlas-operator-api-key in the mongodb-atlas-system namespace.

Run the following command to add a pod that mounts the required secrets:

$ cat secrets-store/ako-patch.yaml
  template:
    spec:
      containers:
      - name: system-secret-placeholder
        image: mongodb/atlas
        command: ["sleep", "infinity"]
        volumeMounts:
        - name: secrets-store-mount
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-mount
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: atlas
$ kubectl apply -f secrets-store/ako-patch.yaml

Deploy the SecretProviderClass custom resource that references the secret that contains database user credentials. You must set spec.target.template.metadata.labels to atlas.mongodb.com/type with the value credentials for Atlas Kubernetes Operator to find the secret that Secrets Store CSI creates.

Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path secret/data/kube01/external-secrets/db-user with the password property.

$ cat external-secrets/dbuser.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretProviderClass
metadata:
  name: dbuser
  namespace: default
spec:
  provider: vault
  secretObjects:
  - data:
    - key: password
      objectName: dbuser
    secretName: dbuser-password
    type: Opaque
    labels:
      atlas.mongodb.com/type: credentials
 parameters:
    vaultAddress: https://vault.internal.io
    vaultKubernetesMountPath: k8s-kube01
    roleName: k8s-kube01-role
    objects: |
      - objectName: "dbuser"
        secretPath: "secret/data/kube01/secrets-store/db-user"
        secretKey: "password"
$ kubectl apply -f secrets-store/dbuser.yaml

Run the following command to create the secret-placeholder sentinel pod, which ensures the Secrets Store CSI driver fetches the dbuser credentials and sync them to Kubernetes:

$ cat secrets-store/placeholder.yaml
kind: Pod
apiVersion: v1
metadata:
  name: secret-placeholder
spec:
  containers:
  - image: mongodb/atlas
    command: ["sleep", "infinity"]
    name: secret-placeholder
    volumeMounts:
    - name: secrets-store-mount
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-mount
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: dbuser
$ kubectl apply -f secrets-store/placeholder.yaml

Ensure the secrets return as expected when you run the following commands:
```
$ kubectl get -n mongodb-atlas-system secrets/mongodb-atlas-operator-api-key
```
```
$ kubectl get -n default secrets/dbuser-password
```

Deploy Atlas Kubernetes Operator custom resources.

You can now deploy Atlas Kubernetes Operator custom resources. Atlas Kubernetes Operator authenticates with the Kubernetes secrets that reference your HashiCorp Vault. Adjust the timeout values as needed for your deployments.

kubectl apply -f ako/project.yaml
kubectl apply -f ako/deployment.yaml
kubectl apply -f ako/user.yaml
kubectl wait --for=condition=ready atlasdeployment/serverless-deployment --timeout=10m
kubectl wait --for=condition=ready atlasdatabaseuser/user --timeout=10m

To learn more about these custom resources, see Custom Resources.

Test your Atlas Kubernetes Operator deployment.

To test your Atlas Kubernetes Operator deployment, run the following command:

export ATLAS_DEPLOYMENT_CONN_STR=$(kubectl get secrets/test-atlas-operator-project-test-serverless-deployment-dbuser -o jsonpath='{.data.connectionStringStandardSrv}' |base64 -d)
mongosh $(ATLAS_DEPLOYMENT_CONN_STR) --apiVersion 1 --eval "show dbs"

Atlas Kubernetes Operator returns a list of your database deployments.

Back

Atlas Access

Network Peering