Integrate Azure Key Vault with MongoDB Client-Side Field Level Encryption
Rate this tutorial
When implementing MongoDB’s client-side field level encryption (CSFLE), you’ll find yourself making an important decision: Where do I store my customer master key? In another tutorial, I guided readers through the basics of CSFLE by using a locally-generated and stored master key. While this works for educational and local development purposes, it isn’t suitable for production! In this tutorial, we’ll see how to use Azure Key Vault to generate and securely store our master key.
- A MongoDB Atlas cluster running MongoDB 4.2 (or later) OR MongoDB 4.2 Enterprise Server (or later)—required for automatic encryption
- MongoDB .NET Driver 2.13.0 (or later)
- An Azure Account with an active subscription and the same permissions as those found in any of these Azure AD roles (only one is needed):
Prepare Azure Infrastructure
Configure your Client Application to use Azure Key Vault and CSFLE
In order to establish a trust relationship between our application and the Microsoft identity platform, we first need to register it.
- If you have access to multiple tenants, in the top menu, use the “Directory + subscription filter” to select the tenant in which you want to register an application.
- In the main search bar, search for and select “Azure Active Directory.”
- On the left-hand navigation menu, find the Manage section and select “App registrations,” then “+ New registration.”
- Enter a display name for your application. You can change the display name at any time and multiple app registrations can share the same name. The app registration's automatically generated Application (client) ID, not its display name, uniquely identifies your app within the identity platform.
- Specify who can use the application, sometimes called its sign-in audience. For this tutorial, I’ve selected “Accounts in this organizational directory only (Default Directory only - Single tenant).” This only allows users that are in my current tenant access to my application.
- Click “Register.” Once the initial app registration is complete, copy the Directory (tenant) ID and Application (client) ID as we’ll need them later on.
- Find the linked application under “Managed application in local directory” and click on it.
- Once brought to the “Properties” page, also copy the “Object ID” as we’ll need this too.
Once your application is registered, we’ll need to create a client secret for it. This will be required when authenticating to the Key Vault we’ll be creating soon.
- On the overview of your newly registered application, click on “Add a certificate or secret”:
- Under “Client secrets,” click “+ New client secret.”
- Enter a short description for this client secret and leave the default “Expires” setting of 6 months.
- Click “Ad." Once the client secret is created, be sure to copy the secret’s “Value” as we’ll need it later. It’s also worth mentioning that once you leave this page, the secret value is never displayed again, so be sure to record it at least once!
Next up, an Azure Key Vault! We’ll create one so we can securely store our customer master key. We’ll be completing these steps via the Azure CLI, so open up your favorite terminal and follow along:
- Sign in to the Azure CLI using the
az login
command. Finish the authentication steps by following the steps displayed in your terminal. - Create a resource group:
1 az group create --name "YOUR-RESOURCE-GROUP-NAME" --location <YOUR-AZURE-REGION> - Create a key vault:
1 az keyvault create --name "YOUR-KEYVAULT-NAME" --resource-group "YOUR-RESOURCE-GROUP-NAME" --location <YOUR-AZURE-REGION>
With a key vault, we can now create our customer master key! This will be stored, managed, and secured by Azure Key Vault.
Create a key and add to our key vault:
1 az keyvault key create --vault-name "YOUR-KEYVAULT-NAME" --name "YOUR-KEY-NAME" --protection software
The
--protection
parameter designates the key protection type. For now, we'll use the software
type. Once the command completes, take note of your key’s "name" as we‘ll need it later!To enable our client application access to our key vault, some permissions need to be granted:
- Give your application the wrapKey and unwrapKey permissions to the keyvault. (For the
--object-id
parameter, paste in the Object ID of the application we registered earlier. This is the Object ID we copied in the last "Register App in Azure Active Directory" step.)1 az keyvault set-policy --name "YOUR-KEYVAULT-NAME" --key-permissions wrapKey unwrapKey --object-id <YOUR-APP-OBJECT-ID> - Upon success, you’ll receive a JSON object. Find and copy the value for the “vaultUri” key. For example, mine is
https://csfle-mdb-demo-vault.vault.azure.net
.
Now that our cloud infrastructure is configured, we can start integrating it into our application. We’ll be referencing the sample repo from our prerequisites for these steps, but feel free to use the portions you need in an existing application.
- If you haven’t cloned the repo yet, do so now!
1 git clone https://github.com/adriennetacke/mongodb-csfle-csharp-demo-azure.git - Navigate to the root directory
mongodb-csfle-csharp-demo-azure
and open theEnvoyMedSys
sample application in Visual Studio. - In the Solution Explorer, find and open the
launchSettings.json
file (Properties
>launchSettings.json
). - Here, you’ll see some scaffolding for some variables. Let’s quickly go over what those are:
MDB_ATLAS_URI
: The connection string to your MongoDB Atlas cluster. This enables us to store our data encryption key, encrypted by Azure Key Vault.AZURE_TENANT_ID
: Identifies the organization of the Azure account.AZURE_CLIENT_ID
: Identifies theclientId
to authenticate your registered application.AZURE_CLIENT_SECRET
: Used to authenticate your registered application.AZURE_KEY_NAME
: Name of the Customer Master Key stored in Azure Key Vault.AZURE_KEYVAULT_ENDPOINT
: URL of the Key Vault. E.g.,yourVaultName.vault.azure.net
.
- Replace all of the placeholders in the
launchSettings.json
file with your own information. Each variable corresponds to a value you were asked to copy and keep track of:AZURE_TENANT_ID
: Directory (tenant) ID.AZURE_CLIENT_ID
: Application (client) ID.AZURE_CLIENT_SECRET
: Secret Value from our client secret.AZURE_KEY_NAME
: Key Name.AZURE_KEYVAULT_ENDPOINT
: Our Key Vault’s vaultUri.
- Save all your files!
Before we run the application, let’s go over what’s happening: When we run our main program, we set the connection to our Atlas cluster and our key vault’s collection namespace. We then instantiate two helper classes: a
KmsKeyHelper
and an AutoEncryptHelper
. The KmsKeyHelper
’s CreateKeyWithAzureKmsProvider()
method is called to generate our encrypted data encryption key. This is then passed to the AutoEncryptHelper
’s EncryptedWriteAndReadAsync()
method to insert a sample document with encrypted fields and properly decrypt it when we need to fetch it. This is all in our Program.cs
file:Program.cs
1 using System; 2 using MongoDB.Driver; 3 4 namespace EnvoyMedSys 5 { 6 public enum KmsKeyLocation 7 { 8 Azure, 9 } 10 11 class Program 12 { 13 public static void Main(string[] args) 14 { 15 var connectionString = Environment.GetEnvironmentVariable("MDB_ATLAS_URI"); 16 var keyVaultNamespace = CollectionNamespace.FromFullName("encryption.__keyVaultTemp"); 17 18 var kmsKeyHelper = new KmsKeyHelper( 19 connectionString: connectionString, 20 keyVaultNamespace: keyVaultNamespace); 21 var autoEncryptHelper = new AutoEncryptHelper( 22 connectionString: connectionString, 23 keyVaultNamespace: keyVaultNamespace); 24 25 var kmsKeyIdBase64 = kmsKeyHelper.CreateKeyWithAzureKmsProvider().GetAwaiter().GetResult(); 26 27 autoEncryptHelper.EncryptedWriteAndReadAsync(kmsKeyIdBase64, KmsKeyLocation.Azure).GetAwaiter().GetResult(); 28 29 Console.ReadKey(); 30 } 31 } 32 }
Taking a look at the
KmsKeyHelper
class, there are a few important methods: the CreateKeyWithAzureKmsProvider()
and GetClientEncryption()
methods. I’ve opted to include comments in the code to make it easier to follow along: KmsKeyHelper.cs
/ CreateKeyWithAzureKmsProvider()
1 public async Task<string> CreateKeyWithAzureKmsProvider() 2 { 3 var kmsProviders = new Dictionary<string, IReadOnlyDictionary<string, object>>(); 4 5 // Pull Azure Key Vault settings from environment variables 6 var azureTenantId = Environment.GetEnvironmentVariable("AZURE_TENANT_ID"); 7 var azureClientId = Environment.GetEnvironmentVariable("AZURE_CLIENT_ID"); 8 var azureClientSecret = Environment.GetEnvironmentVariable("AZURE_CLIENT_SECRET"); 9 var azureIdentityPlatformEndpoint = Environment.GetEnvironmentVariable("AZURE_IDENTIFY_PLATFORM_ENPDOINT"); // Optional, only needed if user is using a non-commercial Azure instance 10 11 // Configure our registered application settings 12 var azureKmsOptions = new Dictionary<string, object> 13 { 14 { "tenantId", azureTenantId }, 15 { "clientId", azureClientId }, 16 { "clientSecret", azureClientSecret }, 17 }; 18 19 if (azureIdentityPlatformEndpoint != null) 20 { 21 azureKmsOptions.Add("identityPlatformEndpoint", azureIdentityPlatformEndpoint); 22 } 23 24 // Specify remote key location; in this case, Azure 25 kmsProviders.Add("azure", azureKmsOptions); 26 27 // Constructs our client encryption settings which 28 // specify which key vault client, key vault namespace, 29 // and KMS providers to use. 30 var clientEncryption = GetClientEncryption(kmsProviders); 31 32 // Set KMS Provider Settings 33 // Client uses these settings to discover the master key 34 var azureKeyName = Environment.GetEnvironmentVariable("AZURE_KEY_NAME"); 35 var azureKeyVaultEndpoint = Environment.GetEnvironmentVariable("AZURE_KEYVAULT_ENDPOINT"); // typically <azureKeyName>.vault.azure.net 36 var azureKeyVersion = Environment.GetEnvironmentVariable("AZURE_KEY_VERSION"); // Optional 37 var dataKeyOptions = new DataKeyOptions( 38 masterKey: new BsonDocument 39 { 40 { "keyName", azureKeyName }, 41 { "keyVaultEndpoint", azureKeyVaultEndpoint }, 42 { "keyVersion", () => azureKeyVersion, azureKeyVersion != null } 43 }); 44 45 // Create Data Encryption Key 46 var dataKeyId = clientEncryption.CreateDataKey("azure", dataKeyOptions, CancellationToken.None); 47 Console.WriteLine($"Azure DataKeyId [UUID]: {dataKeyId}"); 48 49 var dataKeyIdBase64 = Convert.ToBase64String(GuidConverter.ToBytes(dataKeyId, GuidRepresentation.Standard)); 50 Console.WriteLine($"Azure DataKeyId [base64]: {dataKeyIdBase64}"); 51 52 // Optional validation; checks that key was created successfully 53 await ValidateKeyAsync(dataKeyId); 54 55 return dataKeyIdBase64; 56 }
KmsKeyHelper.cs
/ GetClientEncryption()
1 private ClientEncryption GetClientEncryption( 2 Dictionary<string, IReadOnlyDictionary<string, object>> kmsProviders) 3 { 4 // Construct a MongoClient using our Atlas connection string 5 var keyVaultClient = new MongoClient(_mdbConnectionString); 6 7 // Set MongoClient, key vault namespace, and Azure as KMS provider 8 var clientEncryptionOptions = new ClientEncryptionOptions( 9 keyVaultClient: keyVaultClient, 10 keyVaultNamespace: _keyVaultNamespace, 11 kmsProviders: kmsProviders); 12 13 return new ClientEncryption(clientEncryptionOptions); 14 }
With our Azure Key Vault connected and data encryption key encrypted, we’re ready to insert some data into our Atlas cluster! This is where the
AutoEncryptHelper
class comes in. The important method to note here is the EncryptedReadAndWrite()
method: AutoEncryptHelper.cs
/ EncryptedReadAndWrite()
1 public async Task EncryptedWriteAndReadAsync(string keyIdBase64, KmsKeyLocation kmsKeyLocation) 2 { 3 // Construct a JSON Schema 4 var schema = JsonSchemaCreator.CreateJsonSchema(keyIdBase64); 5 6 // Construct an auto-encrypting client 7 var autoEncryptingClient = CreateAutoEncryptingClient( 8 kmsKeyLocation, 9 _keyVaultNamespace, 10 schema); 11 12 // Set our working database and collection to medicalRecords.patientData 13 var collection = autoEncryptingClient 14 .GetDatabase(_medicalRecordsNamespace.DatabaseNamespace.DatabaseName) 15 .GetCollection<BsonDocument>(_medicalRecordsNamespace.CollectionName); 16 17 var ssnQuery = Builders<BsonDocument>.Filter.Eq("ssn", __sampleSsnValue); 18 19 // Upsert (update if found, otherwise create it) a document into the collection 20 var medicalRecordUpdateResult = await collection 21 .UpdateOneAsync(ssnQuery, new BsonDocument("$set", __sampleDocFields), new UpdateOptions() { IsUpsert = true }); 22 23 if (!medicalRecordUpdateResult.UpsertedId.IsBsonNull) 24 { 25 Console.WriteLine("Successfully upserted the sample document!"); 26 } 27 28 // Query by SSN field with auto-encrypting client 29 var result = await collection.Find(ssnQuery).SingleAsync(); 30 31 // Proper result in console should show decrypted, human-readable document 32 Console.WriteLine($"Encrypted client query by the SSN (deterministically-encrypted) field:\n {result}\n"); 33 }
Now that we know what’s going on, run your application!
If all goes well, your console will print out two
DataKeyIds
(UUID and base64) and a document that resembles the following: Sample Result Document (using my information)
1 { 2 _id:UUID('ab382f3e-bc79-4086-8418-836a877efff3'), 3 keyMaterial:Binary('tvehP03XhUsztKr69lxlaGjiPhsNPjy6xLhNOLTpe4pYMeGjMIwvvZkzrwLRCHdaB3vqi9KKe6/P5xvjwlVHacQ1z9oFIwFbp9nk...', 0), 4 creationDate:2021-08-24T05:01:34.369+00:00, 5 updateDate:2021-08-24T05:01:34.369+00:00, 6 status:0, 7 masterKey:Object, 8 provider:"azure", 9 keyVaultEndpoint:"csfle-mdb-demo-vault.vault.azure.net", 10 keyName:"MainKey" 11 }
Here’s what my console output looks like, for reference:
Seeing this is great news! A lot of things have just happened, and all of them are good:
- Our application properly authenticated to our Azure Key Vault.
- A properly generated data encryption key was created by our client application.
- The data encryption key was properly encrypted by our customer master key that’s securely stored in Azure Key Vault.
- The encrypted data encryption key was returned to our application and stored in our MongoDB Atlas cluster.
Here’s the same process in a workflow:
After a few more moments, and upon success, you’ll see a “Successfully upserted the sample document!” message, followed by the properly decrypted results of a test query. Again, here’s my console output for reference:
This means our sample document was properly encrypted, inserted into our
patientData
collection, queried with our auto-encrypting client by SSN, and had all relevant fields correctly decrypted before returning them to our console. How neat! And just because I’m a little paranoid, we can double-check that our data has actually been encrypted. If you log into your Atlas cluster and navigate to the
patientData
collection, you’ll see that our documents’ sensitive fields are all illegible:That wasn’t so bad, right? Let's see what we've accomplished! This tutorial walked you through:
- Registering an App in Azure Active Directory.
- Creating a Client Secret.
- Creating an Azure Key Vault.
- Creating and Adding a Key to your Key Vault.
- Granting Application Permissions to Key Vault.
- Integrating Azure Key Vault into Your Client Application.
- The Results: What You Get After Integrating Azure Key Vault with MongoDB CSFLE.
By using a remote key management system like Azure Key Vault, you gain access to many benefits over using a local filesystem. The most important of these is the secure storage of the key, reduced risk of access permission issues, and easier portability!
For more information, check out this helpful list of resources I used while preparing this tutorial:
And if you have any questions or need some additional help, be sure to check us out on the MongoDB Community Forums and start a topic!
A whole community of MongoDB engineers (including the DevRel team) and fellow developers are sure to help!