Benjamin Flast

Product Manager, Data Lake

Have you ever been looking for something but don’t quite have the words? Do you remember some characteristics of a movie but can’t remember the name? Have you ever been trying to get another sweatshirt just like the one you had back in the day but don’t know how to search for it? Are you using large language models, but they only know information up until 2021? Do you want it to get with the times?! Well then, vector search may be just what you’re looking for.

## What is vector search?

Vector search is a capability that allows you to do semantic search where you are searching data based on meaning. This technique employs machine learning models, often called encoders, to transform text, audio, images, or other types of data into high-dimensional vectors. These vectors capture the semantic meaning of the data, which can then be searched through to find similar content based on vectors being “near” one another in a high-dimensional space. This can be a great compliment to traditional keyword-based search techniques but is also seeing an explosion of excitement because of its relevance to augment the capabilities of large language models (LLMs) by providing ground truth outside of what the LLMs “know.” In search use cases, this allows you to find relevant results even when the exact wording isn't known. This technique can be useful in a variety of contexts, such as natural language processing and recommendation systems.

Note: As you probably already know, MongoDB Atlas has supported full-text search since 2020, allowing you to do rich text search on your MongoDB data. The core difference between vector search and text search is that vector search queries on meaning instead of explicit text and therefore can also search data beyond just text.

## Benefits of vector search

- Semantic understanding: Rather than searching for exact matches, vector search enables semantic searching. This means that even if the query words aren't present in the index, but the meanings of the phrases are similar, they will still be considered a match.
- Scalable: Vector search can be done on large datasets, making it perfect for use cases where you have a lot of data.
- Flexible: Different types of data, including text but also unstructured data like audio and images, can be semantically searched.

## Benefits of vector search with MongoDB

- Efficiency: By storing the vectors together with the original data, you avoid the need to sync data between your application database and your vector store at both query and write time.
- Consistency: Storing the vectors with the data ensures that the vectors are always associated with the correct data. This can be important in situations where the vector generation process might change over time. By storing the vectors, you can be sure that you always have the correct vector for a given piece of data.
- Simplicity: Storing vectors with the data simplifies the overall architecture of your application. You don't need to maintain a separate service or database for the vectors, reducing the complexity and potential points of failure in your system.
- Scalability: With the power of MongoDB Atlas, vector search on MongoDB scales horizontally and vertically, allowing you to power the most demanding workloads.

> Want to experience Vector Search with MongoDB quick and easy? Check out this [automated demo on GitHub](https://github.com/phil2211/vector-search-demo) as you walk through the tutorial.

## Set up a MongoDB Atlas cluster

Now, let's get into setting up a MongoDB Atlas cluster, which we will use to store our embeddings.

**Step 1: Create an account**

To create a MongoDB Atlas cluster, first, you need to create a MongoDB Atlas account if you don't already have one. Visit the [MongoDB Atlas](https://www.mongodb.com/atlas/database) website and click on “Register.”

**Step 2: Build a new cluster**

After creating an account, you'll be directed to the MongoDB Atlas dashboard. You can create a cluster in the dashboard, or using our public API, CLI, or Terraform provider. To do this in the dashboard, click on “Create Cluster,” and then choose the shared clusters option. We suggest creating an M0 tier cluster.

If you need help, [check out our tutorial](https://www.mongodb.com/developer/products/mongodb/5-different-ways-deploy-free-database-mongodb-atlas/) demonstrating the deployment of Atlas using various strategies.

**Step 3: Create your collections**

Now, we’re going to create your collections in the cluster so that we can insert our data. They need to be created now so that you can create an Atlas trigger that will target them.

For this tutorial, you can create your own collection if you have data to use. If you’d like to use our [sample data](https://www.mongodb.com/docs/atlas/sample-data/), you need to first create an empty collection in the cluster so that we can set up the trigger to embed them as they are inserted. Go ahead and create a “sample_mflix” database and “movies” collection now using the UI, if you’d like to use our sample data.

## Setting up an Atlas trigger

We will create an Atlas trigger to call the [OpenAI API](https://platform.openai.com/overview) whenever a new document is inserted into the cluster.

To proceed to the next step using OpenAI, you need to have set up an account on OpenAI and created an API key.

If you don't want to embed all the data in the collection you can use the "sample_mflix.embedded_movies" collection for this which already has embeddings generated by Open AI, and just create an index and run Vector Search queries.

**Step 1: Create a trigger**

To create a trigger, navigate to the “Triggers” section in the MongoDB Atlas dashboard, and click on “Add Trigger.”

![Databases Menu on Atlas Cluster Deployment Tab Arrow Pointing at Triggers](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4908b02bd2ec6000/64931b87616b14211115f06f/1.png )

![Atlas Triggers Menu Arrow Pointing at “Add Trigger”](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt133f7eb1933d9405/64931c3648e15a65ae46c18f/2.png)

**Step 2: Set up secrets and values for your OpenAI credentials**

Go over to “App Services” and select your “Triggers” application.

![App Services Main Page](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb9965f5a7cdece34/649221fbb9a0760f04928276/image13.png)

Click “Values.”

![Triggers App Page in App Services Pointing at “Values” Tab](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5e3daa9f6d369a05/64931c66207042ceb986e0ff/4.png)

You’ll need your OpenAI API key, which you can [create on their website](https://platform.openai.com/overview):

![Open AI Platform Page Arrow Pointing at View API keys in Menu](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt35c951800e86f9b2/64931c80f7411b27b437e181/5.png)

![Open AI API Keys Page Arrow Pointing at “Create new secret key”](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt621df034d220bbda/64931c8ec32681211955e662/6.png)

Create a new Value

![Create New Value button in App Services Value Tab](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta256566d532d8ca0/64922247b9a076fe5692827e/image2.png)

Select “Secret” and then paste in your OpenAI API key.

![Setting a name for a Secret in App Services Values](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf258009ded502970/649222612429af81a4f49508/image6.png)

Then, create another value — this time, a “Value” — and link it to your secret. This is how you will securely reference this API key in your trigger.

![Creating a Value and linking it to a Secret in Values in App Services](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltdaac39b34ab412f0/64922279ea50bc3e05bea47f/image1.png)

Now, you can go back to the “Data Services” tab and into the triggers menu. If the trigger you created earlier does not show up, just add a new trigger. It will be able to utilize the values you set up in App Services earlier.

**Step 3: Configure the trigger**

Select the “Database” type for your trigger. Then, link the source cluster and set the “Trigger Source Details” to be the Database and Collection to watch for changes. For this tutorial, we are using the “sample_mflix” database and the “movies” collection. Set the Operation Type to 'Insert' ‘Update’ ‘Replace’ operation. Check the “Full Document” flag and in the Event Type, choose “Function.”

In the Function Editor, use the code snippet below, replacing DB Name and Collection Name with the database and collection names you’d like to use, respectively.

This trigger will see when a new document is created or updated in this collection. Once that happens, it will make a call to the OpenAI API to create an embedding of the desired field, and then it will insert that vector embedding into the document with a new field name.

```javascript
exports = async function(changeEvent) {
    // Get the full document from the change event.
    const doc = changeEvent.fullDocument;

    // Define the OpenAI API url and key.
    const url = 'https://api.openai.com/v1/embeddings';
    // Use the name you gave the value of your API key in the "Values" utility inside of App Services
    const openai_key = context.values.get("openAI_value");
    try {
        console.log(`Processing document with id: ${doc._id}`);

        // Call OpenAI API to get the embeddings.
        let response = await context.http.post({
            url: url,
             headers: {
                'Authorization': [`Bearer ${openai_key}`],
                'Content-Type': ['application/json']
            },
            body: JSON.stringify({
                // The field inside your document that contains the data to embed, here it is the "plot" field from the sample movie data.
                input: doc.plot,
                model: "text-embedding-ada-002"
            })
        });

        // Parse the JSON response
        let responseData = EJSON.parse(response.body.text());

        // Check the response status.
        if(response.statusCode === 200) {
            console.log("Successfully received embedding.");

            const embedding = responseData.data[0].embedding;

            // Use the name of your MongoDB Atlas Cluster
            const collection = context.services.get("<CLUSTER_NAME>").db("sample_mflix").collection("movies");

            // Update the document in MongoDB.
            const result = await collection.updateOne(
                { _id: doc._id },
                // The name of the new field you'd like to contain your embeddings.
                { $set: { plot_embedding: embedding }}
            );

            if(result.modifiedCount === 1) {
                console.log("Successfully updated the document.");
            } else {
                console.log("Failed to update the document.");
            }
        } else {
            console.log(`Failed to receive embedding. Status code: ${response.statusCode}`);
        }

    } catch(err) {
        console.error(err);
    }
};
```

## Configure index

Now, head over to Atlas Search and create an index. Use the JSON index definition and insert the following, replacing the embedding field name with the field of your choice. If you are using the sample_mflix database, it should be “plot_embedding”, and give it a name. I’ve used “moviesPlotIndex” for my setup with the sample data.

First, click the “atlas search” tab on your cluster

![Databases Page for a Cluster with an arrow pointing at the Search tab][1]

Then, click “Create Search Index.”

![Search tab within the Cluster page with an arrow pointing at Create Search Index](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blteb040dca217d33aa/65a1bae143e8cb9958a4c0e0/Screenshot_2024-01-12_at_4.50.00_PM.png)

Create “JSON Editor.”

![Search tab “Create Search Index” experience with arrow pointing at “JSON Editor”](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt211855d86af82150/65a19c95a0eb302ea45007b6/Screenshot_2024-01-11_at_10.12.55_AM.png)

Then, select your Database and Collection on the left and a drop in the code snippet below for your index definition.

![Search Index Configuration JSON Editor with arrows pointing at the Database and Collection name as well as the JSON editor](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltec7a018c31344563/65a1b23dd838b3a2bc9d90a6/Screenshot_2024-01-12_at_4.41.47_PM.png)

```json
{
  "type": "vectorSearch",
  "fields": [{
    "path": "plot_embedding",
    "dimensions": 1536,
    "similarity": "cosine",
    "type": "vector"
  }]
}
```

## Insert your data

Now, you need to insert your data. As your data is inserted, it will be embedded using the script and then indexed using the KNN index we just set.

If you have your own data, you can insert it now using something like [MongoImports](https://www.mongodb.com/docs/database-tools/mongoimport/).

If you’re going to use the sample movie data, you can just go to the cluster, click the … menu, and load the sample data. If everything has been set up correctly, the sample_mflix database and movies collections will have the plot embeddings created on the “plot” field and added to a new “plot_embeddings” field.

## Now, to query your data with JavaScript

Once the documents in your collection have their embeddings generated, you can perform a query. But because this is using vector search, your query needs to be transformed into an embedding. This is an example script of how you could add a function to get both an embedding of the query and a function to use that embedding inside of your application. 

```javascript
const axios = require('axios');
const MongoClient = require('mongodb').MongoClient;

async function getEmbedding(query) {
    // Define the OpenAI API url and key.
    const url = 'https://api.openai.com/v1/embeddings';
    const openai_key = 'your_openai_key'; // Replace with your OpenAI key.
    
    // Call OpenAI API to get the embeddings.
    let response = await axios.post(url, {
        input: query,
        model: "text-embedding-ada-002"
    }, {
        headers: {
            'Authorization': `Bearer ${openai_key}`,
            'Content-Type': 'application/json'
        }
    });
    
    if(response.status === 200) {
        return response.data.data[0].embedding;
    } else {
        throw new Error(`Failed to get embedding. Status code: ${response.status}`);
    }
}

async function findSimilarDocuments(embedding) {
    const url = 'your_mongodb_url'; // Replace with your MongoDB url.
    const client = new MongoClient(url);
    
    try {
        await client.connect();
        
        const db = client.db('<DB_NAME>'); // Replace with your database name.
        const collection = db.collection('<COLLECTION_NAME>'); // Replace with your collection name.
        
        // Query for similar documents.
        const documents = await collection.aggregate([
  {"$vectorSearch": {
    "queryVector": embedding,
    "path": "plot_embedding",
    "numCandidates": 100,
    "limit": 5,
    "index": "moviesPlotIndex",
      }}
]).toArray();
        
        return documents;
    } finally {
        await client.close();
    }
}

async function main() {
    const query = 'your_query'; // Replace with your query.
    
    try {
        const embedding = await getEmbedding(query);
        const documents = await findSimilarDocuments(embedding);
        
        console.log(documents);
    } catch(err) {
        console.error(err);
    }
}

main();
```

This script first transforms your query into an embedding using the OpenAI API, and then queries your MongoDB cluster for documents with similar embeddings.

> Support for the '$vectorSearch' aggregation pipeline stage is available with MongoDB Atlas 6.0.11 and 7.0.2.

Remember to replace 'your_openai_key', 'your_mongodb_url', 'your_query', ‘<DB_NAME>’, and ‘<COLLECTION_NAME>’ with your actual OpenAI key, MongoDB URL, query, database name, and collection name, respectively.

And that's it! You've successfully set up a MongoDB Atlas cluster and Atlas trigger which calls the OpenAI API to embed documents when they get inserted into the cluster, and you’ve performed a vector search query. 

> If you prefer learning by watching, check out the video version of this article!

:youtube[]{vid=1ZIYVNvRVsY}


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb8503d464e800c36/65a1bba2d6cafb29fbf758da/Screenshot_2024-01-12_at_4.45.14_PM.png

Learn how to get started with Vector Search on MongoDB while leveraging the OpenAI.

How to Do Semantic Search in MongoDB Using Atlas Vector Search

For those of you not familiar with [Parquet](https://parquet.apache.org/), it’s an amazing file format that does a lot of the heavy lifting to ensure blazing fast query performance on data stored in files. This is a popular file format in the Data Warehouse and Data Lake space as well as for a variety of machine learning tasks.

One thing we frequently see users struggle with is getting NoSQL data into Parquet as it is a columnar format. Historically, you would have to write some custom code to get the data out of the database, transform it into an appropriate structure, and then probably utilize a third-party library to write it to Parquet. Fortunately, with MongoDB Atlas Data Federation's $out to cloud object storage - Amazon S3 or Microsoft Azure Blob Storage, you can now convert MongoDB Data into Parquet with little effort.

In this blog post, I’m going to walk you through the steps necessary to write data from your Atlas Cluster directly to cloud object storage in the Parquet format and then finish up by reviewing some things to keep in mind when using Parquet with NoSQL data. I’m going to use a sample data set that contains taxi ride data from New York City.

## Prerequisites

In order to follow along with this tutorial yourself, you will need the following:
An Atlas cluster with some data in it. (It can be the sample data.)
An AWS account with privileges to create IAM Roles and cloud object storage buckets (to give us access to write data to your cloud object storage bucket).

## Create a Federated Database Instance and Connect to cloud object storage

The first thing you'll need to do is navigate to the "Data Federation" tab on the left hand side of your Atlas Dashboard and then click “set up manually” in the "create new federated database" dropdown in the top right corner of the UI.

![Data Federation](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc94edfdcc342265b/649c786b84a4c7eb46d8db62/data-federation.png)

Then, you need to connect your cloud object storage bucket to your Federated Database Instance. This is where we will write the Parquet files. The setup wizard should guide you through this pretty quickly but you will need access to your credentials for AWS. (Be sure to give Atlas Data Federation “Read and Write” access to the bucket so it can write the Parquet files there.)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_19_125037_440fb93c33.jpg)

Once you’ve connected your cloud object storage bucket, we’re going to create a simple data source to query the data in cloud object storage so we can verify we’ve written the data to cloud object storage at the end of this tutorial. Our new setup tool makes it easier than ever to configure your Federated Database Instance to take advantage of the partitioning of data in cloud object storage. Partitioning allows us to only select the relevant data to process in order to satisfy your query. (I’ve put a sample file in there for this test that will fit how we’re going to partition the data by \_cab\_type).

``` bash
mongoimport --uri mongodb+srv://<USERNAME>:<PASSWORD>@<MONGODB_URI>/<DATABASE> --collection <COLLECTION> --type json --file <FILENAME>
```

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_20_164216_22544c3908.jpg)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_20_163429_6d7a53327c.jpg)

## Connect Your Federated Database Instance to an Atlas Cluster

Now we’re going to connect our Atlas cluster, so we can write data from it into the Parquet files. This involves picking the cluster from a list of clusters in your Atlas project and then selecting the databases and collections you’d like to create Data Sources from and dragging them into your Federated Database Instance.

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_20_164727_7f084fa42e.jpg)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_20_164926_350b9a7a74.jpg)

## $out to cloud object storage in Parquet

Now we’re going to connect to our Federated Database Instance using the mongo shell and execute the following command. This is going to do quite a few things, so I’m going to explain the important ones.
- First, you can use the ‘filename’ field of the $out stage to have your Federated Database Instance partition files by “_cab_type”, so all the green cabs will go in one set of files and all the yellow cabs will go in another.
- Then in the format, we’re going to specify parquet and determine a maxFileSize and maxRowGroupSize.
   -- maxFileSize is going to determine the maximum size each partition will be.
   -- maxRowGroupSize is going to determine how records are grouped inside of the Parquet file in “row groups” which will impact performance querying your Parquet files, similarly to file size.
- Lastly, we’re using a special Atlas Data Federation aggregation “background: true” which simply tells the Federated Database Instance to keep executing the query even if the client disconnects. (This is handy for long running queries or environments where your network connection is not stable.)

``` js
db.getSiblingDB("clusterData").getCollection("trips").aggregate([
    {
        "$out" : {
            "s3" : {
                "bucket" : "ben.flast",
                "region" : "us-east-1",
                "filename" : {
                    "$concat" : [
                        "taxi-trips/",
                        "$_cab_type",
                        "/"
                    ]
                },
                "format" : {
                    "name" : "parquet",
                    "maxFileSize" : "10GB",
                    "maxRowGroupSize" : "100MB"
                }
            }
        }
    }
], {
    background: true
})
```

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/aws_s3_interface_f21ce56fa9.png)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/aws_s3_yellow_cab_folder_55e21a7e89.png)

## Blazing Fast Queries on Parquet Files

Now, to give you some idea of the potential performance improvements for Object Store Data you can see, I’ve written three sets of data, each with 10 million documents: one in Parquet, one in uncompressed JSON, and another in compressed JSON. And I ran a count command on each of them with the following results.

*db.trips.count()*
10,000,000

| Type | Data Size (GB) | Count Command Latency (Seconds) |
| ---- | -------------- | ------------------------------- |
| JSON (Uncompressed) | \~16.1 | 297.182 |
| JSON (Compressed) | \~1.1 | 78.070 |
| Parquet | \~1.02 | 1.596 |

## In Review

So, what have we done and what have we learned?
- We saw how quickly and easily you can create a Federated Database Instance in MongoDB Atlas.
- We connected an Atlas cluster to our Federated Database Instance.
- We used our Federated Database Instance to write Atlas cluster data to cloud object storage in Parquet format.
- We demonstrated how fast and space-efficient Parquet is when compared to JSON.

## A Couple of Things to Remember About Atlas Data Federation

- Parquet is a super fast columnar format that can be read and written with Atlas Data Federation.
- Atlas Data Federation takes advantage of various pieces of metadata contained in Parquet files, not just the maxRowGroupSize. For instance, if your first stage in an aggregation pipeline was $project: {fieldA: 1, filedB: 1}, we would only read the two columns from the Parquet file which results in faster performance and lower costs as we are scanning less data.
- Atlas Data Federation writes Parquet files flexibly so if you have polymorphic data, we will create union columns so you can have ‘Column A - String’ and ‘Column A - Int’. Atlas Data Federation will read union columns back in as one field but other tools may not handle union types. So if you’re going to be using these Parquet files with other tools, you should transform your data before the $out stage to ensure no union columns.
- Atlas Data Federation will also write files with different schemas if it encounters data with varying schemas throughout the aggregation. It can handle different schemas across files in one collection, but other tools may require a consistent schema across files. So if you’re going to be using these Parquet files with other tools, you should do a $project with $convert’s before the $out stage to ensure a consistent schema across generated files.
- Parquet is a great format for your MongoDB data when you need to use columnar oriented tools like Tableau for visualizations or machine learning frameworks that use data frames. Parquet can be quickly and easily converted into Pandas data frames in Python.

Learn how to transform MongoDB data to Parquet with Atlas Data Federation.

How to Get MongoDB Data into Parquet in 10 Seconds or Less

A common pattern for users of MongoDB running multi-tenant services is to model your data by splitting your different customers into different databases. This is an excellent strategy for keeping your various customers’ data separate from one another, as well as helpful for scaling in the future. But one downside to this strategy is that you can end up struggling to get a holistic view of your data across all of your customers. There are many ways to mitigate this challenge, and one of the primary ones is to copy and transform your data into another storage solution. However, this can lead to some unfortunate compromises. For example, you are now paying more to store your data twice. You now need to manage the copy and transformation process, which can become onerous as you add more customers. And lastly, and perhaps most importantly, you are now looking at a delayed state of your data.

To solve these exact challenges, we’re thrilled to announce two features that will completely transform how you use your cluster data and the ease with which you can remodel it. The first feature is called Provenance. This functionality allows you to tell Data Federation to inject fields into your documents during query time that indicate where they are coming from. For example, you can add the source collection on the Atlas cluster when federating across clusters or you can add the path from your AWS S3 bucket where the data is being read. The great thing is  that you can now also query on these fields to only get data from the source of your choice!

The other feature we’re adding is a bit nuanced, and we are calling it “globbing.” For those of you familiar with Atlas Data Federation, you probably know about our “wildcard collections.”  This functionality allows you to generate collection names based on the collections that exist in your underlying Atlas clusters or based on sections of paths to your files in S3. This is a handy feature to avoid having to explicitly define everything in your storage configuration. “Globbing” is somewhat similar, except that instead of dynamically generating new collections for each collection in your cluster, it will dynamically merge collections to give you a “global” view of your data automatically. To help illustrate this, I’m going to walk you through an example.

Imagine you are running a successful travel agency on top of MongoDB. For various reasons, you have chosen to store your customers data in different databases based on their location. (Maybe you are going to shard based on this and will have different databases in different regions for compliance purposes.)

This has worked well, but now you’d like to query your data based on this information and get a holistic view of your data across geographies in real time (without impacting your operational workloads). So let’s discuss how to solve this challenge!

## Prerequisites
In order to follow along with this tutorial yourself, you will need the following:
1. Experience with Atlas Data Federation.
2. An Atlas cluster with the sample data in it. 

Here is how the data is modeled in my cluster (data in your cluster can be spread out among collections however your application requires):

* Cluster: MongoTravelServices
    * Database: ireland
        * Collection: user_feedback (8658 Documents)
        * Collection: passengers
        * Collection: flights
    * Database: israel
        * Collection: user_feedback (8658 Documents)
        * Collection: passengers
        * Collection: flights
    * Database: usa
        * Collection: user_feedback (8660 Documents)
        * Collection: passengers
        * Collection: flights

![collection inside of Atlas Cluster that needs to be remodeled](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/collection_inside_atlas_cluster_3f2689960b.png)

The goal here is to consolidate this data into one database, and then have each of the collections for user feedback, passengers, and flights represent the data stored in the collections from each database on the cluster. Lastly, we also want to be able to query on the “database” name as if it were part of our documents.

## Create a Federated Database instance

* The first thing you’ll need to do is navigate to the “Data Federation” tab on the left-hand side of your Atlas dashboard and then click “set up manually” in the "create new federated database" dropdown in the top right corner of the UI.

![initial state of the Data Federation section of Atlas](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc94edfdcc342265b/649c786b84a4c7eb46d8db62/data-federation.png)

![Visual Storage Configuration editor for a Federated Database Instance](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/visual_storage_configuration_159cc5ab05.png)

* Then, for this example, we’re going to manually edit the storage configuration as these capabilities are not yet available in the UI editor. 

![JSON Storage Configuration editor for a Federated Database Instance](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/json_storage_configuration_f99a397725.png)

```
{
  "databases": [
    {
      "name": "GlobalVirtualDB",
      "collections": [
        {
          "name": "user_feedback",
          "dataSources": [
            {
              "collection": "user_feedback",
              "databaseRegex": ".*", // This syntax triggers the globbing or combination of each collection named user_feedback in each database of the MongoTravelServices cluster.
              "provenanceFieldName": "_provenance_data", // The name of the field where provenance data will be added.
              "storeName": "MongoTravelServices"
            }
          ]
        }
      ],
      "views": []
    }
  ],
  "stores": [
    {
      "clusterName": "MongoTravelServices",
      "name": "MongoTravelServices",
      "projectId": "5d9b6aba014b768e8241d442",
      "provider": "atlas",
      "readPreference": {
        "mode": "secondary",
        "tagSets": []
      }
    }
  ]
}
```

Now when you connect, you will see:

```
AtlasDataFederation GlobalVirtualDB> show dbs
GlobalVirtualDB  0 B
AtlasDataFederation GlobalVirtualDB> use GlobalVirtualDB
already on db GlobalVirtualDB
AtlasDataFederation GlobalVirtualDB> show tables
user_feedback
AtlasDataFederation GlobalVirtualDB>
```

And a simple count results in the count of all three collections globbed together:

```
AtlasDataFederation GlobalVirtualDB> db.user_feedback.countDocuments()
25976
AtlasDataFederation GlobalVirtualDB>
```

25976 is the sum of 8660 feedback documents from the USA, 8658 from Israel, and 8658 from Ireland.

And lastly, I can query on the provenance metadata using the field *“provenancedata.databaseName”*:

```
AtlasDataFederation GlobalVirtualDB> db.user_feedback.findOne({"_provenance_data.databaseName": "usa"})
{
  _id: ObjectId("63a471e1bb988608b5740f65"),
  'id': 21037,
  'Gender': 'Female',
  'Customer Type': 'Loyal Customer',
  'Age': 44,
  'Type of Travel': 'Business travel',
  'Class': 'Business',
  …
  'Cleanliness': 1,
  'Departure Delay in Minutes': 50,
  'Arrival Delay in Minutes': 55,
  'satisfaction': 'satisfied',
  '_provenance_data': {
    'provider': 'atlas',
    'clusterName': 'MongoTravelServices',
    'databaseName': 'usa',
    'collectionName': 'user_feedback'
  }
}
AtlasDataFederation GlobalVirtualDB>
```

## In review
So, what have we done and what have we learned?

1. We saw how quickly and easily you can create a Federated Database in MongoDB Atlas.
2. We learned how you can easily combine and reshape data from your underlying Atlas clusters inside of Atlas Data Federation with Collection Globbing. Now, you can easily query one user_feedback collection and have it query data in the user_feedback collections in each database.
3. We saw how to add provenance data to our documents and query it.

### A couple of things to remember about Atlas Data Federation
1. Collection globbing is a new feature that applies to Atlas cluster sources and allows dynamic manipulation of source collections similar to “wildcard collections.”
2. Provenance allows you to include additional metadata with your documents. You can indicate that data federation should include additional attributes such as source cluster, database, collection, the source path in S3, and more.
3. Currently, this is only supported in the Data Federation JSON editor or via setting the Storage Configuration in the shell, not the visual storage configuration editor.
4. This is particularly powerful for multi-tenant implementations done in MongoDB.

To learn more about [Atlas Data Federation](https://www.mongodb.com/atlas/data-federation) and whether it would be the right solution for you, check out our [documentation](https://www.mongodb.com/docs/atlas/data-federation/) and [tutorials](https://www.mongodb.com/developer/products/atlas/data-federation/tutorials/) or [get started today](https://cloud.mongodb.com/go?l=https://cloud.mongodb.com/v2/%3Cproject%3E#/dataFederation).

Learn how to model and transform your MongoDB Atlas Cluster data for real-time query-ability with Data Federation.

Benjamin Flast

Content by Benjamin Flast

How to Do Semantic Search in MongoDB Using Atlas Vector Search

How to Get MongoDB Data into Parquet in 10 Seconds or Less

Utilizing Collection Globbing and Provenance in Data Federation