Babu Srinivasan

Senior Partner Solutions Architect

In this tutorial, you'll find a tangible showcase of how AWS Glue, Amazon Kinesis, and MongoDB Atlas seamlessly integrate, creating a streamlined data streaming solution alongside extract, transform, and load (ETL) capabilities. This repository also harnesses the power of AWS CDK to automate deployment across diverse environments, enhancing the efficiency of the entire process.

To follow along with this tutorial, you should have intermediate proficiency with AWS and MongoDB services.

## Architecture diagram

![architecture diagram][1]

In the architecture described above, various streams of data, such as Order and Customer, are retrieved via the Amazon Kinesis stream. Subsequently, AWS Glue Studio is utilized to enrich the data. The enriched data is backed up in an S3 bucket, while the consolidated stream is stored in MongoDB Atlas and made accessible for downstream systems.

## Implementation steps

### Prerequisites

 - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed and configured
 - [NVM/NPM](https://nvm.sh/) installed and configured
 - [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) installed and configured
 - [MongoDB Atlas account](https://www.mongodb.com/docs/atlas/tutorial/create-atlas-account/), with the [Organization](https://www.mongodb.com/docs/atlas/government/tutorial/create-organization/#create-an-organization) set up
 - [Python packages](https://www.python.org/downloads/)
     - Python3 - `yum install -y python3`
     - Python Pip - `yum install -y python-pip`
     - Virtualenv - `pip3 install virtualenv`

>This repo is developed taking us-east-1 as the default region. Please update the scripts to your specific region (if required). This repo will create a MongoDB Atlas project and a free-tier database cluster automatically. No need to create a database cluster manually. This repo is created for a demo purpose and IP access is not restricted (0.0.0.0/0). Ensure you strengthen the security by updating the relevant IP address (if required).

### Setting up the environment

#### Get the application code

`git clone https://github.com/mongodb-partners/Stream_Data_into_MongoDB_AWS_Glue
cd kinesis-glue-aws-cdk`

#### Prepare the dev environment to run AWS CDK

a. Set up the AWS Environment variable AWS Access Key ID, AWS Secret Access Key, and optionally, the AWS Session Token.

```
export AWS_ACCESS_KEY_ID = <"your AWS access key">
   export AWS_SECRET_ACCESS_KEY =<"your AWS secret access key">
   export AWS_SESSION_TOKEN = <"your AWS session token">
```
b. We will use CDK to make our deployments easier. 

You should have npm pre-installed.
If you don’t have CDK installed:
`npm install -g aws-cdk`

Make sure you’re in the root directory.
`python3 -m venv .venv`
`source .venv/bin/activate`
`pip3 install -r requirements.txt`

> For development setup, use requirements-dev.txt.

c. Bootstrap the application with the AWS account.

`cdk bootstrap`

d. Set the ORG_ID as an environment variable in the .env file. All other parameters are set to default in global_args.py in the kinesis-glue-aws-cdk folder. MONGODB_USER and MONGODB_PASSWORD parameters are set directly in mongodb_atlas_stack.py and glue_job_stack.py

The below screenshot shows the location to get the Organization ID from MongoDB Atlas.

![Organization ID in MongoDB Atlas][3]

Please note that using "0.0.0.0/0" as an IP_ADDRESS, we are allowing access to the database from anywhere. This might be suitable for development or testing purposes but is highly discouraged for production environments because it exposes the database to potential attacks from unauthorized sources.

e. List the CDKs:

`cdk ls`

You should see an output of the available stacks:

    `aws-etl-mongo-atlas-stack`
    `aws-etl-kinesis-stream-stack`
    `aws-etl-bucket-stack`
   ` aws-etl-glue-job-stack`

### Deploying the application

Let’s walk through each of the stacks:

**Stack for MongoDB Atlas: aws-etl-mongo-atlas-stack**
This stack will create a MongoDB Atlas project and a free-tier database cluster with user and network permission (open).

#### Prerequisites

a. Create an AWS role with its trust relationship as a CloudFormation service.

Use the [template](https://github.com/mongodb/mongodbatlas-cloudformation-resources/blob/master/examples/execution-role.yaml) to create a new [CloudFormation stack](https://console.aws.amazon.com/cloudformation/home#/stacks/create) to create the execution role.

![Creating a new CloudFormation stsack][4]

b.The following Public Extension in the CloudFormation Registry should be activated with the Role created in the earlier step. After logging into the AWS console, use this link to register extensions on CloudFormation.

```
          MongoDB::Atlas::Project,
          MongoDB::Atlas::DatabaseUser,
          MongoDB::Atlas::Cluster,
          MongoDB::Atlas::ProjectIpAccessList
```
Pass the ARN of the role from the earlier step as input to activate the MongoDB resource in Public Extension.

MongoDB Resource Activation in Public Extension:

![Activation of Registry Public Extensions][5]

The above screenshot shows the activation of the Registry Public Extensions for the MongoDB::Atlas::Cluster.

Alternatively, you can activate the above public extension through AWS CLI.

Command to list the MongoDB Public Extensions. Note down the arns for the above four public extensions.

```
aws cloudformation list-types \
  --visibility PUBLIC \
  --filters "Category=THIRD_PARTY,TypeNamePrefix=Mongodb"
```

Command to activate the Public Extension. Use this command to activate all four public extensions mentioned in the previous steps.

```
aws cloudformation activate-type --region us-east-1 --public-type-arn "<arn for the public extension noted down in the previous step>" --execution-role-arn "<arn of the role created in step a>"
```

c. Log into the MongoDB console and note down the organization ID. Ensure the Organization ID is updated in the global_args.py

![Cluster organization settings][6]

The above screenshot shows the MongoDB Cluster Organization settings.

d. Create an API Key in an organization with Organization Owner access. Note down the API credentials.

![Edit API key][7]

The above screenshot shows the access managers for the API Key created in the MongoDB Atlas cluster.

Restrict the access to the Organization API with the API Access list. We provided an open access 0.0.0.0/1 for demo purposes only. We strongly discourage the use of this in any production environment or equivalent.

e. A profile should be created in the AWS Secrets Manager containing the MongoDB Atlas Programmatic API Key.

Use the [template](https://github.com/mongodb/mongodbatlas-cloudformation-resources/blob/master/examples/profile-secret.yaml) to create a new CloudFormation [stack](https://console.aws.amazon.com/cloudformation/home#/stacks/create) for the default profile that all resources will attempt to use unless a different override is specified.

#### Profile secret stack

![AWS CloudFormation stack][8]

The above screenshot shows the parameters for the AWS CloudFormation stack.

Initiate the deployment with the following command:

```bash
  cdk deploy aws-etl-mongo-atlas-stack
  ```
After successfully deploying the stack, validate the Outputs section of the stack and MongoDB Atlas cluster. You will find the stdUrl and stdSrvUrl for the connection string.

#### Stack:

![Output of CloudFormation stack][8]

The above screenshot shows the output of the CloudFormation stack.

#### MongoDB Atlas cluster:

![Creation of a MongoDB Atlas cluster][9]

The above screenshot shows the successful creation of the MongoDB Atlas cluster.

### Stack for creating the Kinesis stream: aws-etl-kinesis-stream-stack

This stack will create two Kinesis data streams. Each producer runs an ingesting stream of events for different customers with their orders.

Initiate the deployment with the following command:

  ```bash
  cdk deploy aws-etl-kinesis-stream-stack
  ```
After successfully deploying the stack, Check the Outputs section of the stack. You will find the CustomerOrderKinesisDataStream Kinesis function.

#### Stack:

![Output of the CloudFormation stack][10]

The above screenshot shows the output of the CloudFormation stack for Kinesis streams.

#### Amazon Kinesis data stream:

![Kinesis stream][11]

The above screenshot shows the Kinesis stream created.

### Stack for creating the S3 bucket: aws-etl-bucket-stack

This stack will create an S3 bucket that will be used by AWS Glue jobs to persist the incoming customer and order details.

  ```bash
  cdk deploy aws-etl-bucket-stack
  ```
After successfully deploying the stack, check the Outputs section of the stack. You will find the 
S3SourceBucket resource.

#### Stack:

![Output of the CloudFormation stack][12]

The above screenshot shows the Output of the CloudFormation stack.

AWS S3 Bucket:

![S3 buckets created][13]

The above screenshot shows the S3 buckets created.

### Stack for creating the AWS Glue job and parameters: aws-etl-glue-job-stack

This stack will create two AWS Glue jobs: one job for the customer and another for the order. The code is in the location glue_job_stack/glue_job_scripts/customer_kinesis_streams_s3.py and glue_job_stack/glue_job_scripts/order_kinesis_streams_s3.py.

  ```bash
  cdk deploy aws-etl-glue-job-stack
  ```
#### Stack:

![Output of CloudFormation stack][14]

The above screenshot shows the output of the CloudFormation stack.

#### AWS Glue job:

![AWS Glue Studio][15]

The above screenshot shows the AWS Glue Studio job to set up the job.

#### Note

The MongoDB URL of the newly created cluster and other parameters will be passed to the AWS Glue job programmatically. Update these parameters to your values (if required).

"Spark UI logs path" and "Temporary path" details will be maintained in the same bucket location with folder name /sparkHistoryLogs and /temporary.

#### Spark UI logs path:

  `s3://<S3_BUCKET_NAME>/sparkHistoryLogs` 

#### Temporary path:

  `s3://<S3_BUCKET_NAME>/temporary/`

#### Screenshot of the AWS Glue parameters

![AWS Glue parameters][16]

The above screenshot shows the AWS Glue parameters.

Once you are ready with all stacks, start the producers for the customer and order. The code is in this location, to ingest data into a kinesis data stream and also start the Glue job for both.

  `producer/customer.py` and `producer/order.py`

#### Sample record in MongoDB Atlas:

```
{
  "_id": "1",
  "country_id": "1",
  "customer_name": "NICK",
  "email_id": "nick@gmail.com",
  "orders": [
    {
      "order_id": "8",
      "product_name": "Artisanal Cheese Selection",
      "quantity": "8",
      "price": "29.17"
    }
  ]
}
```

## Clean up

Use `cdk destroy` to clean up all the AWS CDK resources. 

## Troubleshooting

Refer to [GitHub](https://github.com/mongodb/mongodbatlas-cloudformation-resources/tree/master#troubleshooting) to resolve some common issues encountered when using AWS CloudFormation/CDK with MongoDB Atlas Resources.

## Useful commands

`cdk ls` lists all stacks in the app.
`cdk synth` emits the synthesized CloudFormation template.
`cdk deploy` deploys this stack to your default AWS account/region.
`cdk diff` compares the deployed stack with the current state.
`cdk docs` opens CDK documentation.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt23fd6a56157b9c41/675b1787cc57be7a2703736d/img-1-fix.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt95d66e4812fd56ed/661e9e36e9e603e1aa392ef0/architecture-diagram.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt401e52a6c6ca2f6f/661ea008f5bcd1bf540c99bd/organization-settings.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltffa00dde10848a90/661ea1fe190a257fcfbc5b4e/cloudformation-stack.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcc9e58d51f80206d/661ea245ad926e2701a4985b/registry-public-extensions.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc5a089f4695e6897/661ea2c60d5626cbb29ccdfb/cluster-organization-settings.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1a0bed34c4fbaa4f/661ea2ed243a4fa958838c90/edit-api-key.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt697616247b2b7ae9/661ea35cdf48e744da7ea2bd/aws-cloud-formation-stack.png

In this tutorial, find a tangible showcase of how AWS Glue, Amazon Kinesis, and MongoDB Atlas seamlessly integrate.

Stream Data Into MongoDB Atlas Using AWS Glue

Igor Alekseev

Partner Solutions Architect, Data & Analytics

Ashwin S.

Associate Architect, WeKan

In this tutorial, you'll discover how to securely host a Lambda application designed with a microservice architecture, utilizing AWS Lambda, Amazon API Gateway, and MongoDB Atlas. AWS Lambda is pivotal for enterprises, offering standardized serverless deployment, enhanced security, and streamlined management, which collectively lead to consistent performance and simplified operations. 

This guide not only details the process of setting up and deploying a MongoDB Atlas-based application with a microservice and API pattern but also provides a robust framework applicable to any such deployment. By leveraging AWS CDK for automated deployment, you ensure a scalable, efficient, and secure solution that meets enterprise needs and accelerates your deployment process across various environments.

To follow along with this tutorial, you should have intermediate proficiency with AWS and MongoDB services.

## Architecture diagram

![AWS Lambda architecture diagram][1]

The architecture described here includes the following components:

1. [AWS Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/): Public Repository for AWS Lambda application 
2. [AWS Lambda](https://aws.amazon.com/pm/lambda/): Hosts the microservices logic
3. [Amazon API Gateway](https://aws.amazon.com/es/api-gateway/): Exposes the Lambda functions as RESTful APIs
4. [MongoDB Atlas](https://www.mongodb.com/products/platform/atlas-database): Stores the application data securely

## Implementation steps to set up your Lambda application

This tutorial assumes `us-east-1` as the default region. Please update the scripts to your specific region. if required.

### Prerequisites

Before proceeding, ensure you have the following prerequisites installed:

- AWS account with IAM creation role
- AWS CLI: [Install AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
- IAM user for AWS CLI with access keys and secret keys
- SAM CLI: [Install SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html)
- Docker: [Install Docker](https://docs.docker.com/engine/install/)
- Python 3.10: [Install Python 3.10](https://www.python.org/downloads/release/python-3100/)

### Setting up the environment

Clone the application repository.

```
git clone https://github.com/mongodb-partners/Microservice-api-lambda-MongoDBAtlas
cd Microservice-api-lambda-MongoDBAtlas
```

Configure AWS CLI.

```
aws configure
```
Enter your AWS Access Key ID, Secret Access Key, and preferred region.

Set up the Python virtual environment.

```
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r lambda/python-crud-lambda/requirements.txt
```
To build the project locally, execute the following command:

```
sam build
```

This command will locally build the project, and output files will be saved in the .aws-sam folder.

### Set up MongoDB Atlas 

- [Set up a MongoDB Atlas cluster](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/).
- Set up the [network access](https://www.mongodb.com/docs/atlas/security/add-ip-address-to-list/).
- Set up the [database users](https://www.mongodb.com/docs/atlas/tutorial/create-mongodb-user-for-cluster/).
- Copy and note the [connection string](https://www.mongodb.com/docs/atlas/driver-connection/). This string will be used to set up the Lambda application.

Use an existing connection string: Ensure you have the complete MongoDB SRV connection string with your username and password included. Ensure that Lambda can access your cluster. For testing purposes, you can use `0.0.0.0/0` in your network IP access list. In production, consider setting up a VPC peering connection between your AWS VPC and the Atlas VPC. This will restrict public access and only accept connections from your VPC. Learn more about VPC peering in the [MongoDB Atlas documentation](https://docs.atlas.mongodb.com/vpc-peering/).

### Deploying the application

Execute the following command to deploy the application:

bash
```
sam deploy --guided
```

Follow the guided deploy instructions:

* Stack Name: Enter a name for your stack.
* AWS Region: Enter your preferred region.
* Parameter ConnectionString: Enter your MongoDB connection string.
* Parameter DBName: Enter the name of your MongoDB database.
* Parameter CollectionName: Enter the name of your MongoDB collection.

Sample values for deployment

```
  Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [sam-app]: sam-app-api-mdb
        AWS Region [us-east-1]: 
        Parameter ConnectionString []: mongodb+srv://test1:test1@demo.mongodb.net/?retryWrites=true&w=majority
        Parameter DBName [test]: sam-app-api-mdb
        Parameter CollectionName [test]: microservice-lambda-app
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [y/N]: y
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: y
        #Preserves the state of previously provisioned resources when an operation fails
        Disable rollback [y/N]: y
        CRUDLambda has no authentication. Is this okay? [y/N]: y
        Save arguments to configuration file [Y/n]: y
        SAM configuration file [samconfig.toml]: 
        SAM configuration environment [default]:
```

![Sample deployment screenshot][2]

Wait for the CloudFormation stack to finish creating the stack. After deployment, you will receive the API Gateway link in the outputs.

![Cloud formation stack completion][3]

![AWS Cloudformation console][4]

### Create a S3 bucket

Create an S3 public bucket to store the Lambda code and restrict the access through bucket policy.

```
aws s3api create-bucket --bucket aws-sam-cli-mongodb-app1 --region us-east-1

aws s3api put-bucket-policy --bucket my-unique-bucket-name --policy '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service":  "serverlessrepo.amazonaws.com"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::aws-sam-cli-mongodb-app1/*",
            "Condition" : {
                "StringEquals": {
                    "aws:SourceAccount": "<aws account id >"
                }
            }
```

### Packaging the application

Package the serverless application model (SAM) deployed application to the S3 bucket created in the previous steps of SAM build.

```
sam package --template-file template.yaml --output-template-file packaged.yaml --s3-bucket aws-sam-cli-mongodb-app1
```

### Publishing the application

To publish your application as a serverless application model package, execute the following command:

bash
```
sam publish --template packaged.yml --region us-east-1
```

Follow the instructions to publish your application. By default, your application will be published as a private app. After publication, you can convert it to a public app if needed. For detailed guidance, refer to the [SAM documentation on publishing an application](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-publishing-applications.html).

#### Verification and validation

Use the API Gateway link received in the deployment outputs to test the APIs. You can use Postman or any other request-making tool.

* Postman collection: Import the Postman collection `postman/Python-Microservice.postman_collection.json` into Postman to test the API.

### Clean up

Use the following command to clean up all the AWS resources created during deployment:

```
sam delete
```

### Troubleshooting

Refer to the [GitHub repository](https://github.com/mongodb-partners/Microservice-api-lambda-MongoDBAtlas) to resolve common issues encountered when using AWS SAM with MongoDB Atlas.

### Useful links

* [Serverless Application Repository (SAR)](https://docs.aws.amazon.com/serverlessrepo/latest/devguide/what-is-serverlessrepo.html)
* [SAM deployment](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-publishing-applications.html)

### Useful commands

* `sam build`: Builds the project locally
* `sam deploy --guided`: Deploys the application with guided prompts
* `sam publish --template packaged.yaml`: Publishes the application as a SAM package
* `sam delete`: Deletes the deployed application

By following these steps, you can set up a private hosting environment for your microservices-based Lambda application using AWS services and MongoDB Atlas. This setup ensures a secure, scalable, and cost-effective solution for hosting serverless microservices applications.

Try out the AWS [API Gateway](https://aws.amazon.com/api-gateway/)! For any MongoDB related queries, visit the [MongoDB community support](https://www.mongodb.com/community/). 



  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf1cfa4e5d1b8c13e/66ec5b5afbc1b9d136d70704/aws-lambda-architecture-diagram.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0dded2b9d1c42dbb/66ec5cbc9ae20569ca6856fb/sample-deployment-screenshot.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt9b404fedc472dc72/66ec5ce0164dbf1f8a5afd2f/cloud-formation-stack-completion.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt815080f2f6d5309a/66ec5d06a924ba186bdf54ee/aws-cloudformation-console.png

Learn to set up and deploy an Atlas-based application with a microservice and API pattern and get a framework applicable to any such deployment.

Securely Hosting a Lambda Application With a Microservice Architecture and MongoDB Atlas

Abilash Ravikumar

Product Manager at Datadog

> Some features mentioned below will be deprecated on Sep. 30, 2025. [Learn more](https://www.mongodb.com/products/updates/product-support-deprecation?tck=notice_product_deprecation_2024).

[Datadog](https://www.datadoghq.com/) is a well-known monitoring and security platform for cloud applications. Datadog’s software-as-a-service (SaaS) platform integrates and automates infrastructure monitoring, application performance monitoring, and [log management](https://docs.datadoghq.com/logs/log_collection/) to provide unified, real-time observability of a customer’s entire technology stack.

MongoDB Atlas on Amazon Web Services (AWS) already supports [easy integration with Datadog](https://www.mongodb.com/docs/atlas/tutorial/datadog-integration/) for alerts and events right within the Atlas UI (select the three vertical dots → Integration → Datadog). With the Log Forwarding feature, it's now possible to send Atlas Application Services logs to Datadog. This blog outlines the configuration steps necessary as well as strategies for customizing the view to suit the need.

![Datadog Configuration from Atlas ](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/datadog_configuration_atlas_00347e84a6.png)

[**Atlas Application Services**](https://www.mongodb.com/docs/atlas/app-services/) (formerly MongoDB Realm) is a set of enhanced services that complement the Atlas database to simplify the development of backend applications. App Services-based apps can react to changes in your MongoDB Atlas data, connect that data to other systems, and scale to meet demand without the need to manage the associated server infrastructure.

App Services provides user authentication and management, schema validation and data access rules, event-driven serverless functions, secure client-side queries with HTTPS Endpoints, and best of all, synchronization of data across devices with the Realm Mobile SDK.

With App Services and Datadog, you can simplify the end-to-end development and monitoring of your application. **Atlas App Services** specifically enables the forwarding of logs to Datadog via a serverless function that can also give more fine-grained control over how these logs appear in Datadog, via customizing the associated tags.

## Atlas setup

We assume that you already have an Atlas account. If not, you can sign up for a [free account](https://www.mongodb.com/realm/register) on MongoDB or the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34?trk=38088f03-4928-4b00-9a39-86781df6ab9b&sc_channel=el). Once you have an Atlas account, if you haven't had a chance to try App Services with Atlas, you can follow one of our [tutorials](https://www.mongodb.com/docs/atlas/app-services/tutorials/) to get a running application working quickly.

To initiate custom log forwarding, follow the instructions for App Services to configure [log forwarding](https://www.mongodb.com/docs/atlas/app-services/logs/forward-logs/). Specifically, choose the “To Function” option:

![MongoDB Atlas Log configuration](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/mongodb_atlas_log_configuration_1f15e0de05.png)

Within Atlas App Services, we can create a custom function that provides the mapping and ingesting logs into Datadog. Please note the intake endpoint URL from Datadog first, which is [documented by Datadog](https://docs.datadoghq.com/logs/log_collection/?tab=host).

Here’s a sample function that provides that basic capability:

```
exports = async function(logs) {
  // `logs` is an array of 1-100 log objects
  // Use an API or library to send the logs to another service.
  await context.http.post({
    url: "https://http-intake.logs.datadoghq.com/api/v2/logs",
    headers: {
      "DD-API-KEY": ["XXXXXX"],
      "DD-APPLICATION-KEY": ["XXXXX"], 
      "Content-Type": ["application/json"]
    },
    body: logs.map(x => {return {
      "ddsource": "mongodb.atlas.app.services",
      "ddtags": "env:test,user:igor",
      "hostname": "RealmApp04",
      "service": "MyRealmService04",
      "message" : JSON.stringify(x)
    }}),
    encodeBodyAsJSON: true
  });
}
```

One of the capabilities of the snippet above is that it allows you to modify the function to supply your Datadog [API and application keys](https://docs.datadoghq.com/account_management/api-app-keys/). This provides the capability to customize the experience and provide the appropriate context for better observability. You can change [ddtags](https://docs.datadoghq.com/getting_started/tagging/), the hostname, and service parameters to reflect your organization, team, environment, or application structure. These parameters will appear as facets helping with filtering the logs.

Note: Datadog supports log ingestion pipelines that allow it to better parse logs. In order for the MongoDB log pipeline to work, your *ddsource* must be set to `mongodb.atlas.app.services`.

![Select the MongoDB as a source in Datadog](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/select_mongodo_source_datadog_c9abd3a85a.png)

## Viewing the logs in Datadog

Once log forwarding is configured, your Atlas App Services logs will appear in the Datadog Logs module.

![Datadog Logs](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/datadog_logs_73ed6659cd.png)

You can click on an individual log entry to see the detailed view:

![Datadog Event Attributes](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/datadog_event_attributes_d4d8228ed3.png)

## Conclusion

In this blog, we showed how to configure log forwarding for Atlas App Services logs. If you would like to try configuring log forwarding yourself, sign up for a [14-day free trial of Datadog](https://www.datadoghq.com/free-datadog-trial/) if you don’t already have an account. To try Atlas App Services in AWS Marketplace, sign up for a [free account](https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34).

With the Log Forwarding feature, you can send Atlas App Services logs to Datadog. Learn the configuration steps as well as strategies for customizing the view.

Integrate Atlas Application Services Logs Into Datadog on AWS

Partner

Erik Onnen

Senior Partner SA at AWS

## Introduction

MongoDB Atlas is now natively integrated with Amazon Bedrock Knowledge Base, making it even easier to build generative AI applications backed by enterprise data. 

Amazon Bedrock, Amazon Web Services’ (AWS) managed cloud service for generative AI, empowers developers to build applications on top of powerful foundation models like Anthropic's Claude, Cohere Embed, and Amazon Titan. By integrating with Atlas Vector Search, Amazon Bedrock enables customers to leverage the vector database capabilities of Atlas to bring up-to-date context to Foundational Model outputs using proprietary data. 

With the click of a button (see below), Amazon Bedrock now integrates MongoDB Atlas as a vector database into its fully managed, end-to-end retrieval-augmented generation (RAG) workflow, negating the need to build custom integrations to data sources or manage data flows. 

Companies using MongoDB Atlas and Amazon Bedrock can now rapidly deploy and scale generative AI apps grounded in the latest up-to-date and accurate enterprise data. For enterprises with the most demanding privacy requirements, this capability is also available via AWS PrivateLink (more details at the bottom of this article).

## What is retrieval-augmented generation?

One of the biggest challenges when working with generative AI is trying to avoid hallucinations, or erroneous results returned by the foundation model (FM) being used. The FMs are trained on public information that gets outdated quickly and the models cannot take advantage of the proprietary information that enterprises possess.

One way to tackle hallucinating FMs is to supplement a query with your own data using a workflow known as retrieval-augmented generation, or RAG. In a RAG workflow, the FM will seek specific data — for instance, a customer's previous purchase history — from a designated database that acts as a “source of truth” to augment the results returned by the FM. For a generative AI FM to search for, locate, and augment its responses, the relevant data needs to be turned into a vector and stored in a vector database.

## How does the Knowledge Base integration work?

Within Amazon Bedrock, developers can now “click to add” MongoDB Atlas as a knowledge base for their vector data store to power RAG.

In the workflow, a customer chooses two different models: an embedding model and a generative model. These models are then orchestrated and used by Bedrock Agents during the interaction with the knowledge base — in this case, MongoDB Atlas.

Bedrock reads your text data from an S3 bucket, chunks the data, and then uses the embedding model chosen by the user to create the vector embeddings, storing these text chunks, embeddings, and related metadata in MongoDB Atlas’ vector database. An Atlas vector search index is also created as part of the setup for querying the vector embeddings.

![High level block diagram depicting each of the services and its data flow][1]

## Why choose MongoDB Atlas as a Bedrock knowledge base?

[MongoDB Atlas](https://www.mongodb.com/atlas) combines operational, vector, and metadata in a single platform, making it an ideal knowledge base for Amazon Bedrock users who want to augment their generative AI experiences while also simplifying their generative AI stack.

In addition, MongoDB Atlas gives developers the ability to set up dedicated infrastructure for search and vector search workloads, optimizing compute resources to scale search and database independently. 

## Solution architecture

![architecture diagram][2]

In the architecture diagram above, documents are uploaded to the Amazon Bedrock Knowledge Base (via S3) and stored within the MongoDB Atlas vector store. User queries are then addressed through specialized Amazon Bedrock Agents tailored to individual use cases, utilizing the MongoDB Atlas vector search functionality.

## Dataset

In this demo, we use the [Best Practices Guide for MongoDB](https://www.mongodb.com/collateral/best-practices-guide-for-mongodb) to populate our knowledge base. Please download the PDF (by clicking on “Read Whitepaper” or “Email me the PDF”). Alternatively, you can download it from the GitHub [repository](https://github.com/mongodb-partners/mongodb_atlas_as_aws_bedrock_knowledge_base/blob/main/data/MongoDB_Best_Practices_Guide.pdf). Once you have the PDF, upload it into an S3 bucket for hosting. (Note the bucket name as we will use it later in the article.) 

## Prerequisites

* MongoDB Atlas account
* AWS account 

## Implementation steps

### Atlas Cluster and Database Setup 

* [Login or Signup][3] to MongoDB Atlas 
* [Setup][4] the MongoDB Atlas cluster with a M10 or greater configuration. *Note M0 or free cluster will not support this setup.*
* Setup the [database user][5] and [Network access][6].
* Copy the [connection string][7].
* [Create][8] a database and collection

![The screenshot shows the navigation of creating a database in MongoDB Atlas.][9]

### Atlas Vector Search index

Before we create an Amazon Bedrock knowledge base (using MongoDB Atlas), we need to create an Atlas Vector Search index.

* In the MongoDB Atlas Console, navigate to your cluster and select the _Atlas Search_ tab. 

![Atlas console navigation to create the search index][10]

* Select _Create Search Index_, select _Atlas Vector Search_, and select _Next_.

![The screenshot shows the MongoDB Atlas Search Index navigation.][11]

* Select the database and the collection where the embeddings are stored.

![MongoDB Atlas Search Index navigation][12]    

*  Supply the following JSON in the index definition and click _Next_, confirming and creating the index on the next page.

    ```
   {
      "fields": [
        {
          "numDimensions": 1536,
          "path": "bedrock_embedding",
          "similarity": "cosine",
          "type": "vector"
        },
        {
          "path": "bedrock_metadata",
          "type": "filter"
        },
        {
          "path": "bedrock_text_chunk",
          "type": "filter"
        }
      ]
    }
    ```
![The screenshot shows the MongoDB Atlas Search Index navigation][13]

Note: The fields in the JSON are customizable but should match the fields we configure in the Amazon Bedrock AWS console. If your source content contains [filter metadata](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-ds.html#kb-ds-metadata), the fields need to be included in the JSON array above in the same format: `{"path": "&lt;attribute_name>","type":"filter"}`.

### Amazon Bedrock Knowledge Base 

* In the AWS console, navigate to Amazon Bedrock, and then click _Get started_.

![Landing page of the Amazon Bedrock console][14]

* Next, click on _Model Access._ 

![Overview page of the Amazon Bedrock console][15]
    
* Ensure that the Amazon and Anthropic models are selected. 

![Model Access page of the Amazon Bedrock console][16]

* Next, navigate to _Knowledge Bases_ in the left-hand menu and select _Create Knowledge Base_. 

![Amazon Bedrock console page for selecting the Knowledge bases][17]
    
* Give your knowledge base a name and select _Next_. (Add an optional description, if you’d like.)  

![Configuration page of the Amazon Bedrock Knowledge bases console][18]

* Supply the S3 bucket where you uploaded the PDF of “Best Practices Guide for MongoDB” from earlier and select _Next_. 

![Navigation to set up the data source][19]
        
* Next, select _Titan Embeddings Model._ 

![Navigation for the selection of the embedding model][20]  

* Scroll down to configure MongoDB Atlas as the vector database which was set up earlier. 

![Navigation for the vector store selection][21]

 * Scroll down to fill out the MongoDB configuration options. 
     * The below configuration steps assume connectivity to MongoDB Atlas over the Internet and it’s recommended only for non-production use cases.
     * To configure the connectivity over the secured PrivateLink (PL), follow the additional steps (detailed in the ReadMe) of the [CDK script][22] to configure Endpoint Service.
     * To supply the secret ARN, create a secret in this format: `{"username":"xxxx","password":"xxx"}`. \
Note: As a recommended security practice, the credentials should NOT have _Atlas Admin_ privileges.  They should be no more permissive than the _Read and write to any database_ permission. 

![Configuration details for the MongoDB Atlas Vector as a Knowledge base][23]

* And metadata field mappings from the JSON file you set up earlier on Atlas via the JSON editor. 

![Configuration of the metadata field mapping for the vector search][24]

* Next, review and create the knowledge base.

![Review panel for all the configurations made earlier][25] 

* Once the creation is complete, navigate to the Data Source and click the Sync button to sync the data source. 

![Successful completion of the Knowledge base creation][26]

* When the sync completes, you can navigate to your database collection in the MongoDB Atlas console. _Note the vector size matches the embeddings model vector size._ 

![MongoDB database collections with vector embedding][27]

## Amazon Bedrock Agent

[Amazon Bedrock Agents](https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html) orchestrate interactions between foundation models, data sources, software applications, and user conversations. In addition, agents automatically call APIs to take actions and invoke knowledge bases to supplement information for these actions

* In the AWS Bedrock console, create an Agent. 

![Landing page of the Amazon Bedrock Agents][28]    

* Provide the agent Name and an optional description. 

![Configurations for the Amazon Bedrock Agent][29]

* Select a model and provide the prompt. 

![Selection of models for the Amazon Bedrock Agent][30]

* For our agent, we will skip the _Action Group_ and configure our knowledge base instead.  Select the knowledge base configured earlier, supply instructions for the Agent and select Add. 

![Selection of knowledge base for the Amazon Bedrock Agent][31]        

* Next, save the configuration to create the agent. 

![Success message after saving agent’s configuration.][32]

* Once the agent is successfully created, go ahead and test it by asking a question. 

![Chat output for the given query][33]

## Conclusion

This article demonstrates the process of establishing a knowledge base in Amazon Bedrock, using MongoDB Atlas as the vector database. Once set up, Amazon Bedrock will use your MongoDB Atlas Knowledge Base for data ingestion, and subsequently craft an Agent capable of responding to inquiries based on your accurate, proprietary data. 

## Useful links and resources:

* [MongoDB docs about Bedrock integration](https://dochub.mongodb.org/core/amazon-bedrock)
* [AWS docs about MongoDB Bedrock integration](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html)
* [MongoDB Vector Search](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/)
* [Bedrock User Guide](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html)
* [MongoDB Atlas on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34)


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt511ed709f8f6d72c/66323fa5ba17b0c937cb77a0/1_image.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt33b9187a37c516d8/66323fa65319a05c071f59a6/2_image.png
  [3]: https://www.mongodb.com/docs/guides/atlas/account/
  [4]: https://www.mongodb.com/docs/guides/atlas/cluster/
  [5]: https://www.mongodb.com/docs/guides/atlas/db-user/
  [6]: https://www.mongodb.com/docs/guides/atlas/network-connections/
  [7]: https://www.mongodb.com/docs/guides/atlas/connection-string/
  [8]: https://www.mongodb.com/basics/create-database#using-the-mongodb-atlas-ui
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt6fe6d13267c2898e/663bb48445868a5510839ee6/27_bedrock.png
  [10]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt81882c0004f72351/66323fa5368bfca5faff012c/3_image.png
  [11]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7f47b6bde3a5c6e0/66323fa6714a1b552cb74ee7/4_image12.png
  [12]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt2a5dcee8c6491baf/66323fa63c98e044b720dd9f/5_image.png
  [13]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt9fec5112de72ed56/663bb5552ff97d53f17030ad/28_bedrock.png
  [14]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1fd932610875a23b/66323fa65b8ef39b7025bd85/7_image.png
  [15]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltbc8d532d6dcd3cc5/66323fa6ba17b06b7ecb77a8/8_image.png
  [16]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0a24ae8725a51d07/66323fa6e664765138d445ee/9_image.png
  [17]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt88d2e77e437da43f/66323fa6e664767b99d445ea/10_image.png
  [18]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltad5b66cb713d14f7/66323fa63c98e0b22420dd97/11_image.png
  [19]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0da5374bed3cdfc4/66323fa6e66476284ed445e6/12_image.png
  [20]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1a5f530f0d77cb25/66323fa686ffea3e4a8e4d1e/13_image.png
  [21]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3c5a3bbfd93fc797/66323fa6ba17b003bccb77a4/14_image.png
  [22]: https://github.com/mongodb-partners/mongodb_atlas_as_aws_bedrock_knowledge_base
  [23]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt17ef52eaa92859b7/66323fa6f5bf2dff3c36e840/15_image.png
  [24]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt039df3ec479a51b3/66323fa6dafc457afab1d9ca/16_image18.png
  [25]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltff47aa531f588800/66323fa65319a08f491f59aa/17_image.png
  [26]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb886edc28380eaad/66323fa63c98e0f4ca20dda1/18_image.png
  [27]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt94659b32f1eedb41/66323fa657623318c954d39d/19_image.png
  [28]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltce5243d13db956ac/66323fa6599d112fcc850538/20_image.png
  [29]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5cac340ac53e5630/66323fa6d63d2215d9b8ce1e/21_image.png
  [30]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltecb09d22b3b99731/66323fa586ffea4e788e4d1a/22_image.png
  [31]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcc59300cd4b46845/66323fa6deafa962708fcb0c/23_image.png
  [32]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt381ef4a7a68c7b40/66323fa54124a57222a6c45d/24_image.png
  [33]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blte3d2b90f86472bf1/66323fa5ba17b0c29fcb779c/25_image.png

Atlas Vector Search and Amazon Bedrock enable the vector database capabilities of Atlas to bring up-to-date context to Foundational Model outputs.

Babu Srinivasan

Content by Babu Srinivasan

Stream Data Into MongoDB Atlas Using AWS Glue

Securely Hosting a Lambda Application With a Microservice Architecture and MongoDB Atlas

Integrate Atlas Application Services Logs Into Datadog on AWS

Launch a Fully Managed RAG Workflow With MongoDB Atlas and Amazon Bedrock