MongoDB

Anuj Panchal

Partner Solutions Architect

Igor Alekseev

Partner Solutions Architect, Data & Analytics

Partner

Dan Kiuna

Specialist Solutions Architect at AWS

In today’s fast-paced digital landscape, turning innovative ideas into fully functional applications quickly and efficiently is essential for developers, startups, and business consultants. This repository is designed to empower you to do just that. By leveraging [AWS Amplify](https://aws.amazon.com/amplify) for seamless front-end development, [AWS AppSync](https://aws.amazon.com/pm/appsync/) for robust and scalable backend services, and [MongoDB Atlas](https://www.mongodb.com/products/platform/atlas-database) for a flexible and powerful database solutions, developers can build  comprehensive, low-code solutions while drastically reducing coding time and maintaining high standards of performance and scalability.

This tutorial and repository provides a starter template for building applications with React and AWS Amplify, seamlessly connecting to MongoDB Atlas. It simplifies the setup for authentication, API integration, and database capabilities. With a foundational React application pre-integrated with AWS Amplify, this template is optimized for scalability and performance, making it perfect for developers, startups, and business consultants seeking to jumpstart their projects with pre-configured AWS services like Cognito, AppSync, and MongoDB Atlas


# AWS Amplify

AWS Amplify simplifies the creation of full-stack applications with minimal configuration. It provides a set of tools and services that integrate seamlessly with popular frameworks like React, Angular, and Vue, enabling developers to easily set up scalable backends, manage authentication, and deploy applications with a few simple commands. Amplify recently launched "Amplify Gen 2" which enhances support for CI/CD workflows, improves the developer experience with the Amplify CLI, and expands capabilities for backend infrastructure as code. These updates facilitate easier management and deployment of cloud resources, integration of serverless functions, and connection to MongoDB Atlas data sources through AWS AppSync and Lambda, thus significantly reducing the time and effort required to build and scale modern applications


# AWS AppSync

AWS AppSync is a fully managed service that simplifies the development of GraphQL APIs by automatically handling data fetching, real-time updates, and offline synchronization. It seamlessly integrates with various data sources, including MongoDB Atlas, allowing developers to build scalable and flexible applications with ease. AppSync’s features include built-in support for real-time data updates and offline access, reducing the complexity of managing data interactions and enhancing application performance


# MongoDB Atlas

MongoDB Atlas is a fully managed cloud database service known for its operational excellence and scalability. It offers advanced features like [automated backups](https://www.mongodb.com/docs/atlas/backup-restore-cluster/), [scaling](https://www.mongodb.com/docs/atlas/cluster-autoscaling/), and [monitoring](https://www.mongodb.com/docs/atlas/monitoring-alerts/), ensuring high availability and performance. Notably, Atlas includes [Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) capabilities, which are optimized for Generative AI use cases, enabling efficient search and retrieval of high-dimensional data. This feature enhances the ability to perform complex queries and data analysis, making it ideal for applications requiring sophisticated data interactions and AI-driven insights


# Application Features

In this blog we implement a Todo Application that supports authentication by leveraging Amazon Cognito.  The application frontend is written in React, while the backend is a GraphQL API powered by AWS AppSync and AWS Lambda.  The persistence layer is provided by MongoDB Atlas.  For the application’s CI/CD and hosting we rely on out-of-the-box capabilities for AWS Amplify. 


# Reference Architecture

![Reference Architecture][1]

# Prerequisites

In order to deploy the application you need to have [MongoDB Atlas Cluster](https://www.mongodb.com/docs/atlas/reference/amazon-aws/) created and an AWS Account.


# Deploying to AWS


## Step 1 

Create a fork of this repository: `https://github.com/mongodb-partners/amplify_appsync_mongodb_atlas_startup.git`


## Step 2

Set up the MongoDB Atlas cluster.  Follow the [link](https://www.mongodb.com/docs/atlas/tutorial/create-atlas-account/) to the setup the [MongoDB Atlas cluster](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster/), Database, [User](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/) and [Network access](https://www.mongodb.com/docs/atlas/security/add-ip-address-to-list/).


## Step 3

Obtain a connection URL

![Obtain Connection URL][2]

## Step 4

Deploy the To-do Application in AWS Amplify.  Open the AWS Amplify console and select the Github option

![Deploy to Amplify][3]

Authenticate to GitHub and select the repository you created earlier by forking.

![Authenticate to Github][4]

Set all other options to default and deploy

![Set options and deploy][5]

Configure the Environment Variables.  Note the deployments will not succeed until the environment variables are set.

![Configure Environment Variables][6]

## Step 5

Test the deployed application

![Test Deployed App][7]

Verify objects are being stored in the MongoDB Atlas Collection

![Verify Objects in Atlas][8]

# Conclusion

In this tutorial we showed how to deploy an application using MongoDB Atlas and AWS Amplify.  If you have ideas on how to enhance the application, please submit a pull request for this [repository](https://github.com/mongodb-partners/amplify_appsync_mongodb_atlas_startup.git).  Try out [Amplify](https://aws.amazon.com/amplify) and [MongoDB Atlas](https://www.mongodb.com/products/platform/atlas-cloud-providers/aws). 


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta049d55e02416f45/67476b746cdeea5d78739042/1_image.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta216e9c18d842ecd/67476b7590460d1dc6eec0c1/2_image.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt96b97ad3d3a74d5b/67476b75de20c047b3d2e24d/3_image.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt84a5be280ad41dc8/67476b74e19373eaeca5c960/4_image7.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0ad8b3e59f91dcd2/67476b743aff4e4673d5ffaf/5_image.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt13b8f42aef4040fc/67476b744dc5f075b29ba75e/6_image.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt47ec9c14b9922001/67476b749a83052898868d03/7_image.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta3205cb067033e6e/67476b7546524214bfd349fa/8_image.png

This article and repository provides a starter template for building applications with React and AWS Amplify, seamlessly connecting to MongoDB Atlas. It simplifies the setup for authentication, API integration, and database capabilities.

Fullstack Application Development with AWS Amplify, AppSync, and MongoDB Atlas

Venkatesh Shanbhag

Partner Solution Architect

[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration service. It simplifies data processing for customers. Migrating data between MongoDB Atlas and AWS becomes efficient with AWS Glue's visual ETL (Extract, Transform, and Load) capabilities. The visual interface and built-in features allow data extraction and insertion to and from MongoDB Atlas collections, with optional transformations, before loading it to the target location. This serverless and scalable approach ensures cost-effective data movement while maintaining security through IAM roles. In this post, we will go through a visual approach to utilize MongoDB Atlas and AWS Glue for building pipelines between the two platforms. 

To follow this post and test out AWS Glue’s capabilities with MongoDB Atlas, we need an [AWS account](https://aws.amazon.com/free/) and a subscription to MongoDB Atlas. You can subscribe to MongoDB Atlas from the [AWS marketplace](https://aws.amazon.com/marketplace/seller-profile?id=c9032c7b-70dd-459f-834f-c1e23cf3d092). 

The post describes a process for transferring data between MongoDB Atlas and [AWS S3](https://aws.amazon.com/s3/) using AWS Glue's visual ETL capabilities. These capabilities allow developers to create ETL pipelines without the knowledge of Spark or SQL by leveraging [AWS Glue Studio](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html). This post highlights the benefits of using AWS Glue for data transformation and integration with other AWS services. AWS S3—being highly scalable, durable, and cost-effective object storage—can be used as data lakes, a data warehousing solution, machine learning, media streaming, backup and recovery, and web hosting.


## Set up MongoDB Atlas



1. Configure a MongoDB cluster on AWS. For instructions, refer to [How to Set Up a MongoDB Cluster](https://www.mongodb.com/basics/clusters/mongodb-cluster-setup).
2. Configure PrivateLink by following the steps described in [Connecting Applications Securely to a MongoDB Atlas Data Plane with AWS PrivateLink](https://aws.amazon.com/blogs/apn/connecting-applications-securely-to-a-mongodb-atlas-data-plane-with-aws-privatelink/). With AWS PrivateLink, we will simplify our networking architecture and make sure the traffic stays on the AWS network.
3. To obtain your MongoDB cluster connection string from the Connect UI on the MongoDB Atlas console, navigate to your Atlas home screen and click on **Connect** for the AWS cluster you want to connect. Select the Private Endpoint and Connection Method.
4. Copy the SRV connection string. We use this SRV connection string in the subsequent steps.

![Image 1: Connect to MongoDB Atlas using connection string][1]

The following screenshot shows that we have loaded a sample collection (in this case, sample weather data) in MongoDB Atlas, which we will connect to in the next steps. Note: The records in this collection include several arrays as well as nested data.

![Image 2: MongoDB Sample dataset][2]

## Set up the MongoDB Atlas connection with AWS Glue

Before we can configure the AWS Glue crawler, we need to create the MongoDB Atlas connection in AWS Glue.

1. On the AWS Glue Studio console, choose **Connectors** in the navigation pane.
2. Choose **Create connection**.

![Image 3: AWS Glue Data Connections interface][3]

1. When filling out the connection details, use the SRV connection string we obtained earlier in MongoDB Atlas.
2. In the Network options section, add the VPC and subnet. Important: The VPC and the subnet must correspond to the PrivateLink settings you configured earlier.

![Image 4: Create Glue connection for MongoDB Atlas][4]

## **Create and run an ETL job to extract data from MongoDB Atlas into S3**

Once the connection is configured, Navigate to the Glue | ETL jobs | Visual ETL to create the ETL job. 

To open the Glue visual editor, either click on **Visual ETL** or **Author and edit ETL jobs**, and then click on **visual ETL** on the next screen. 

![Image 5: Navigation to AWS Glue ETL interface][5] 

If you have saved jobs, you can access them from the AWS Glue studio, as well. 

![Image 6: Creating a Visual ETL job on AWS Glue][6] 

Click on **(+)** on the home screen of the Glue visual editor to add nodes. Search for MongoDB and select MongoDB as the source to read from the previously created connection. 

![Image 7: MongoDB connector on AWS Glue visual ETL][7] 

Select the MongoDB connection and provide the database and collection name. Save the pipeline. 

![Image 8: Configuring the connector on AWS Glue Visual ETL][8]

Click on **(+)** to add one more node and search for S3. Select Amazon S3 as Target. Select MongoDB as Node parent.

![Image 9: Configuring S3 bucket connection AWS Glue Visual ETL][9]

Select the data format for your destination data file. Select the S3 bucket where you want to write the data and click on **Save**.

![Image 10:  Configuring S3 bucket connection AWS Glue Visual ETL][10]

Run the job by clicking the button in the top right corner. 

![Image 11: Run ETL job to move data between MongoDB Atlas and AWS S3][11]

The job will take a few minutes to complete. You can monitor your job on the **Runs** tab, as shown below.

![Image 12: Visualizing the ETL job running][12]

You can verify your data written to S3 by navigating to your S3 bucket.

![ Image 13: Verify the the document moved to S3 bucket in parquet format][13]

Watch this Video to see the steps in action 

:youtube[]{vid=s6tcfJmCfOk}

AWS Glue Visual ETL simplifies creating and managing data transformations, allowing developers to create ETL pipelines without the specialized knowledge of data engineering tools.  It offers connectors to numerous third-party and AWS-native products and services. This enables you to enrich data from various sources for analysis in data warehousing, while building efficient pipelines effortlessly with Glue Visual ETL. For advanced data transformations involving MongoDB Atlas, refer to the AWS Glue [documentation](https://docs.aws.amazon.com/glue/latest/dg/setting-up.html). 

> Questions? Join us in the [MongoDB Developer Community](https://www.mongodb.com/community/forums/).


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4a9fa2fff6d9b885/673fc6d8465242c67fd32c47/1_image9.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt970ab5fff863c851/673fc6d89a8305490a866d95/2_image1.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt22a70c1fe48d3154/673fc6d86cdeea0ea5737347/3_image10.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt00d124b5c42f118c/673fc6d8cf4732173a8c0809/4_image6.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltbc5a422b09e24d5d/673fc6d856ed541fae12a76b/5_image12.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt494761313ed4d15f/673fc6d8cf47324a718c080d/6_image2.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt891755349ade2bb9/673fc6d91621f678eb6e565c/7_image13.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd93f53ca66c2cae6/673fc6d890460d596ceea3be/8_image7.png
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcfa835145afef9da/673fc6d827811852005d4c20/9_image8.png
  [10]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcdd49bd11d835f96/673fc72456ed5472ed12a774/10_image3.png
  [11]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt6a05686e42a4c2dd/673fc7244b282e80cc80d01c/11_image4.png
  [12]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt017f5312b77c02c3/673fc72412a5ea220b93319b/12_image11.png
  [13]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blted39ae527c5bb575/673fc72456ed548c5812a772/13_image5.png

This post highlights the benefits of using AWS Glue for your data in MongoDB Atlas for data transformation and integration with other AWS services. 

Anuj Panchal

Content by Anuj Panchal

Fullstack Application Development with AWS Amplify, AppSync, and MongoDB Atlas

AWS Glue Visual ETL for Your Data in MongoDB Atlas