Multi-Cloud Data Resilience with MongoDB Atlas

Steve Jurczak
October 10, 2023

MongoDB Atlas is architected to ensure that data remains safe and secure at all times, offering automated database resilience against hardware failures and regional outages. One of the reasons why MongoDB Atlas is able to provide high levels of resilience and availability is because it's the only developer data platform that's available on all three major public cloud platforms, AWS, Microsoft Azure, and Google Cloud. In fact, deploying Atlas to multiple cloud providers or regions is a simple matter of choosing how many nodes you want to deploy and in which cloud providers. And, once you're in the cloud, you can span databases cross-cloud without having to worry about setting up complicated ETL processes, so you're never locked into one cloud provider or running the risk of concentrating all your data in a single location.

By utilizing Atlas to distribute data across multiple clouds, businesses can quickly and painlessly achieve high service levels for critical applications with virtually no latency. In the event of an outage, the self-healing process kicks in — automatically electing a secondary member to take the reins within seconds without operations to the database being affected — all without any manual intervention.

Having access to multiple regions across the world provides businesses with the flexibility to adhere to data sovereignty requirements without needing to compromise on availability; cloud providers that offer just one region in a specific geography can leave users vulnerable to system disruptions, but multi-cloud clusters enable organizations to deploy additional nodes in regions not provided by their primary cloud provider. You also have the option, should you need it, of using another cloud provider in countries where a provider may only have one data center in a given region.

Geo-resilience in Atlas

MongoDB Atlas puts you in control of where your data is stored, with more than 110 regions across AWS, Google Cloud, and Microsoft Azure — to ensure your managed databases are close enough to your application servers for fast response times.

Atlas is designed to ensure maximum uptime, no matter which region or cloud provider you're using. It provides built-in geographic resilience for multi-zone, multi-region, and multi-cloud clusters with built-in data resilience features:

  • Atlas takes proactive measures to ensure the resilience of single-region clusters, automatically distributing replica set members across different cloud availability zones for maximum protection and maximum uptime.
  • Atlas helps businesses stay resilient in the face of regional failure by leveraging multi-region clusters to replicate data across geographic boundaries and keep operations running smoothly.
  • For added protection and peace of mind, multi-cloud clusters provide the perfect solution to address cloud provider failure and ensure data replication across multiple clouds.

With AWS Availability Zones, Google Regions and Zones, and Azure Availability Zones, each independent zone is made up of one or more discrete data centers, all equipped with redundant power, and networking.

A cloud region refers to the actual geographical site within a cloud service provider's infrastructure where a cluster or replica set is deployed. Regardless of the number of zones present in a cloud region, MongoDB Atlas will always deploy replica sets with at least three members to ensure the highest levels of availability and data durability.

Atlas gives you complete flexibility when it comes to configuring multi-region protection. A multi-region cluster can be hosted in multiple regions within a single cloud provider or multiple regions across multiple cloud providers.

Cloud provider disruptions can take various forms, from relatively minor capacity constraints to devastating outages that can wreak havoc on your application deployments. To reduce the risk of a major outage, organizations should consider the benefits of distributing their data across multiple clouds to maximize database and application resilience.

With multi-cloud clusters, you can easily access the powerful and unique tools and services within AWS, Google Cloud, and Azure — giving you global reach, low-latency performance, regional data security, and resilient data replication and migration. Plus, Atlas takes the hassle out of the equation, automatically distributing your data across clouds for maximum fault tolerance and giving you the freedom to explore cross-cloud migration options at any time.

Multi-cloud clusters provide the same features as single-cloud, multi-region clusters — such as continuous cloud backups, automated data tiering, and workload isolation for data analytics and visualization — but they also offer the added advantage of increased cross-cloud resilience.

For more information on multi-cloud features in MongoDB Atlas, download our Data Resilience Strategy with MongoDB Atlas whitepaper.

Find out more about deploying multi-cloud clusters from our documentation.

← Previous

Building AI with MongoDB: How Metaphor Data Uses Atlas Vector Search to Change the World Through Data

Since announcing MongoDB Atlas Vector Search in preview back in June, we’ve already seen rapid adoption from developers building a wide range of AI-enabled apps. Today we're highlighting another customer who has increased efficiency while removing architectural complexity by adopting Atlas Vector Search. Metaphor is a search and discovery tool built for data scientists, data engineers, and AI practitioners. The company’s mission is to empower individuals and companies of all types to change the world through data. Metaphor is the next evolution of the Data Catalog with fully automated support for Data Governance, Data Literacy, and Data Enablement using an intuitive user interface. We recently caught up with Mars Lan, Co-founder and CTO to learn more about the company’s journey with MongoDB and their adoption of Atlas Vector Search. Check out our AI resource page to learn more about building AI-powered apps with MongoDB. Tell us a little bit about your company and what you and the team are building We’re an early-stage startup with a mission to empower individuals and organizations to change the world through data. We refer to ourselves as the social platform for data and have a range of products that support both data teams but also data consumers. Our main product is a SaaS Data Catalog that enables governance and enablement of data across the organization. We’re a small team of around 15 or so with a keen focus on product and engineering. The company was founded about 2.5 years ago. What role does search play at the company and where did your search story begin? Well, I will start by saying that we almost ended up having a very different story to tell you than what actually ended up transpiring! We started off our journey using DocumentDB and Elasticsearch on AWS for our database and search needs. After some time we ran into some scalability issues that caused us to evaluate (and eventually move to) MongoDB Atlas for our database needs. When we saw MongoDB offered Atlas Search which was based on the same underlying Lucene technology we got very excited and began the process of migrating our search efforts over to Atlas — and this eventually laid the groundwork for adopting Atlas Vector Search later on. So starting with those initial search needs, what got you excited about Atlas Search with MongoDB? What were your use cases? We started to face a significant amount of maintenance and upkeep associated between our database and Elasticsearch. We previously had to build data pipelines, so if something changed in the database, it would also change in search. Once we eventually migrated everything to MongoDB Atlas Search, we no longer had to manage those pipelines. This resulted in lower latency and less likelihood of bugs, which excited our team. The other component to this was the scalability disconnect of having two different systems. We realized if we ever needed to spin up more storage or compute, we could just spin up a larger MongoDB cluster and get that extra scalability right away with the Atlas platform. Of course one less thing to worry about is also a huge benefit — Elasticsearch is not the easiest thing to manage, so having it all in MongoDB was another big plus for us. How did you initially learn about Atlas Vector Search and what piqued your interest? We started experimenting with Pinecone as the AI stuff really started to explode a while back, just to try out the tool, as one of our interns had initially started playing around with it. It turns out not to be cost-effective to spin up a Pinecone instance for each customer, and quite difficult to scale up due to API throttling. After some time, we started looking around for other vendors for vector search. However, once we learned that MongoDB had Vector Search we got excited at the prospect of being able to use our existing tech stack for this additional functionality. It quickly became a no-brainer to us — since we knew we were going to move everything to Atlas, it became obvious we should just consolidate everything there, so we ended up migrating to Atlas Vector Search for all of our semantic search needs. This means one query API, one set of dependencies, and build in sync, all in a single platform. What were the key factors that made you pull the trigger and adopt Atlas Vector Search? What were the problems you were trying to solve? So one key unlock for us was the semantic search side of things, where someone can ask a natural language question and get a natural language answer. This is a much more preferable user experience for us compared to your Google-style keyword searches. From day one we always wanted to best serve our core customer the engineer, but another huge constituency for us is the business or non-technical audience. These folks prefer a tool that is more intuitive to use. To best serve them we have a first-class integration into Slack and Microsoft Teams, so they can ask a question and don't have to go to another place or switch tools to get that answer. We didn’t always have the capability to do the natural language question and response, but with Atlas Vector Search this now becomes possible. Using Vector Search we now have the ability to ask the Slack bot questions like “where can I find this type of data” or “where is this one table on revenue from last quarter and who is using it” and get a natural language response back. One of the key considerations for us when looking at vendors was cost - but not just cost in terms of what shows up on an invoice. I would rather scale one system and get benefits on both (search and vector search). We saw that having to scale two systems independently was just not going to be very efficient in the long run. Can you talk about some of the initial benefits you’ve seen so far both on the Atlas Search side as well as with Vector Search specifically? How do you think about and quantify these benefits? Well one obvious thing that stands out on the search side is increased speed and being able to move quickly. MongoDB in general has a great developer experience. Our data model tends to be highly complex documents, and all the metadata tends to be highly structured and complex, so the MongoDB model fits us very well. In terms of productivity, it’s never an exact science. I will say that with the adoption of Atlas we were able to keep our engineering team size relatively constant while serving many more customers and scale our development efforts faster — so we probably saw a 2X - 3X increase in productivity. One last item of note. We adopt the most rigorous security practices because we deal with so much customer data, so we want to ensure the highest security possible. We chose to have dedicated MongoDB clusters per customer, so every customer’s data is totally isolated from each other. When we were on Pinecone, this meant spinning up a new Pinecone pod for each customer, which would be both really hard to do and not at all financially viable. Because we are centralizing this all under MongoDB, it becomes so much easier - you can dynamically scale your cluster sizes up and down depending on the needs or requirements of small vs. large customers. There’s not the sort of waste you’d get with multiple discrete systems. Getting started A big thank you to Mars and the entire Metaphor Data team for sharing more about their story and use of Atlas Vector Search. Want to learn more? Head over to our Vector Search page for a variety of resources, or jump straight into our tutorial . And if you’re a startup building with AI please check out our MongoDB AI Innovators program for Atlas credits, one-on-one technical advice, access to our partner network, and more!

October 10, 2023
Next →

Payments Modernization and the Role of the Operational Data Layer

To stay relevant and competitive, payment solution providers must enhance their payment processes to adapt to changing customer expectations, regulatory demands, and advancing technologies. The imperative for modernization is clear: payment systems must become faster, more secure, and seamlessly integrated across platforms. Driven by multiple factors—real-time payments, regulatory shifts like Payment Services Directive 2 (PSD2), heightened customer expectations, the power of open banking, and the disruptive force of fintech startups—the need for payment modernization has never been more pressing. But transformation is not without its challenges. Complex systems, industry reliance on outdated technology, high upgrade costs, and technical debt all pose formidable obstacles. This article will explore modernization approaches and how MongoDB helps smooth transformations. Approaches to modernization As businesses work to modernize their payment systems, they need to overcome the complexities inherent in updating legacy systems. Forward-thinking organizations embrace innovative strategies to streamline their operations, enhance scalability, and facilitate agile responses to evolving market demands. Two such approaches gaining prominence in the realm of payment system modernization are domain-driven design and microservices architecture : Domain-driven design: This approach focuses on a business's core operations to develop scalable and easier-to-manage systems. Domain-driven design ensures that technology serves strategic business goals by aligning system development with business needs. At its core, this approach seeks to break down complex business domains into manageable components, or "domains," each representing a distinct area of business functionality. Microservices architecture: Unlike traditional monolithic architectures, characterized by tightly coupled and interdependent components, a microservices architecture decomposes applications into a collection of loosely coupled services, each of which is responsible for a specific business function or capability. It introduces more flexibility and allows for quicker updates, facilitating agile responses to changing business requirements. Discover how Wells Fargo launched their next-generation card payments by building an operational data store with MongoDB . Modernizing with an operational data layer In the payments modernization process, the significance of an operational data layer (ODL) cannot be overstated. An ODL is an architectural pattern that centrally integrates and organizes siloed enterprise data, making it available to consuming applications. The simplest representation of this pattern looks something like the sample reference architecture below. Figure 1: Operational Data Layer structure An ODL is deployed in front of legacy systems to enable new business initiatives and to meet new requirements that the existing architecture can’t handle—without the difficulty and risk of fully replacing legacy systems. It can reduce the workload on source systems, improve availability, reduce end-user response times, combine data from multiple systems into a single repository, serve as a foundation for re-architecting a monolithic application into a suite of microservices, and more. The ODL becomes a system of innovation, allowing the business to take an iterative approach to digital transformation. Here's why an ODL is considered ideal for payment operations: Unified data management: Payment systems involve handling a vast amount of diverse data, including transaction details, customer information, and regulatory compliance data. An ODL provides a centralized repository for storing and managing this data, eliminating silos and ensuring data integrity. Real-time processing: An ODL enables real-time processing of transactions, allowing businesses to handle high numbers of transactions swiftly and efficiently. This capability is essential for meeting customer expectations for instant payments and facilitating seamless transactions across various channels. Scalability and flexibility: Payment systems must accommodate fluctuating transaction volumes and evolving business needs. An ODL offers scalability and flexibility, allowing businesses to scale their infrastructure as demand grows. Enhanced security: An ODL incorporates robust security features —such as encryption, access controls, and auditing capabilities—to safeguard data integrity and confidentiality. By centralizing security measures within the ODL, businesses can ensure compliance with regulatory requirements and mitigate security risks effectively. Support for payments data monetization: Payment systems generate a wealth of data that can provide valuable insights into customer behavior, transaction trends, and business performance. An ODL facilitates real-time analytics and reporting by providing a unified platform for collecting, storing, and analyzing this data. Transform with MongoDB MongoDB’s fundamental technology principles ensure companies can reap the advantages of microservices and domain-driven design—specifically, our flexible data model and built-in redundancy, automation, and scalability. Indeed, the document model is tailor-made for the intricacies of payment data, ensuring adaptability and scalability as market demands evolve. Here’s how MongoDB helps with domain-driven design and microservice implementation to adopt industry best practices: Ease of use: MongoDB’s document model makes it simple to model or remodel data to fit the needs of payment applications. Documents are a natural way of describing data. They present a single data structure, with related data embedded as sub-documents and arrays, making it simpler and faster for developers to model how data in the application will be mapped to data stored in the database. In addition, MongoDB guarantees the multi-record ACID transactional semantics that developers are familiar with, making it easier to reason about data. Flexibility: MongoDB’s dynamic schema is ideal for handling the requirements of microservices and a domain-driven design. Domain-driven design emphasizes modeling the domain to reflect the business requirements, which may evolve over time. MongoDB's flexible schema allows you to store domain objects as documents without rigid schema constraints, facilitating agile development and evolution of the domain model. Speed: Using MongoDB for an ODL means you can get better performance when accessing data, and write less code to do so. A document is a single place for the database to read and write data for an entity. This locality of data ensures the complete document can be accessed in a single database operation that avoids the need internally to pull data from many different tables and rows. Data access and microservice-based APIs: MongoDB integrates seamlessly with modern technologies and frameworks commonly used in microservices architectures. MongoDB's flexible data model and ability to handle various data types, including structured and unstructured data, is a great fit for orchestrating your open API ecosystem to make data flow between banks, third parties, and consumers possible. Scalability: Even if an ODL starts at a small scale, you need to be prepared for growth as new source systems are integrated, adding data volume, and new consuming systems are developed, increasing workload. MongoDB provides horizontal scale-out on low-cost, commodity hardware or cloud infrastructure using sharding to meet the needs of an ODL with large data sets and high throughput requirements. High availability: Microservices architectures require high availability to ensure that individual services remain accessible even in the event of failures. MongoDB provides built-in replication and failover capabilities, ensuring data availability and minimal downtime in case of server failures. Payment modernization is not merely a trend but a strategic imperative. By embracing modern payment solutions and leveraging the power of an ODL with MongoDB, organizations can unlock new growth opportunities, enhance operational efficiency, and deliver superior customer experiences. Learn how to build an operational data layer with MongoDB using this Payments Modernization Solution Accelerator . Learn more about how MongoDB is powering industries on our solution library .

May 15, 2024