MongoDB Blog
Announcements, updates, news, and more
Top 4 Reasons to Use MongoDB 8.0
October 2, 2024
Updates
Announcing Hybrid Search Support for LlamaIndex
MongoDB is excited to announce enhancements to our LlamaIndex integration. By combining MongoDB’s robust database capabilities with LlamaIndex’s innovative framework for context-augmented large language models (LLMs), the enhanced MongoDB-LlamaIndex integration unlocks new possibilities for generative AI development. Specifically, it supports vector (powered by Atlas Vector Search ), full-text (powered by Atlas Search ), and hybrid search, enabling developers to blend precise keyword matching with semantic search for more context-aware applications, depending on their use case. Building AI applications with LlamaIndex LlamaIndex is one of the world’s leading AI frameworks for building with LLMs. It streamlines the integration of external data sources, allowing developers to combine LLMs with relevant context from various data formats. This makes it ideal for building application features like retrieval-augmented generation (RAG), where accurate, contextual information is critical. LlamaIndex empowers developers to build smarter, more responsive AI systems while reducing the complexities involved in data handling and query management. Advantages of building with LlamaIndex include: Simplified data ingestion with connectors that integrate structured databases, unstructured files, and external APIs, removing the need for manual processing or format conversion. Organizing data into structured indexes or graphs , significantly enhancing query efficiency and accuracy, especially when working with large or complex datasets. An advanced retrieval interface that responds to natural language prompts with contextually enhanced data, improving accuracy in tasks like question-answering, summarization, or data retrieval. Customizable APIs that cater to all skill levels—high-level APIs enable quick data ingestion and querying for beginners, while lower-level APIs offer advanced users full control over connectors and query engines for more complex needs. MongoDB's LlamaIndex integration Developers are able to build powerful AI applications using LlamaIndex as a foundational AI framework alongside MongoDB Atlas as the long term memory database. With MongoDB’s developer-friendly document model and powerful vector search capabilities within MongoDB Atlas, developers can easily store and search vector embeddings for building RAG applications. And because of MongoDB’s low-latency transactional persistence capabilities, developers can do a lot more with MongoDB integration in LlamIndex to build AI applications in an enterprise-grade manner. LlamaIndex's flexible architecture supports customizable storage components, allowing developers to leverage MongoDB Atlas as a powerful vector store and a key-value store. By using Atlas's Vector Search capabilities, developers can: Store and retrieve vector embeddings efficiently ( llama-index-vector-stores-mongodb ) Persist ingested documents ( llama-index-storage-docstore-mongodb ) Maintain index metadata ( llama-index-storage-index-store-mongodb ) Store Key-value pairs ( llama-index-storage-kvstore-mongodb ) Figure adapted from Liu, Jerry and Agarwal, Prakul (May 2023). “Build a ChatGPT with your Private Data using LlamaIndex and MongoDB”. Medium. https://medium.com/llamaindex-blog/build-a-chatgpt-with-your-private-data-using-llamaindex-and-mongodb-b09850eb154c Adding hybrid and full-text search support Developers may use different approaches to search for different use cases. Full-text search retrieves documents by matching exact keywords or linguistic variations, making it efficient for quickly locating specific terms within large datasets, such as in legal document review where exact wording is critical. Vector search, on the other hand, finds content that is ‘semantically’ similar, even if it does not contain the same keywords. Hybrid search combines full-text search with vector search to identify both exact matches and semantically similar content. This approach is particularly valuable in advanced retrieval systems or AI-powered search engines, enabling results that are both precise and aligned with the needs of the end-user. It is super simple for developers to try out powerful retrieval capabilities on their data and improve the accuracy of their AI applications with this integration. In the LlamaIndex integration, the MongoDBAtlasVectorSearch class is used for vector search. All you have to do is enable full-text search, using VectorStoreQueryMode.TEXT_SEARCH in the same class. Similarly, to use Hybrid search, enable VectorStoreQueryMode.HYBRID . To learn more, check out the GitHub repository . With the MongoDB-LlamaIndex integration’s support, developers no longer need to navigate the intricacies of Reciprocal Rank Fusion implementation or to determine the optimal way to combine vector and text searches—we’ve taken care of the complexities for you. The integration also includes sensible defaults and robust support, ensuring that building advanced search capabilities into AI applications is easier than ever. This means that MongoDB handles the intricacies of storing and querying your vectorized data, so you can focus on building! We’re excited for you to work with our LlamaIndex integration. Here are some resources to expand your knowledge on this topic: Check out how to get started with our LlamaIndex integration Build a content recommendation system using MongoDB and LlamaIndex with our helpful tutorial Experiment with building a RAG application with LlamaIndex, OpenAI, and our vector database Learn how to build with private data using LlamaIndex, guided by one of its co-founders
Strengthen Data Security with MongoDB Queryable Encryption
MongoDB Queryable Encryption is a groundbreaking, industry-first innovation developed by the MongoDB Cryptography Research Group that allows customers to encrypt sensitive application data, store it securely in an encrypted state in the MongoDB database, and perform equality and range queries directly on the encrypted data—with no cryptography expertise required. Adding range query support to Queryable Encryption significantly enhances data retrieval capabilities by enabling more flexible and powerful searches. Queryable Encryption is available in MongoDB Atlas, Enterprise Advanced, and Community Edition. Encryption: Protecting data through every stage of its lifecycle Encryption is a critical security method for ensuring protection of sensitive data and compliance with regulations like GDPR, CCPA, and HIPAA. It involves rendering data unreadable to anyone without the decryption key. It can protect data in three ways: in-transit (over networks), at-rest (when stored), and in-use (during processing). While encryption in-transit and at-rest are standard for all databases and are well-supported by MongoDB , encryption in-use presents a unique challenge. Encryption in-use is difficult because encrypted data is unreadable—it looks like random characters and symbols. Traditionally, the database can’t run queries on encrypted data without decrypting it first to make it readable. However, if the database doesn’t have a decryption key, it has to send encrypted data back to the application or system (i.e., the client) that has the key so it can be decrypted before querying. This is a pattern that doesn’t scale well for real-world applications. This puts organizations in a difficult spot: in-use encryption is important for data privacy and regulatory compliance, but it's hard to implement. In the past, companies have either chosen not to encrypt sensitive data in-use or have employed less secure workarounds that complicate their operations. MongoDB Queryable Encryption: Safeguarding data in use without sacrificing efficiency MongoDB Queryable Encryption solves this problem. It allows organizations to encrypt their sensitive data, like personally identifiable information (PII) or protected health information (PHI), and to run equality and range queries directly on that data without having to decrypt it. Queryable Encryption was developed by the MongoDB Cryptography Research Group , drawing on their pioneering expertise in cryptography and encrypted search, and Queryable Encryption has been peer-reviewed by leading cryptography experts worldwide. Unmatched in the industry, MongoDB is the only data platform that allows customers to run expressive queries directly on non-deterministically encrypted data. This represents a groundbreaking advantage for customers, allowing them to maintain robust protection for their sensitive data without sacrificing operational efficiency or developer productivity by still enabling expressive queries to be performed on it. Organizations of all sizes, across all industries, can benefit from the impactful outcomes enabled by Queryable Encryption, such as: Stronger data protection: Data stays encrypted at every stage—whether in-transit, at-rest, or in-use—reducing the risk of sensitive data exposure or breaches. Enhanced regulatory compliance: Provides customers with the necessary tools to comply with data protection regulations like GDPR, CCPA, and HIPAA by ensuring robust encryption at every stage. Streamlined operations: Simplifies the encryption process without needing costly custom solutions, specialized cryptography teams, or complex third-party tools. Solidified separation of duties: Supports stricter access controls, where MongoDB and even a customer's database administrators (DBAs) don’t have access to sensitive data. Use cases for Queryable Encryption MongoDB Queryable Encryption has many use cases for organizations that host sensitive data, regardless of their size or industry. The recent addition of range query support to Queryable Encryption broadens those use cases even wider. Here are some examples to help illustrate how Queryable Encryption could be used to protect and query sensitive data: Financial Services Credit Scoring: Assess creditworthiness by querying encrypted data such as credit scores and income levels. For example, segment your customers based on credit scores between 600 and 750. Fraud Detection: Detect anomalies by querying encrypted transaction amounts for values that exceed typical spending patterns, such as transactions above $10,000. Insurance Risk Assessment: Personalize policy offerings by querying encrypted client data for risk levels within specified ranges, enhancing customer service without exposing sensitive information. Claims Processing: Automate claims processing by querying encrypted claims data for amounts within specific ranges or for claims within time periods, streamlining operations while safeguarding information. Healthcare Medical Research: Execute range-based searches on encrypted medical records, such as querying encrypted datasets for patients within specific age ranges or for abnormal lab results for medical research. Billing and Insurance Processing: Perform secure range queries on encrypted billing data to process insurance claims and payments while protecting patient financial details. Education Grading Systems: Process encrypted student scores to award grades within specific ranges, ensuring compliance with FERPA while protecting student privacy and maintaining data security. Financial Aid Distribution: Analyze encrypted income data within certain ranges to determine eligibility for scholarships and financial aid. Comprehensive data protection at every stage With Queryable Encryption, MongoDB offers unmatched protection for sensitive data throughout its entire lifecycle—whether in-transit, at-rest, or in-use. Now, with the addition of range query support, Queryable Encryption meets even more of the demands of modern applications, unlocking new use cases. To get started, explore the Queryable Encryption documentation .
Unlocking Seamless Data Migrations from Cosmos DB to MongoDB Atlas with Adiom
As enterprises continue to scale, the need for powerful, seamless data migration tools becomes increasingly important. Adiom , founded by industry veterans with deep expertise in data mobility and distributed systems, is addressing this challenge head-on with its open-source tool, dsync. By focusing on high-stakes, production-level migrations, Adiom has developed a solution that works effortlessly with MongoDB Atlas and makes large-scale migrations to it from Cosmos DB and other NoSQL databases faster, safer, and more predictable. The real migration struggles Enterprises often approach migrations with apprehension, and for good reason. When handling massive datasets powering mission-critical services or user-facing applications, even small mistakes can have significant consequences. Adiom understands these challenges deeply, particularly when migrating to MongoDB Atlas. Here are a few of the common pain points that enterprises face: Time-consuming processes: Moving large datasets involves extensive planning, testing, and iteration. What’s more, enterprises need migrations that are repeatable and can handle the same dataset efficiently multiple times—something traditional tools often struggle to provide. Risk management: From data integrity issues to downtime during the migration window, the stakes are high. Tools that worked for smaller datasets and in lower-tier environments no longer meet the requirements. Custom migration scripts often introduce unforeseen risks, while databases like Cosmos DB come with their own unique limitations. Cost overruns: Enterprises frequently encounter hidden migration costs—whether it's the need to provision special infrastructure, reworking application code for compatibility with migration plans, or paying SaaS vendors by the row. These complications can balloon the overall migration budget or send the project into the approval death spiral. To make things even more complicated, the pains feed into each other. The longer the project takes, the more risks need to be accounted for, the longer the planning and testing, and the bigger the cost. Adiom’s dsync: Power and simplicity in one tool Dsync was built with these challenges in mind. Designed specifically for large production workloads, dsync enables enterprises to handle complex migrations more easily, lowering the hurdles that typically slow down the process, reducing risks and uncertainty. Here’s why dsync stands out: Ease of deployment: Starting with dsync is incredibly simple. All it takes is downloading a single binary—there’s no need for specialized infrastructure, and it runs seamlessly on VMs or Docker. Users can monitor migrations through the command line or a web interface, giving flexibility depending on the team’s preferences. Resilience and Safety: dsync is not only efficient, but it’s also resumable. Should a migration be interrupted, there’s no need to start over. This means that migrations can continue smoothly from where they left off, reducing the risk of downtime and minimizing the complexity of the process. Verification: dsync is designed to protect the integrity of migrated data. Dsync features embedded data verification mechanisms that automatically check for consistency between the source and destination databases after migration. Security: dsync doesn't store data, doesn't send it outside the organization other than to the designated destination, and supports network encryption. No hidden costs: As an open-source tool, dsync eliminates the need to onboard expensive SaaS solutions or purchase licenses in the early stages of the process. It operates independently of third-party vendors, giving enterprises flexibility and control over their migrations without the additional financial burden. Enhancing MongoDB customers' experiences For MongoDB customers, the ability to migrate data quickly and efficiently can be the key to unlocking new products, features, and cost savings. With dsync, Adiom provides a solution that can accelerate migrations, reduce risks, and enable enterprises to leverage MongoDB Atlas without the usual headaches. Faster time-to-market: By significantly accelerating migrations, dsync allows companies to take advantage of MongoDB Atlas offerings and integrations sooner, offering a direct path to quicker returns on investment. Self-service and support: Many migrations can be handled entirely in-house, thanks to dsync’s intuitive design. However, for organizations that need additional guidance, Adiom offers support and has partnered with MongoDB Professional Services and PeerIslands to offer comprehensive coverage during the migration process. Five compelling advantages of migrating to MongoDB Flexible schema: MongoDB’s schema-less design reduces development time by up to 30% by allowing you to change data structures. Scalability: You can scale MongoDB to multiple petabytes of data seamlessly using sharding. High performance: MongoDB helps to improve read and write speeds by up to 50% compared to traditional databases. Expressive Query API: Its advanced querying capabilities reduce query writing time and increase execution efficiency by 70%. Partner Ecosystem: MongoDB’s strong partner ecosystem helps with service integrations, AI capabilities, purpose-built solutions, and other significant competitive differentiators. Conclusion Dsync is more than just a migration tool—it’s a powerful engine that abstracts away the complexity of managing large datasets across different systems. By seamlessly tying together initial data copying, change-data-capture, and all the nuances of large-scale migrations, dsync lets enterprises focus on building their future, not on the logistics of data transfer. For those interested in technical details, some of those logistics and nuances can be found in our CEO’s blog . With Adiom and dsync, enterprises no longer have to choose between performance, correctness, or ease of use when planning a migration from Cosmos DB or another NoSQL database. Whether you’re moving from Cosmos DB or another NoSQL database, dsync provides an enterprise-grade solution that helps to enable faster, more secure, and more reliable migrations. By partnering with MongoDB, Adiom supports you in continuing to innovate without being held back by the limitations of legacy databases. Try dsync yourself or contact Adiom for a demo. Head over to our product page to learn more about MongoDB Atlas .
From Chaos to Control: Real-Time Data Analytics for Airlines
Delays are a significant challenge for the airline industry. They disrupt travel plans, erode customer loyalty, and inflict significant financial losses. In an industry built on precision and punctuality, even minor setbacks can have cascading effects. Whether due to adverse weather conditions or unforeseen technical issues, these delays ripple through flight schedules, affecting both passengers and operations managers. While neither group is typically at fault, the ability to quickly reallocate resources and return to normal operations is crucial. To mitigate these disruptions and restore passenger trust, airlines must have the tools and strategies to quickly identify delays and efficiently reallocate resources. This blog explores how a unified platform with real-time data analysis can be a game-changer in this regard especially in saving costs. The high cost of delays Delays from disruptions, like weather events or crew unavailability, pose major challenges for the airline industry. These delays have significant financial impact according to some studies, costing European airlines on average € 4,320 per hour per flight . They also create operational challenges like crew disruptions and reduced airplane availability, leading to further delays, which is known in the industry as delay propagation. To address these challenges, airlines have traditionally focused on optimizing their pre-flight planning processes. However, while planning is crucial, effective recovery strategies are equally essential for minimizing the impact of disruptions. Unfortunately, many airlines have underinvested in recovery systems, leaving them ill-prepared to respond to unexpected events. The consequences of this imbalance include: Delay propagation: Initial delays can cascade, causing widespread schedule disruptions. Financial and operational damage: Increased costs and inefficiencies strain airline resources. Customer dissatisfaction: Poor disruption management leads to negative passenger experiences. The power of real-time data analysis In response to the significant challenges posed by flight delays, a real-time unified platform offers a powerful solution designed to enhance how airlines manage disruptions. Event-driven architectural approach The diagram below showcases an event-driven architecture that can be used to build a robust and decoupled platform that supports real-time data flow between microservices. In an event-driven architecture, services or components communicate by producing and consuming events, which is why this architecture relies on Pub/Sub (messaging middleware) to manage data flows. Moreover, MongoDB’s flexible document model and ability to handle high volumes of data make it ideal for event-driven systems. Combining these features with PubSub’s, this approach proves to offer a powerful solution for modern applications that require scalability, flexibility, and real-time processing. Figure 1: Application architecture In this architecture, the blue line in the diagram shows the operational data flow. The data simulation is triggered by the application’s front-end and is initialized in the FastAPI microservice. The microservice, in turn, starts publishing airplane sensor data to the custom Pub/Sub topics, which forwards these data to the rest of the architecture components, such as cloud functions, for data transformation and processing. The data is processed in each microservice, including the creation of analytical data, as shown by the green lines in the diagram. Afterward, data is introduced in MongoDB and fetched from the application to provide the user with organized, up-to-date information regarding each flight. This leads to more precise and detailed analysis of real-time data for flight operations managers. As a result, new and improved opportunities for resource reallocation can be explored, helping to minimize delays and reduce associated costs for the company. Microservice overview As mentioned earlier, the primary goal is to create an event-driven, decoupled architecture founded on MongoDB and Google Cloud services integrations. The following components contribute to this: FastAPI: Serves as the main data source, generating data for analytical insights, predictions, and simulation. Telemetry data: Pulls and transforms operational data published in the PubSub topic in real-time, storing it in a MongoDB time series collection for aggregation and optimization. Application data: Subscribed to a different PubSub topic, this service acknowledges static operational data, including initial route, recalculated route, and disruption status. Therefore, this service will only be triggered provided any of the previous fields are altered. Finally, this data is updated in its MongoDB collection accordingly. Vertex AI integration—analytical data flow: A cloud function triggered by PubSub messages that executes data transformations and forwards data to the Vertex AI deployed machine learning (ML) model. Predictions are then stored in MongoDB. MongoDB: A flexible, scalable, and real-time data solution Building a unified real-time platform for the airline industry requires efficient management of massive, diverse datasets. From aircraft sensor data to flight cost calculations, data processing and management are central to operations. To meet these demands, the platform needs a flexible data platform capable of handling multiple data types and integrating with various systems. This enables airlines to extract valuable insights from their data and develop features that improve operations and the passenger experience. Real-time data processing is a must-have feature. This allows airlines to receive immediate alerts about delays, minimizing disruptions and ensuring smooth operations. In fast-paced airport environments, where every minute counts, real-time data processing is indispensable. For example, integrating MongoDB with Google Cloud's Vertex AI allows for the real-time processing and storage of airplane sensor data, transforming it into actionable insights. Business benefits This solution provides real-time access to critical flight data, enabling efficient cost management and operational planning. Immediate access to this information allows flight operation managers to plan ahead, reallocate existing resources, or even initiate recovery procedures in order to diminish the consequences of the identified delay. Moreover, its ML model customization ensures adaptability to various use cases. Regarding the platform’s long-term sustainability, it has been purposely designed to integrate highly reliable and scalable products in order to excel in three key standards: Scalability The platform’s compatibility with both horizontal and vertical scaling is clearly demonstrated by its integral design. The decoupled architecture illustrates how this solution is divided into different components—and therefore instances—that work together as a cohesive whole. Vertical scalability can be achieved by simply increasing the computing power allocated to the designed Vertex AI model, if needed. Availability The decoupled architecture exemplifies the central importance of availability in any project’s design. Using different tracks to introduce operational and analytical data into the database allows us to handle any issues in a way that remains unnoticeable to end users. Latency Optimizing the connections between components and integrations within the product is key to achieving the desired results. Using PubSub as our asynchronous messaging service helps minimize unnecessary delays and avoid holding resources needlessly. Get started! To sum up, this blog has explored how MongoDB can be integrated into an airline flight management system, offering significant benefits in terms of cost savings and enhanced customer experience. Check out our AI resource page to learn more about building AI-powered apps with MongoDB, and try out the demo yourself via this repo . To learn more and get started with MongoDB Vector Search, visit our Vector Search Quick Start page .
Grab Drives 50% Efficiencies with MongoDB Atlas
Grab is Southeast Asia’s leading ‘super application,’ offering a wide range of services, targeting both consumers and businesses, including deliveries, mobility, financial services, enterprise and more. Their range of applications, such as the popular Grab Taxi, Grab Pay, Grab Mart, Grab Ads, and more, count approximately 38 million active users monthly across 500 cities and eight countries. Managing a high volume of constantly growing users and handling regular spikes in demand and activity means that Grab needs to maintain a robust, scalable, and flexible digital infrastructure. Presenting at MongoDB.local Singapore in 2024, Grab shared their journey of migrating one of their key service apps— GrabKios —from the Community Edition of MongoDB to MongoDB Atlas . Grab also described how they are expanding their use of MongoDB to support semantic search. “Transitioning to MongoDB Atlas was not just a migration—it was a strategic move aimed at enhancing our database infrastructure,” said Jude Dulaj Lakshan De Croos, Database Engineering Manager at Grab. A smooth transition to MongoDB Atlas Grab’s journey with MongoDB Atlas began with the realization that their existing database infrastructure, while functional, was not equipped to handle the scale and complexity of their operations. Grab’s eventual migration to MongoDB Atlas was meticulously planned and executed, including extensive testing to ensure a smooth transition. During the critical testing phase, the creation of a replica “prod clone” environment, allowed Grab to test and refine their migration strategy. This minimized the possibility of unforeseen issues. The migration also involved the use of Mongomirror . This facilitated the seamless transfer of data from Grab’s self-hosted clusters to MongoDB Atlas. “We were able to ensure that migration was actually smooth and ran without any issues,” said Swarit Arora, Senior Database Engineer at Grab. MongoDB Atlas’s developer data platform offers Grab high levels of flexibility and scalability, accommodating Grab’s fast growth (the company recorded a 23% revenue growth YoY in 2024) in an ever-changing digital landscape. MongoDB Atlas also delivers unique automation and streamlining capabilities, as well as enterprise-grade support which led to improved process and database management efficiency. Efficiency gains with greater scalability, flexibility, performance MongoDB Atlas provided Grab with an automated, scalable, and secure platform, which empowered its engineering teams to focus on product development rather than database maintenance. “With MongoDB Atlas, we don’t have to worry about the scaling changes. And with hands-on security we can deliver secure and fast applications,” said Arora. “Being able to configure the exact resources required and then scale up and down based on our requirements is a plus. Considering we don't have to manage the scalability part, this is, I think, saving us around 50% of the time.” Furthermore, MongoDB Atlas delivers proactive recommendations to Grab’s team. For example, MongoDB Atlas’s Performance Advisor saves the team time by delivering real-time insights and recommendations to optimize query performance, ultimately reducing manual management tasks and increasing database efficiency. “It is now easy to set up our MongoDB clusters compared to what we were doing when we self-hosted, which was more time-consuming,” added Arora. “Secondly, if we are required to upgrade the cluster version, it is as easy as the click of a button.” Dedicated analytics nodes mean that Grab’s team is able to enhance the analytical capabilities of any application running on MongoDB. The successful migration to MongoDB Atlas has positioned Grab to explore new possibilities, including leveraging MongoDB’s advanced features for use cases such as semantic search and AI applications. Learn more about MongoDB Atlas .
MongoDB Atlas + PowerSync: The Future of Offline-First Enterprise Apps
Picture this: your field team is miles from the nearest cell tower, but their apps still work flawlessly—tracking assets, updating data, and syncing the moment they're back online. Since even a second of downtime can cost millions, the future of enterprise apps is one where "offline" doesn't mean "out of commission." Enter MongoDB Atlas and PowerSync, two organizations behind cutting-edge, always-on experiences. The recently announced MongoDB Atlas-PowerSync integration delivers seamless performance and real-time data syncing that keeps businesses running—even when internet connectivity is spotty. By pairing MongoDB Atlas (the gold standard for cloud databases) with PowerSync (a game-changing sync engine) enterprises can create fully offline-first applications that keep teams productive, no matter where they are. Across a variety of industries— from energy, to manufacturing, to field services, to retail—the new integration delivers smooth performance and reliable bi-directional data sync between backend and frontend systems. We are excited to bring support for MongoDB into PowerSync, and to offer MongoDB customers a proven enterprise-grade sync engine to drive their offline-first applications. Conrad Hofmeyr, CEO of JourneyApps/PowerSync MongoDB Atlas + PowerSync: A power move The partnership between MongoDB Atlas and PowerSync offers a new level of app performance, especially for those that need to operate seamlessly in both online and offline environments. PowerSync transforms your applications by enabling data synchronization between MongoDB on the backend and SQLite on the front end. This means your apps don’t just cache data while offline—they remain fully operational. Users can perform queries, update records, and sync millions of objects, all while disconnected. As soon as they reconnect, the data syncs effortlessly with MongoDB Atlas, ensuring that no data is lost in the process. This solution has been refined over a decade, delivering enterprise-grade reliability and scalability. With MongoDB Atlas as the backend, it brings the stability and compliance that enterprises demand. Both PowerSync and MongoDB Atlas are SOC 2 Type 2 audited, ensuring that your data remains secure and compliant with even the strictest regulatory standards. Whether you’re handling a small team or synchronizing millions of users, this combination scales effortlessly, making it a perfect fit for teams of any size. What’s more, integrating PowerSync into your system doesn’t require a major overhaul. Using MongoDB’s change streams, PowerSync delivers high-performance syncing without the need for configuring complex database triggers. The integration is seamless and designed to minimize disruption, making it a low-stress solution for developers looking for a high-efficiency sync tool. Finally, PowerSync gives developers complete control over how data is handled. Whether you want to sync immediately or start with a local-first architecture, PowerSync allows for full customization, letting you determine what data is synced and when business logic is applied. No matter your stack or environment, PowerSync is designed to adapt to your specific needs, providing the flexibility and control to create robust, scalable applications. Where this combo shines: Real-world use cases Energy & field services: Technicians deep in the field, in low connectivity conditions, but equipped with PowerSync apps. They continue collecting and syncing data, from equipment diagnostics to work orders. Once they reconnect, MongoDB Atlas captures everything in real-time, ensuring no critical data gets lost in the shuffle. Manufacturing: Whether it’s tracking production metrics or running diagnostics on machinery, PowerSync’s offline-first architecture keeps operations running. Workers can continue making updates and querying data locally, with MongoDB Atlas acting as the central hub for oversight and reporting when the network is back up. Utilities & Mining: From power grids to remote mining operations, where outages are common, PowerSync delivers secure, bi-directional syncing that ensures your teams have the most up-to-date information, no matter the conditions. Retail: Network connectivity issues should never prevent app users from completing core retail activities like accepting orders, tracking inventory or making deliveries. Point of sale (POS) apps are typically expected to keep working even when network connectivity is not available. Unlock your enterprise's potential today The ability to maintain operations without interruption is increasingly a necessity. By leveraging the powerful integration of MongoDB Atlas and PowerSync, your enterprise can ensure its applications are always ready to perform, regardless of connectivity challenges. Are you ready to transform your offline operations and unlock the full potential of your data? Visit PowerSync’s migration guide today to explore how to seamlessly transition to this innovative solution and empower your teams with the tools they need to thrive in any environment. Don’t let downtime hold you back—take the first step toward a more resilient, agile future!
Introducing: Multi-Kubernetes Cluster Deployment Support
Resilience and scalability are critical for today's production applications. MongoDB and Kubernetes are both well known for their ability to support those needs to the highest level. To better enable developers using MongoDB and Kubernetes, we’ve introduced a series of updates and capabilities that makes it easier to manage MongoDB across multiple Kubernetes clusters. In addition to the previously released support for running MongoDB replica sets and Ops Manager across multiple Kubernetes clusters, we're excited to announce the public preview release of support for Sharded Clusters spanning multiple Kubernetes clusters (GA to follow in November 2024). Support for deployment across multiple Kubernetes clusters is facilitated through the Enterprise Kubernetes Operator. As a recap for anyone unaware, the Enterprise Operator automates the deployment, scaling, and management of MongoDB clusters in Kubernetes. It simplifies database operations by handling tasks such as backups, upgrades, and failover, ensuring consistent performance and reliability in the Kubernetes environment. Multi-Kubernetes cluster deployment support enhances availability, resilience, and scalability for critical MongoDB workloads, empowering developers to efficiently manage these workloads within Kubernetes. This approach unlocks the highest level of availability and resilience by allowing shards to be located closer to users and applications, increasing geographical flexibility and reducing latency for globally distributed applications. Deploying replica sets across multiple Kubernetes clusters MongoDB replica sets are engineered to ensure high availability, data redundancy, and automated failover in database deployments. A replica set consists of multiple MongoDB instances—one primary and several secondary nodes—all maintaining the same dataset. The primary node handles all write operations, while the secondary nodes replicate the data and are available to take over as primary if the original primary node fails. This architecture is critical for maintaining continuous data availability, especially in production environments where downtime can be costly. Support for deploying MongoDB replica sets across multiple Kubernetes clusters helps ensure this level of availability for MongoDB-based applications running in Kubernetes. Deploying MongoDB replica sets across multiple Kubernetes clusters enables you to distribute your data, not only across nodes in the Kubernetes cluster, but across different clusters and geographic locations, ensuring that the rest of your deployments remain operational (even if one or more Kubernetes clusters or locations fail) and facilitating faster disaster recovery. To learn more about how to deploy replica sets across multiple Kubernetes clusters using the Enterprise Kubernetes Operator, visit our documentation . Sharding MongoDB across multiple Kubernetes clusters While replica sets duplicate data for resilience (and higher read rates), MongoDB sharded clusters divide the data up between shards, each of which is effectively a replica set, providing resilience for each portion of the data. This helps your database handle large datasets and high-throughput operations since each shard has a primary member handling write operations to that portion of the data; this allows MongoDB to scale up the write throughput horizontally, rather than requiring vertical scaling of every member of a replica set. In a Kubernetes environment, these shards can now be deployed across multiple Kubernetes clusters, giving higher resilience in the event of a loss of a Kubernetes cluster or an entire geographic location. This also offers the ability to locate shards in the same region as the applications or users accessing that portion of the data, reducing latency and improving user experience. Sharding is particularly useful for applications with large datasets and those requiring high availability and resilience as they grow. Support for sharding MongoDB across multiple Kubernetes clusters is currently in public preview and will be generally available in November. Deploying Ops Manager across multiple Kubernetes clusters Ops Manager is the self-hosted management platform that supports automation, monitoring, and backup of MongoDB on your own infrastructure. Ops Manager's most critical function is backup, and deploying it across multiple Kubernetes clusters greatly improves resilience and disaster recovery for your MongoDB deployments in Kubernetes. With Ops Manager distributed across several Kubernetes clusters, you can ensure that backups of deployments remain robust and available, even if one Kubernetes cluster or site fails. Furthermore, it allows Ops Manager to efficiently manage and monitor MongoDB deployments that are themselves distributed across multiple clusters, improving resilience and simplifying scaling and disaster recovery. To learn more about how to deploy Ops Manager across multiple Kubernetes clusters using the Enterprise Kubernetes Operator, visit our documentation . To leverage multi-Kubernetes-cluster support, you can get started with the Enterprise Kubernetes Operator .
Building Gen AI with MongoDB & AI Partners | September 2024
Last week I was in London for MongoDB.local London —the 19th stop of the 2024 MongoDB.local tour—where MongoDB, our customers, and our AI partners came together to share solutions we’ve been building that enable companies to accelerate their AI journey. I love attending these events because they offer an opportunity to celebrate our collective achievements, and because it’s great to meet so many (mainly Zoom) friends in person! One of the highlights of MongoDB.local London 2024 was the release of our reference architecture with our MAAP partners AWS and Anthropic , which supports memory-enhanced AI agents. This architecture is already helping businesses streamline complex processes and develop smarter, more responsive applications. We also announced a robust set of vector quantization capabilities in MongoDB Atlas Vector Search that will help developers build powerful semantic search and generative AI applications with more scale—and at a lower cost. Now, with support for the ingestion of scalar quantized vectors, you can import and work with quantized vectors from your embedding model providers of choice, including MAAP partners Cohere, Nomic, and others. A big thank you to all of MongoDB’s AI partners, who continually amaze me with their innovation. MongoDB.local London was another great reminder of the power of collaboration, and I’m excited for what lies ahead as we continue to shape the future of AI together. As the Brits say: Cheers! Welcoming new AI and tech partners In September we also welcomed seven new AI and tech partners that offer product integrations with MongoDB. Read on to learn more about each great new partner! Arize Arize AI is a platform that helps organizations visualize and debug the flow of data through AI applications by quickly identifying bottlenecks in LLM calls and understanding agentic paths. "At Arize AI, we are committed to helping AI teams build, evaluate, and troubleshoot cutting-edge agentic systems. Partnering with MongoDB allows us to provide a comprehensive solution for managing the memory and retrieval that these systems rely on”, said Jason Lopatecki, co-founder and CEO of Arize AI. “With MongoDB’s robust vector search and flexible document storage, combined with Arize’s advanced observability and evaluation tools, we’re empowering developers to confidently build and deploy AI applications." Baseten Baseten provides the applied AI research and infrastructure needed to serve custom and open-source machine learning models performantly, scalably, and cost-efficiently. " We're excited to partner with MongoDB to combine their scalable vector database with Baseten's high-performance inference infrastructure and high-performance models. Together, we're enabling companies to build and deploy generative AI applications, such as RAG apps, that not only scale infinitely but also deliver optimal performance per dollar,” said Tuhin Srivastava, CEO of Baseten. “This partnership empowers developers to bring mission-critical AI solutions to market faster, while maintaining cost-effectiveness at every stage of growth." Doppler Doppler is a cloud-based platform that helps teams manage, organize, and secure secrets across environments and applications that can be used throughout the entire development lifecycle. “Doppler rigorously focuses on making the easy path, the most secure path for developers. This is only possible with deep product partnerships with all the tooling developers have come to love. We are excited to join forces with MongoDB to make zero-downtime secrets rotation for non-relational databases effortlessly simple to set up and maintenance-free,” said Brian Vallelunga, founder and CEO of Doppler. “This will immediately bolster the security posture of a company’s most sensitive data without any additional overhead or distractions." Haize Labs Haize Labs automates language model stress testing at massive scales to discover and eliminate failure modes. This, alongside their inference-time mitigations and observability tools, enables the risk-free adoption of AI. " We're thrilled to partner with MongoDB in empowering companies to build RAG applications that are both powerful yet secure, safe, and reliable,” said Leonard Tang, co-founder and CEO of Haize Labs. “MongoDB Atlas has streamlined the process of developing production-ready GenAI systems, and we're excited to work together to accelerate customers' journey to trust and confidence in their GenAI initiatives." Modal Modal is a serverless platform for data and AI/ML engineers to run and deploy code in the cloud without having to think about infrastructure. Run generative AI models, large-scale batch jobs, job queues, and more, all faster than ever before. “The coming wave of intelligent applications will be built on the potent combination of foundation models, large-scale data, and fast search,” explained Charles Frye, AI Engineer at Modal. “MongoDB Atlas provides an excellent platform for storing, querying, and searching data, from hot new techniques like vector indices to old standbys like lexical search. It's the perfect counterpart to Modal's flexible compute, like serverless GPUs. Together, MongoDB and Modal make it easy to get started with this new paradigm, and then they make it easy to scale it out to millions of users querying billions of records & maxing out thousands of GPUs.” Portkey AI Portkey AI is an AI gateway and observability suite that helps companies develop, deploy, and manage LLM-based applications. " Our partnership with MongoDB is a game-changer for organizations looking to operationalize AI at scale. By combining Portkey's LLMOps expertise with MongoDB's comprehensive data solution, we're enabling businesses to deploy, manage, and scale AI applications with unprecedented efficiency and control,” said Ayush Garg, Chief Technology Officer of Portkey AI. “Together, we're not just streamlining the path from POC to production; we're setting a new standard for how businesses can leverage AI to drive innovation and deliver tangible value." Reka Reka offers fully multimodal models including images, videos with audio, text, and documents to empower AI agents that can see, hear, and speak. "At Reka, we know how challenging it can be to retrieve information buried in unstructured multimodal data. We are excited to join forces with MongoDB to help companies test and optimize multimodal RAG features for faster production deployment,” said Dani Yogatama, CEO of Reka. “Our models understand and reason over multimodal data including text, tables, and images in PDF documents or conversations in videos. Our joint solution streamlines the whole RAG development lifecycle, speeding up time to market and helping companies deliver real values to their customers faster." But wait, there's more! To learn more about building AI-powered apps with MongoDB, check out our AI Resources Hub , and stop by our Partner Ecosystem Catalog to read about our integrations with MongoDB’s ever-evolving AI partner ecosystem.
Introducing Dark Mode for MongoDB Documentation
We’re excited to announce a highly requested feature: Dark mode is now available for MongoDB Documentation ! Every day, developers from all backgrounds—beginners to experts—turn to the MongoDB Documentation. It’s packed with comprehensive resources that help you build modern applications using MongoDB and the Atlas developer data platform. With detailed information and step-by-step guides, it’s an invaluable tool for improving your skills and making your development work smoother. From troubleshooting tricky queries to exploring new features, MongoDB Documentation is there to support your projects and help you succeed. With dark mode, you can now switch to a darker interface that’s easier on the eyes. Whether you’re working late or prefer a subdued color palette, dark mode enhances your MongoDB Documentation experience. How to enable dark mode Enabling dark mode is simple. Just click on the sun and moon icon at the top right of the page to switch between dark mode, light mode, and system settings. It will initially default to your system settings. This is a personal setting and won't affect other users within the project or organization. We’ve designed dark mode to provide the same user-friendly experience you’re used to and stay consistent across different tools in the developer workflow, including MongoDB Atlas, which is also available in dark mode . We're all about making your reading experience top-notch! Dark mode is here because you asked for it through our feedback widget on the Docs page. Whether you’re an early adopter of dark mode or just trying it out, we’d love your opinion. Just drop your feedback in the widget next to the color theme selector on the MongoDB Documentation page. Less strain, more gain Dark mode offers a sleek, modern look that brings a refreshing change from the traditional light mode. Beyond its stylish appearance, dark mode also provides significant practical benefits. Reducing the amount of bright light emitted from your screen helps minimize eye strain and fatigue, making extended periods of device use more comfortable. For those using OLED screens, dark mode can help conserve battery life, as these screens consume less power by displaying darker pixels. Whether you’re coding into the late hours or just looking for a more comfortable viewing experience, dark mode is a simple yet powerful tool to enhance your development experience. Try out dark mode on MongoDB Documentation today and enjoy a more comfortable, stylish, and efficient reading experience!
Introducing Two MongoDB Generative AI Learning Badges
Want to boost your resume quickly? MongoDB is introducing two new Learning Badges , Building gen AI Apps and Deploying and Evaluating gen AI Apps. Unlike high-stakes certifications, which cover a large breadth and depth of subjects, these digital credentials are focused on specific topics, making them easier and quicker to earn. Best of all, they’re free! The Building Gen AI Applications with MongoDB Learning Badge validates users’ knowledge of developing gen AI applications using MongoDB Atlas Vector Search. It recognizes your understanding of semantic search and how to build chatbots with retrieval-augmented generation (RAG), MongoDB, and Langchain. The Deploying and Evaluating Gen AI Applications with MongoDB Learning Badge validates users’ knowledge of optimizing the performance and evaluating the results of gen AI applications. It recognizes your understanding of chunking strategies, performance evaluation techniques, and deployment options within MongoDB for both prototyping and production stages. Learn, prepare, and earn To earn your badge, simply complete the Learning Badge Path and take a short assessment at the end. Once you pass the short assessment, you'll receive an email with your official Credly badge and digital certificate. You can share it on social media, in email signatures, or on digital resumes. Additionally, you'll gain inclusion in the Credly Talent Directory , where you will be visible to recruiters from top employers and can open up new career opportunities. Learning paths are like curated roadmaps that guide you through essential concepts and skills needed for the assessment. Each badge has its own learning path: Building Gen AI Apps Learning Badge Path: This learning path guides you through the foundations of building a gen AI application with MongoDB Atlas Vector Search. You'll learn what semantic search is and how you can leverage it across a variety of use cases. Then you'll learn how to build your own chatbot by creating a RAG application with MongoDB and Langchain. Deploying and Evaluating Gen AI Apps Learning Badge Path: This learning path will help you take a gen AI application from creation to full deployment, with a focus on optimizing performance and evaluating results. You'll explore chunking strategies, performance evaluation techniques, and deployment options in MongoDB for both prototyping and production stages. We recommend completing the Building gen AI Apps Learning Badge Path before beginning this path. Badge up with MongoDB MongoDB Learning Badges offer a valuable opportunity to showcase your commitment to continuous learning and expertise in specific topics. These digital credentials not only recognize your educational achievements but also serve as a testament to your knowledge and skills. Whether you're a seasoned developer, an aspiring data scientist, or an enthusiastic student, earning a MongoDB badge can significantly enhance your profile and open up new opportunities in your field. Start earning your badges today—it’s quick, effective, and free! Visit MongoDB Learning Badges to begin your journey toward becoming a gen AI application expert and boosting your career prospects.
THL Simplifies Architecture with MongoDB Atlas Search
Tourism Holdings Limited (THL) originally became a MongoDB customer in 2019, using MongoDB Atlas to help manage a wide variety of telematics data. I was very excited to welcome Charbel Abdo, Solutions Architect for THL at MongoDB .local Sydney in July 2024 to hear more about how the company has significantly expanded its use of MongoDB. The largest RV rental company in the world, THL has branches in New Zealand (where it is headquartered), Australia, the US, Canada, the UK and Europe. Specializing in building, renting, and selling camper vans, THL has a number of well-known brands under its umbrella. In recent years, THL has made a number of significant digital transformation and technology stack optimization efforts, moving from a ‘bolt-on’ approach that necessitated the use of a distributed search and analytics engine to an integrated search solution with MongoDB Atlas . THL operates a complex ecosystem managed by their in-house platform, Motek, which handles booking, pricing, fleet management, and more—with MongoDB Atlas as the central database. Its +7,000 RVs are fitted with telematics devices that send information—such as location, high-speed events, engine problems, and geofences or restricted areas (for example, during the Australian bushfires of 2020)—to vehicles’ onboard computers. THL initially used a bolt-on approach for complex search functionalities by extending their deployment footprint to include a stand-alone instance of Elasticsearch. This setup, while functional, introduced significant data synchronization and performance issues, as well as increased maintenance overhead. Elasticsearch struggled under heavy loads which led to critical failures and system instability, resulting in THL experiencing frequent outages and data inconsistencies. After two years of coping with these challenges, THL resolved to migrate away from ElasticSearch. After doing due diligence, they identified the MongoDB developer data platform’s integrated Search capabilities as the optimum solution. "A couple of months later, we had migrated everything," said Abdo. "Kudos to the MongoDB account team. They were exceptional." The migration process turned out to be relatively straightforward. By iteratively replacing Elasticsearch with MongoDB Atlas Search , THL was able to simplify its architecture, reduce costs, and eliminate the synchronization issues that had plagued the system. The simplification also led to significant performance and reliability improvements. Because it no longer needed the dedicated sync resources processing millions upon millions of records per day, THL was able to turn off its Elasticsearch cluster and to consolidate its resources. “All data sync related issues were gone, eliminated. But also we got our Friday afternoons back, which is always a good thing!” added Abdo. Abdo’s team can now also use existing monitoring tools rather than having to set up something completely separate from the standalone search engine they were using. “Sometimes, changes are easier than you think,” said Abdo. “We spent two-and-a-half years with our faulty solutions just looking for ways to patch up all the problems that we were having. We tried everything except actually looking into how much it would actually take to migrate. We wasted so much time, so much effort, so much money. While if we had thought about this a couple of years ago, it would have been a breeze.” “Over-engineering is bad, simple is better,” he noted. To learn more about how MongoDB Atlas Search can help you build or deepen your search capabilities, visit our MongoDB Atlas Search page .
Vector Quantization: Scale Search & Generative AI Applications
We are excited to announce a robust set of vector quantization capabilities in MongoDB Atlas Vector Search . These capabilities will reduce vector sizes while preserving performance, enabling developers to build powerful semantic search and generative AI applications with more scale—and at a lower cost. In addition, unlike relational or niche vector databases, MongoDB’s flexible document model—coupled with quantized vectors—allows for greater agility in testing and deploying different embedding models quickly and easily. Support for scalar quantized vector ingestion is now generally available, and will be followed by several new releases in the coming weeks. Read on to learn how vector quantization works and visit our documentation to get started! The challenges of large-scale vector applications While the use of vectors has opened up a range of new possibilities , such as content summarization and sentiment analysis, natural language chatbots, and image generation, unlocking insights within unstructured data can require storing and searching through billions of vectors—which can quickly become infeasible. Vectors are effectively arrays of floating-point numbers representing unstructured information in a way that computers can understand (ranging from a few hundred to billions of arrays), and as the number of vectors increases, so does the index size required to search over them. As a result, large-scale vector-based applications using full-fidelity vectors often have high processing costs and slow query times, hindering their scalability and performance. Vector quantization for cost-effectiveness, scalability, and performance Vector quantization, a technique that compresses vectors while preserving their semantic similarity, offers a solution to this challenge. Imagine converting a full-color image into grayscale to reduce storage space on a computer. This involves simplifying each pixel's color information by grouping similar colors into primary color channels or "quantization bins," and then representing each pixel with a single value from its bin. The binned values are then used to create a new grayscale image with smaller size but retaining most original details, as shown in Figure 1. Figure 1: Illustration of quantizing an RGB image into grayscale Vector quantization works similarly, by shrinking full-fidelity vectors into fewer bits to significantly reduce memory and storage costs without compromising the important details. Maintaining this balance is critical, as search and AI applications need to deliver relevant insights to be useful. Two effective quantization methods are scalar (converting a float point into an integer) and binary (converting a float point into a single bit of 0 or 1). Current and upcoming quantization capabilities will empower developers to maximize the potential of Atlas Vector Search. The most impactful benefit of vector quantization is increased scalability and cost savings through reduced computing resources and efficient processing of vectors. And when combined with Search Nodes —MongoDB’s dedicated infrastructure for independent scalability through workload isolation and memory-optimized infrastructure for semantic search and generative AI workloads— vector quantization can further reduce costs and improve performance, even at the highest volume and scale to unlock more use cases. "Cohere is excited to be one of the first partners to support quantized vector ingestion in MongoDB Atlas,” said Nils Reimers, VP of AI Search at Cohere. “Embedding models, such as Cohere Embed v3, help enterprises see more accurate search results based on their own data sources. We’re looking forward to providing our joint customers with accurate, cost-effective applications for their needs.” In our tests, compared to full-fidelity vectors, BSON-type vectors —MongoDB’s JSON-like binary serialization format for efficient document storage—reduced storage size by 66% (from 41 GB to 14 GB). And as shown in Figures 2 and 3, the tests illustrate significant memory reduction (73% to 96% less) and latency improvements using quantized vectors, where scalar quantization preserves recall performance and binary quantization’s recall performance is maintained with rescoring–a process of evaluating a small subset of the quantized outputs against full-fidelity vectors to improve the accuracy of the search results. Figure 2: Significant storage reduction + good recall and latency performance with quantization on different embedding models Figure 3: Remarkable improvement in recall performance for binary quantization when combining with rescoring In addition, thanks to the reduced cost advantage, vector quantization facilitates more advanced, multiple vector use cases that would have been too computationally-taxing or cost-prohibitive to implement. For example, vector quantization can help users: Easily A/B test different embedding models using multiple vectors produced from the same source field during prototyping. MongoDB’s document model —coupled with quantized vectors—allows for greater agility at lower costs. The flexible document schema lets developers quickly deploy and compare embedding models’ results without the need to rebuild the index or provision an entirely new data model or set of infrastructure. Further improve the relevance of search results or context for large language models (LLMs) by incorporating vectors from multiple sources of relevance, such as different source fields (product descriptions, product images, etc.) embedded within the same or different models. How to get started, and what’s next Now, with support for the ingestion of scalar quantized vectors, developers can import and work with quantized vectors from their embedding model providers of choice (such as Cohere, Nomic, Jina, Mixedbread, and others)—directly in Atlas Vector Search. Read the documentation and tutorial to get started. And in the coming weeks, additional vector quantization features will equip developers with a comprehensive toolset for building and optimizing applications with quantized vectors: Support for ingestion of binary quantized vectors will enable further reduction of storage space, allowing for greater cost savings and giving developers the flexibility to choose the type of quantized vectors that best fits their requirements. Automatic quantization and rescoring will provide native capabilities for scalar quantization as well as binary quantization with rescoring in Atlas Vector Search, making it easier for developers to take full advantage of vector quantization within the platform. With support for quantized vectors in MongoDB Atlas Vector Search, you can build scalable and high-performing semantic search and generative AI applications with flexibility and cost-effectiveness. Check out these resources to get started documentation and tutorial . Head over to our quick-start guide to get started with Atlas Vector Search today.