Introducing: Multi-Kubernetes Cluster Deployment Support

Resilience and scalability are critical for today's production applications. MongoDB and Kubernetes are both well known for their ability to support those needs to the highest level. To better enable developers using MongoDB and Kubernetes, we’ve introduced a series of updates and capabilities that makes it easier to manage MongoDB across multiple Kubernetes clusters. In addition to the previously released support for running MongoDB replica sets and Ops Manager across multiple Kubernetes clusters, we're excited to announce the public preview release of support for Sharded Clusters spanning multiple Kubernetes clusters (GA to follow in November 2024).

Support for deployment across multiple Kubernetes clusters is facilitated through the Enterprise Kubernetes Operator. As a recap for anyone unaware, the Enterprise Operator automates the deployment, scaling, and management of MongoDB clusters in Kubernetes. It simplifies database operations by handling tasks such as backups, upgrades, and failover, ensuring consistent performance and reliability in the Kubernetes environment.

Multi-Kubernetes cluster deployment support enhances availability, resilience, and scalability for critical MongoDB workloads, empowering developers to efficiently manage these workloads within Kubernetes. This approach unlocks the highest level of availability and resilience by allowing shards to be located closer to users and applications, increasing geographical flexibility and reducing latency for globally distributed applications.

Deploying replica sets across multiple Kubernetes clusters

MongoDB replica sets are engineered to ensure high availability, data redundancy, and automated failover in database deployments. A replica set consists of multiple MongoDB instances—one primary and several secondary nodes—all maintaining the same dataset. The primary node handles all write operations, while the secondary nodes replicate the data and are available to take over as primary if the original primary node fails. This architecture is critical for maintaining continuous data availability, especially in production environments where downtime can be costly.

Support for deploying MongoDB replica sets across multiple Kubernetes clusters helps ensure this level of availability for MongoDB-based applications running in Kubernetes. Deploying MongoDB replica sets across multiple Kubernetes clusters enables you to distribute your data, not only across nodes in the Kubernetes cluster, but across different clusters and geographic locations, ensuring that the rest of your deployments remain operational (even if one or more Kubernetes clusters or locations fail) and facilitating faster disaster recovery.

To learn more about how to deploy replica sets across multiple Kubernetes clusters using the Enterprise Kubernetes Operator, visit our documentation.

Sharding MongoDB across multiple Kubernetes clusters

While replica sets duplicate data for resilience (and higher read rates), MongoDB sharded clusters divide the data up between shards, each of which is effectively a replica set, providing resilience for each portion of the data. This helps your database handle large datasets and high-throughput operations since each shard has a primary member handling write operations to that portion of the data; this allows MongoDB to scale up the write throughput horizontally, rather than requiring vertical scaling of every member of a replica set. In a Kubernetes environment, these shards can now be deployed across multiple Kubernetes clusters, giving higher resilience in the event of a loss of a Kubernetes cluster or an entire geographic location. This also offers the ability to locate shards in the same region as the applications or users accessing that portion of the data, reducing latency and improving user experience. Sharding is particularly useful for applications with large datasets and those requiring high availability and resilience as they grow.

Support for sharding MongoDB across multiple Kubernetes clusters is currently in public preview and will be generally available in November.

Deploying Ops Manager across multiple Kubernetes clusters

Ops Manager is the self-hosted management platform that supports automation, monitoring, and backup of MongoDB on your own infrastructure. Ops Manager's most critical function is backup, and deploying it across multiple Kubernetes clusters greatly improves resilience and disaster recovery for your MongoDB deployments in Kubernetes.

With Ops Manager distributed across several Kubernetes clusters, you can ensure that backups of deployments remain robust and available, even if one Kubernetes cluster or site fails. Furthermore, it allows Ops Manager to efficiently manage and monitor MongoDB deployments that are themselves distributed across multiple clusters, improving resilience and simplifying scaling and disaster recovery.

To learn more about how to deploy Ops Manager across multiple Kubernetes clusters using the Enterprise Kubernetes Operator, visit our documentation.

To leverage multi-Kubernetes-cluster support, you can get started with the Enterprise Kubernetes Operator.

← Previous

Building Gen AI with MongoDB & AI Partners | September 2024

Last week I was in London for MongoDB.local London —the 19th stop of the 2024 MongoDB.local tour—where MongoDB, our customers, and our AI partners came together to share solutions we’ve been building that enable companies to accelerate their AI journey. I love attending these events because they offer an opportunity to celebrate our collective achievements, and because it’s great to meet so many (mainly Zoom) friends in person! One of the highlights of MongoDB.local London 2024 was the release of our reference architecture with our MAAP partners AWS and Anthropic , which supports memory-enhanced AI agents. This architecture is already helping businesses streamline complex processes and develop smarter, more responsive applications. We also announced a robust set of vector quantization capabilities in MongoDB Atlas Vector Search that will help developers build powerful semantic search and generative AI applications with more scale—and at a lower cost. Now, with support for the ingestion of scalar quantized vectors, you can import and work with quantized vectors from your embedding model providers of choice, including MAAP partners Cohere, Nomic, and others. A big thank you to all of MongoDB’s AI partners, who continually amaze me with their innovation. MongoDB.local London was another great reminder of the power of collaboration, and I’m excited for what lies ahead as we continue to shape the future of AI together. As the Brits say: Cheers! Welcoming new AI and tech partners In September we also welcomed seven new AI and tech partners that offer product integrations with MongoDB. Read on to learn more about each great new partner! Arize Arize AI is a platform that helps organizations visualize and debug the flow of data through AI applications by quickly identifying bottlenecks in LLM calls and understanding agentic paths. "At Arize AI, we are committed to helping AI teams build, evaluate, and troubleshoot cutting-edge agentic systems. Partnering with MongoDB allows us to provide a comprehensive solution for managing the memory and retrieval that these systems rely on”, said Jason Lopatecki, co-founder and CEO of Arize AI. “With MongoDB’s robust vector search and flexible document storage, combined with Arize’s advanced observability and evaluation tools, we’re empowering developers to confidently build and deploy AI applications." Baseten Baseten provides the applied AI research and infrastructure needed to serve custom and open-source machine learning models performantly, scalably, and cost-efficiently. " We're excited to partner with MongoDB to combine their scalable vector database with Baseten's high-performance inference infrastructure and high-performance models. Together, we're enabling companies to build and deploy generative AI applications, such as RAG apps, that not only scale infinitely but also deliver optimal performance per dollar,” said Tuhin Srivastava, CEO of Baseten. “This partnership empowers developers to bring mission-critical AI solutions to market faster, while maintaining cost-effectiveness at every stage of growth." Doppler Doppler is a cloud-based platform that helps teams manage, organize, and secure secrets across environments and applications that can be used throughout the entire development lifecycle. “Doppler rigorously focuses on making the easy path, the most secure path for developers. This is only possible with deep product partnerships with all the tooling developers have come to love. We are excited to join forces with MongoDB to make zero-downtime secrets rotation for non-relational databases effortlessly simple to set up and maintenance-free,” said Brian Vallelunga, founder and CEO of Doppler. “This will immediately bolster the security posture of a company’s most sensitive data without any additional overhead or distractions." Haize Labs Haize Labs automates language model stress testing at massive scales to discover and eliminate failure modes. This, alongside their inference-time mitigations and observability tools, enables the risk-free adoption of AI. " We're thrilled to partner with MongoDB in empowering companies to build RAG applications that are both powerful yet secure, safe, and reliable,” said Leonard Tang, co-founder and CEO of Haize Labs. “MongoDB Atlas has streamlined the process of developing production-ready GenAI systems, and we're excited to work together to accelerate customers' journey to trust and confidence in their GenAI initiatives." Modal Modal is a serverless platform for data and AI/ML engineers to run and deploy code in the cloud without having to think about infrastructure. Run generative AI models, large-scale batch jobs, job queues, and more, all faster than ever before. “The coming wave of intelligent applications will be built on the potent combination of foundation models, large-scale data, and fast search,” explained Charles Frye, AI Engineer at Modal. “MongoDB Atlas provides an excellent platform for storing, querying, and searching data, from hot new techniques like vector indices to old standbys like lexical search. It's the perfect counterpart to Modal's flexible compute, like serverless GPUs. Together, MongoDB and Modal make it easy to get started with this new paradigm, and then they make it easy to scale it out to millions of users querying billions of records & maxing out thousands of GPUs.” Portkey AI Portkey AI is an AI gateway and observability suite that helps companies develop, deploy, and manage LLM-based applications. " Our partnership with MongoDB is a game-changer for organizations looking to operationalize AI at scale. By combining Portkey's LLMOps expertise with MongoDB's comprehensive data solution, we're enabling businesses to deploy, manage, and scale AI applications with unprecedented efficiency and control,” said Ayush Garg, Chief Technology Officer of Portkey AI. “Together, we're not just streamlining the path from POC to production; we're setting a new standard for how businesses can leverage AI to drive innovation and deliver tangible value." Reka Reka offers fully multimodal models including images, videos with audio, text, and documents to empower AI agents that can see, hear, and speak. "At Reka, we know how challenging it can be to retrieve information buried in unstructured multimodal data. We are excited to join forces with MongoDB to help companies test and optimize multimodal RAG features for faster production deployment,” said Dani Yogatama, CEO of Reka. “Our models understand and reason over multimodal data including text, tables, and images in PDF documents or conversations in videos. Our joint solution streamlines the whole RAG development lifecycle, speeding up time to market and helping companies deliver real values to their customers faster." But wait, there's more! To learn more about building AI-powered apps with MongoDB, check out our AI Resources Hub , and stop by our Partner Ecosystem Catalog to read about our integrations with MongoDB’s ever-evolving AI partner ecosystem.

October 9, 2024
Next →

网易游戏分享游戏场景中MongoDB运行和分析实践

在游戏行业中，数据库的稳定和性能直接影响了游戏质量和用户满意度。在竞争激烈的游戏市场中，一个优秀的数据库产品无疑能为游戏的开发和后期的运营奠定良好的基础。伴随着MongoDB在不同类型游戏场景中的应用越来越广泛，许多知名的游戏公司都在使用MongoDB来处理他们的游戏数据。 网易游戏自成立以来与广大游戏热爱者一同成长。经过20年的快速发展，网易已跻身全球七大游戏公司之一。作为中国领先的游戏开发公司，网易一直处于网络游戏自主研发领域的前端。目前在网易游戏中，包括手游、业务数据中心以及其他内部产品等在内的多项产品都已广泛应用MongoDB。 综合考虑游戏生态、架构、稳定性和成本等因素，网易游戏的数据库架构采用了多类型服务架构，根据游戏在不同的生命周期进行选型适配。MongoDB支持副本集和分片集群模式，可根据需求进行水平扩展，选择合适的分片策略和副本集设置，为网易游戏提供了灵活性和高可用性的帮助。 如何有计划地分析MongoDB数据的过程行为以便产品决策？ 随着游戏行业的不断发展，游戏数据的规模和复杂性也在不断增加。在游戏开发过程中还需要进行大量的数据分析，用以了解玩家行为、优化游戏体验、制定营销策略等。 网易互娱数据库业务负责人郑良榉表示：“我们希望有计划地对MongoDB的过程行为数据进行体系化分析，并应用到常态化管理中，以便在与MongoDB数据库交互的过程中，能够对按需触发的业务进行实时响应。” 灵活的文档模型，让数据分析更从容 MongoDB的灵活文档模型在处理游戏中的复杂数据结构时表现尤为出色。游戏存在不同场景下大量的全量更新和增量更新行为，游戏中包括玩家信息、游戏状态、物品信息等在内的数据结构会频繁发生变化，与此同时，优化占比大的字段还可能带来工作负载的更新。MongoDB采用文档存储的方式，使用JSON格式来存储数据。每个文档可以有不同的字段和结构，这使得它非常适合处理游戏中的复杂数据。通过使用MongoDB，游戏开发人员可以随时添加或修改字段而不需要进行复杂的数据库迁移操作。 当运营需要查证玩家反馈时，比如游戏中出现装备丢失、属性不对、需要查证奖励发放情况等问题，网易游戏内部对文档历史分析提出了时效性(周期性)、可查性(可用性)、对比性(变更差异)和及时性(oplog不带索引）的要求。这是一个既要又要的选择。MongoDB使用副本集来实现数据的冗余和高可用性，而操作日志（oplog）是副本集的核心机制之一。网易游戏能够通过副本集中的oplog实现了与原始文档数据联动管理，在主节点发生故障时，可以根据oplog中的信息快速恢复数据，审计并排查问题。 多场景多应用，也能游刃有余 在一些特定的游戏场景中，历史榜单会保存所有场次的数据，周期性活动的副本战斗需要结算，单一玩家的数据玩法会不断叠加，玩家的装备属性也会越来越复杂……诸如此类的大文档数据会持续增加，从而导致批量更新慢、cache使用异常、队列等待长等问题。网易游戏通过MongoDB的定期、自动巡检，帮助产品及时发现大文档潜在问题，例如数据库负载过高、请求超时、异常错误等，从而采取相应措施，规避线上运营风险。 在游戏运营期间，索引对于服务的访问效率至关重要，无索引行为会导致实例负载变高，严重时会导致整个请求阻塞，影响游戏体验和服务稳定性。依托MongoDB，网易游戏对于人工日常的数据查询行为，在访问入口代码层面实现了实时索引检测与报警，用户可以根据提示是否继续查询行为。同时在系统层面，用户访问可按实例级别配置是否允许无索引查询，以进一步减少非预期内的误操作行为。除了使用索引外，MongoDB还可以通过查询优化来加速热点数据的查询，进一步提升QPS（每秒查询推理响应速度）。 游戏运营中会有一些突发状况，比如高峰期单一玩法异常导致请求突增、某个服务卡住导致整个实例请求阻塞等，此时分析库表级别的访问QPS（非mongostat和mongotop结果）对于找到引起问题的根因点至关重要。借助于 MongoDB 的内置功能，网易游戏实现了实时的库表 QPS 和访问延时采集以帮助产品及时发现瓶颈点。此外，客户端行为的跟踪和分析也能帮助业务发现问题来源，借助MongoDB的CurrentOP功能，网易游戏能够统计业务客户端实时会话情况，并针对当前实例节点操作进行行为检索，还能追踪某个连接操作行为。 与此同时，在数据迁移和数据分布异常的情况下，MongoDB也能有效解决阻碍技术负责人执行的最大障碍——大数据量的数据同步工作及落地后的数据管理，从而提升运维效率。 郑良榉表示，MongoDB的接入只是一个开始，如何更好地贴近业务的使用需要长期的多场景实践、探索与总结，这也是当前持续发展的方向。对MongoDB数据行为的分析能够帮助网易游戏在数据决策方面提供有效参考，从而产生更大的收益和价值。 点击注册，免费开始使用 MongoDB Atlas

November 26, 2024