Legacy Modernization with MongoDB and Confluent

Robert Walters, Paresh Saraf, and Rankesh Kumar
January 7, 2021 | Updated: September 9, 2024
#partners #modernization

>> Announcement: Some features mentioned below will be deprecated on Sep. 30, 2025. Learn more.

In many organizations, crucial enterprise data is locked in dozens or hundreds of silos that may be, controlled by different teams, and stuck in systems that aren’t able to serve new workloads or access patterns. This is a blocker for innovation and insight ultimately hampering the business. For example, imagine building a new mobile app for your customers that enables them to view their account data in a single view. Designing the app could require months of time to simply navigate the internal processes necessary to gain access to the legacy systems and even more time to figure out how to integrate them.

An Operational Data Layer, or ODL, can offer a “best of both worlds” approach, providing the benefits of modernization without the risk of a full rip and replace. Legacy systems are left intact – at least at first – meaning that existing applications can continue to work as usual without interruption. New or improved data consumers will access the ODL rather than the legacy data stores, protecting those stores from new workloads that may strain their capacity and expose single points of failure. At the same time, building an ODL offers a chance to redesign the application’s data model, allowing for new development and features that aren’t possible with the rigid tabular structure of existing relational systems. With an ODL, it’s possible to combine data from multiple legacy sources into a single repository where new applications, such as a customer single view or artificial intelligence processes, can access the entire corpus of data. Existing workloads can gradually shift to the ODL, delivering value at each step. Eventually, the ODL can be promoted to a system of record and legacy systems can be decommissioned. Read our blog covering DaaS with MongoDB and Confluent to learn more.

There’s also a push today for applications and databases to be entirely cloud-based, but the reality is that current business applications are often too complex to be migrated easily or completely. Instead, many businesses are opting to move application data between on-premises and cloud deployments in an effort to leverage the full advantage of public cloud computing without having to undertake a complete, massive data lift-and-shift.

Confluent can be used for both one-time and real-time data synchronization between legacy data sources and modern data platforms like MongoDB, whose fully managed global cloud database service, MongoDB Atlas, is supported across AWS, Google Cloud, and Azure. Confluent Platform can be self-managed in your own data center while Confluent Cloud can be used on the public clouds.

Whether leaving your application on-premise is a personal choice or a corporate mandate, there are many good reasons to integrate with MongoDB Atlas.

Bring your data closer to your users in more than 70 regions with Atlas’s global clusters
Address your most intense workloads with one-click, automated sharding for scale out and zero-downtime scale up
Quickly provision TBs of database storage, all on high performance SSDs with dedicated I/O bandwidth
Natively query and analyze data across AWS S3 and MongoDB Atlas with MongoDB Atlas Data Lake
Perform full-text search queries with MongoDB Atlas Search
Build native mobile applications that seamlessly synchronize data with MongoDB Realm
Create powerful visualizations and dashboards of your MongoDB data with MongoDB Charts
Off-load older data to cost effective storage with MongoDB Atlas Online Archive

In this video we will show one time migration and Real time continuous data synchronization from a Relational System to MongoDB Atlas using Confluent Platform and the MongoDB Connector for Apache Kafka. Also we will be talking about different ways to store and consume the data within MongoDB Atlas. Git repository for the demo is here. Learn more about the MongoDB and Confluent partnership here and download the joint Reference Architecture here.

Click here to learn more about modernizing to MongoDB.

← Previous

Built With MongoDB: Coursedog

Nicholas Diao and Justin Wenig had just joined the undergraduate program at Columbia University and were excited to take their first computer science class. They registered and, to their delight, were accepted into the course. When they showed up to the first class, however, they were greeted with distressing news: the class was double-booked. As they quickly discovered, their situation was hardly unique. Universities, such as their own, often lack the software to manage the complexities of class scheduling. They decided to fix that. What started as an undergraduate project to solve a local problem turned into Coursedog , a Y Combinator backed curriculum success platform for higher education. Coursedog works with more than 70 universities to modernize the way they propose, schedule, and publish their classes to students. Coursedog has raised $4.2M in funding and been building with MongoDB since Day 1. Both founders were recently named to Forbes 30 under 30. In this edition of #BuiltWithMongoDB, we talk to Nicholas about being a student founder, building prototypes to find product-market fit, and growing with the MongoDB platform. Siya Raj Purohit: Coursedog started while you were still in college. Let’s talk about how you came across the problem your platform is built to address. Nicholas Diao: My co-founder Justin and I both wanted to be in a specific CS class. On the first day of class, we had our textbooks, our coffee, and our snacks, and were ready to go - and then we realized the professor wasn’t there. It turns out the professor had been double-booked. As aspiring CS majors, we thought “how can this be a problem of the 21st century?!” We assumed there was some automated system that ensured smooth scheduling for university classes. But it turned out that what we thought would be an automated system was a couple of overworked administrators with Microsoft Excel spreadsheets in a dark back room where they had to build the schedule themselves. And, of course, that process is error-prone and not the best use of time for university administrators. Justin and I spoke to around 400 to 500 university administrators to better understand the scheduling process, which is far more complex than it appears and has an immense impact on students and their ability to get the courses they need to learn and graduate. We realized this was a problem that was mostly ignored. Columbia Law School joined us as a design partner, and we started building a simple prototype to address this problem and make the system better for all universities. SRP: What was your initial prototype like? ND: We used simple HTML, CSS, JavaScript, and Node.JS server with a MongoDB database. Part of the reason we chose MongoDB is its ability to be really flexible, because we were learning and iterating day after day. A few months later, we ended up signing our first official school contract at Brigham Young University based on good references from Columbia Law School. In the winter of 2019, we entered Y Combinator. After we graduated from YC, we raised $4M in Series A funding, and now we’re working with more than 70 institutions and have released three additional products focused on curriculum management, event management, and catalog management. We have a team of more than 40 people across three countries. SRP: What does your tech stack consist of? ND: It's the MEVN stack: MongoDB, Express.js, Vue.js, and Node.js. We use AWS for architecture. SRP: When you started building Coursedog, you were still a CS student. How did you decide to choose MongoDB? ND: There are a couple of reasons why we started building with MongoDB. Early on, we wanted to quickly build demos that our customers could provide immediate feedback on. We knew we were tackling a complex problem and we didn’t know what our ultimate data structure would be, so we wanted a database that could be as flexible and iterative as possible and that ended up being a NoSQL database that was cloud-hosted. MongoDB came to the top of that list. It was a great decision, because we made many modifications to our data model and MongoDB managed the complexity of our solution while being easier than any other database. What got our team to really fall in love with MongoDB was the power of what we were able to do with it . There are relationships between different data objects (for example, for an economics class, managing all the components you need to know: room size, class size, department, and so on). We’re able to do very powerful joins in the SQL database and complex filters. We can build out everything we need in an iterative fashion, and MongoDB has all the functions to enable us to build more complicated features over time. We once spoke with a MongoDB technical advisor too: aggregation pipelines were new to us, and our conversation gave us great footing for getting started. From there on, the MongoDB documentation has been detailed enough to help us navigate scaling challenges. SRP: Now you’re used by more than 70 universities. How has your experience been scaling up with MongoDB? ND: We’ve found it really easy to scale because of the seamlessness and flexibility of the product and its easy communication. We have a clear sense of how much data we’re using and what the performance metrics are, and we get timely notifications. We really appreciate that if you hit 80% of a specific metric, MongoDB will send you a notification. This has been hugely helpful to our DevOps and infrastructure folks for monitoring. We’ve actually taken some cues from that: if you have 80% of a room booked, we send a notification to our users. As we’ve scaled and worked with an increasing amount of data (typically there are between 150 and 200 data types for each school), we have found that MongoDB has the flexibility and customizations we required. This is sometimes a contrast to AWS, our architectural tool. Doing the same things in AWS is much harder. For example, to change notifications (or even set them up), you have to read AWS’s incomprehensible online documentation, and then there are about 30 different places you can go to make that change on the site. In contrast, MongoDB makes it so easy to manage the back-end so we can focus on building the business. SRP: What advice would you have for college students who aspire to build their own company and move into the CTO role for that company? ND: The most important skill to pick up during college is collaboration. The way schools evaluate students is all content-focused (test, papers, problem sets) but what really matters in your career is your ability to collaborate with other people. I wouldn’t have gotten anywhere if Justin and I were not able to build a strong partnership and work with other smart, hard-working people to get Coursedog off the ground. For college students, I would say that in the long run, it doesn’t matter what grade you get on that problem set or lab; it matters if you’re able to work well with the people around you. My second piece of advice is when building solutions, start small, start local, and start trying to solve problems for the people around you - as we did with Coursedog. The best companies come from solving personal problems and then building it out from there. Looking to build something cool? Get started with the MongoDB for Startups program.

January 6, 2021

Next →

3 Ways MongoDB EA Azure Arc Certification Serves Customers

One reason more than 50,000 customers across industries choose MongoDB is the freedom to run anywhere—across major cloud providers, on-premises in data centers, and in hybrid deployments. This is why MongoDB is always working to meet customers where they are. For example, many customers choose MongoDB Atlas (which is available in more than 115 cloud regions across major cloud providers) for a fully managed experience. Other customers choose MongoDB Enterprise Advanced (EA) to self-manage their database deployments to meet specific on-premises or hybrid requirements. To that end, we’re pleased to announce that MongoDB EA is one of the first certified Microsoft Azure Arc-enabled Kubernetes applications, which provides customers even more choice of where and how they run MongoDB. Customer adoption of Azure Arc has grown by leaps and bounds. This new certification, and the launch of MongoDB EA as an Arc-enabled Kubernetes application on Azure Marketplace , means that more customers will be able to leverage the unparalleled security, availability, durability, and performance of MongoDB across environments with the centralized management of their Kubernetes deployments. We are very excited to have MongoDB available for our customers on the Azure Marketplace. By extending Azure Arc’s management capabilities to your MongoDB deployments, customers gain the benefit of centralized governance, enhanced security, and deeper insights into database performance. Azure Arc makes hybrid database management with MongoDB efficient and consistent. Collaboration between MongoDB and Microsoft represents an opportunity for many of our customers to further accelerate their digital transformation when building enterprise-class solutions with Azure Arc. Christa St Pierre, Partner Group Manager, Azure Edge Devices, Microsoft Here are three ways the launch of MongoDB EA on Azure Marketplace for Arc-enabled Kubernetes applications gives customers greater flexibility. 1. MongoDB EA supports multi-Kubernetes cluster deployments, simplifies management MongoDB Enterprise Advanced seamlessly integrates market-leading MongoDB capabilities along with robust enterprise support and tools for self-managed deployments at any scale. This powerful solution includes advanced automation, comprehensive auditing, strong authentication, reliable backup, and insightful monitoring capabilities, all of which work together to ensure security compliance and operational efficiency for organizations of any size. The relationship between MongoDB and Kubernetes is one of strong synergy. With Kubernetes, MongoDB EA really can run anywhere, such as a single deployment spanning on-premises and more than one public cloud Kubernetes cluster. Customers can use the MongoDB Enterprise Kubernetes Operator, a key component of MongoDB Enterprise Advanced, to simplify the management and automation of self-managed MongoDB deployments in Kubernetes. This includes tasks like creating and updating deployments, managing backups, and integrating with various Kubernetes services. The ability of the MongoDB Enterprise Kubernetes Operator to deploy and manage MongoDB deployments that span multiple Kubernetes clusters significantly enhances resilience, improves disaster recovery, and minimizes latency by allowing data to be co-located closer to where it is needed, ensuring optimal performance and reliability. 2. Azure Arc complements MongoDB EA, providing centralized management While MongoDB Enterprise Advanced is already among a select group of databases capable of operating across multiple Kubernetes clusters , it is now also supported in Azure Arc-enabled Kubernetes environments. Azure Arc enables the standardized management of Kubernetes clusters across various environments—including in Azure, on-premises, and even other clouds—while harnessing the power of Azure services. Azure Arc accomplishes this by extending the Azure control plane to standardize security and governance across a wide range of resources and locations. For instance, organizations can centrally monitor all of the Azure Arc-enabled Kubernetes clusters using Azure Monitor for containers , or they can enforce threat protection at scale using Microsoft Defender for Kubernetes. This centralized control significantly reduces the complexity of managing Kubernetes clusters running anywhere, as customers can oversee all resources and apply consistent security and compliance policies across their hybrid environment. 3. Customers can leverage the resilience of MongoDB EA and the centralized governance of Azure Arc Together, these solutions empower organizations to build robust applications across a wide array of environments, whether on-premises or in multi-cloud settings. The combination of MongoDB Enterprise Advanced and the MongoDB Enterprise Operator simplifies the deployment of MongoDB across Kubernetes clusters, allowing organizations to fully leverage enhanced resilience and geographic distribution that surpasses the capabilities of a single Kubernetes cluster. Azure Arc further enhances this synergy by providing centralized management for all of these Kubernetes clusters, regardless of where they are running; for customers running entirely in the public cloud, we recommend using MongoDB’s fully managed developer data platform, MongoDB Atlas. If you’re interested in learning more, we invite you to explore the Azure Marketplace listing for MongoDB Enterprise Advanced for Arc-enabled Kubernetes applications. Please note that aside from use for evaluation and development purposes, this offering requires the purchase of a MongoDB Enterprise Advanced subscription. For licensing inquiries, we encourage you to reach out to MongoDB at https://www.mongodb.com/contact to secure your license and to begin harnessing the full potential of these powerful solutions.

November 19, 2024