News From MongoDB.local NYC: Game-changing Migration Tool Now Available

Jon Lewis
June 22, 2023 | Updated: October 2, 2023
#Relational Migrator

This post is also available in: Deutsch, Français, Español, Português

The database has always been the heart of an application. Teams recognize that the choice of a database is one of the most critical decisions in developing software, influencing not only the speed at which developers can build and iterate, but also an application’s performance, functionality, and overall user experience.

Today, tens of thousands of organizations are being held back by their legacy systems built on relational databases. They struggle with the headaches, high costs, and poor developer agility caused by data silos and architecture sprawl built up over decades as they try to address new requirements and build transformative experiences.

Modernizing the data infrastructure means reducing architectural complexity, and adopting flexible data platforms like MongoDB that can support a wider range of use cases and data types. But every IT leader knows that transitioning away from legacy databases is easier said than done, and that they are often overwhelming and time-consuming projects. This is even more true when trying to shake the relational database mindset and instead adopt a developer-first approach to data.

MongoDB's Relational Migrator is now available to help organizations streamline their transition from relational databases to MongoDB by tackling some of the most common challenges we see in migration projects: effective data modeling, migrating data, and modernizing app code. Today, Relational Migrator supports migrations from Oracle, SQL Server, MySQL, and PostgreSQL directly.

By switching to MongoDB, companies can reap major rewards when it comes to modernizing an essential component of their software. Not only do they enjoy faster development and performance, but also improved reliability, scalability, and user experience. Additionally, MongoDB's advanced features, such as full-text search, time-series data support, and edge-to-cloud synchronization enable applications to take on more use cases on an efficient, easy-to-maintain data infrastructure.

With Relational Migrator, teams preparing for app modernization projects can accelerate and reduce the risk of their relational database migrations so they can start taking advantage of everything MongoDB has to offer.

Take the guesswork out of the migration process

Data modeling in MongoDB is one of the most common barriers we see when organizations come over from a relational background. MongoDB can be treated like a relational database, but to unlock the full potential of the document model organizations need to change how they think about their data and model it for how their applications are going to use it.

Relational Migrator takes the guesswork out of data modeling by analyzing an application's existing relational schema and proposing a recommended MongoDB schema based on best practices. As teams use the visual interface to compare and build their schema, Relational Migrator will continue to offer recommended mappings to further expedite schema design, but more importantly, help developers do it well.

Not only does Relational Migrator visually compare schemas, but it also applies the transformations as it moves data into MongoDB in an efficient manner, providing users with the ability to run both systems in parallel and apply changes from the source database to MongoDB on an ongoing basis.

Migrating data is only one part of application modernization. To give developers a head start, Relational Migrator even generates app code for various development languages and frameworks that reflects the new MongoDB schema they’ve designed.

With the launch of Relational Migrator, MongoDB is unlocking the full potential of its powerful developer data platform, providing new implementation possibilities for customers and a wider range of applications.

Learn more about what Relational Migrator can do for you.

Head to the MongoDB.local hub to see where we'll be showing up next.

← Previous

Introducing Atlas Stream Processing - Simplifying the Path to Reactive, Responsive, Event-Driven Apps

Update May 2, 2024: Atlas Stream processing is now generally available. Read our blog to learn more . This post is also available in: Deutsch , Français , Español , Português , 中文 Atlas Stream Processing is now in public preview. Learn more about what’s new! Today, we’re excited to announce the private preview of Atlas Stream Processing ! The world is increasingly fast-paced and your applications need to keep up. Responsive, event-driven applications bring digital experiences to life for your customers and accelerate time to insight and action for the business. Think: Notifying your users as soon as their delivery status changes Blocking fraudulent transactions during payment processing Analyzing sensor telemetry as it is generated to detect and remediate potential equipment failures before costly outages. In each of these examples, data loses its value as the seconds tick by. It needs to be queried and actioned continuously and with low latency. To do this, developers are increasingly turning to event-driven applications fueled by streaming data so that they can instantly react and respond to the constantly changing world around them. Atlas Stream Processing will help developers make the shift to event-driven apps faster. Over the years, developers have adopted the MongoDB database because they love the flexibility and ease of use of the document model, along with the MongoDB Query API which allows them to work with data as code. These foundational principles dramatically remove friction from developing software and applications. Now, we are bringing those same principles to streaming data. Atlas Stream Processing is redefining the developer experience for working with complex streams of high velocity, rapidly changing data, and unifying how developers work with data in motion and at rest. While existing products and technologies have offered many innovations to streaming and stream processing, we think MongoDB is naturally well suited to help developers with some key remaining challenges. These challenges include the difficulty of working with variable, high volume, and high-velocity data; the contextual overhead of learning new tools, languages, and APIs; and the additional operational maintenance and fragmentation that can be introduced through point technologies into complex application stacks. Introducing Atlas Stream Processing Atlas Stream Processing enables processing high-velocity streams of complex data with a few unique advantages for the developer experience: It’s built on the document model, allowing for flexibility when dealing with the nested and complex data structures common in event streams. This alleviates the need for pre-processing steps while allowing developers to work naturally and easily with data that has complex structures. Just as the database allows. It unifies the experience of working across all data, offering a single platform – across API, query language, and tools – to process rich, complex streaming data alongside the critical application data in your database. And it’s fully managed in MongoDB Atlas , building on an already robust set of integrated services. With just a few API calls and lines of code, you can stand up a stream processor, database, and API serving layer across any of the major cloud providers. Watch the MongoDB .local NYC Keynote to see Atlas Stream Processing announced by our Chief Product Officer, Sahir Azam. He covers the emergence of streaming data and how it powers a variety of use cases, key streaming challenges, and how Atlas Stream Processing can help you build modern, event-driven applications. Head of Streaming Products, Kenny Gorman, then goes through a live demo of Atlas Stream Processing in action. How does Atlas Stream Processing work? Atlas Stream Processing connects to your critical data, whether that lives in MongoDB (through change streams ) or in an event streaming platform like Apache Kafka. Developers can easily and seamlessly connect to Confluent Cloud, Amazon MSK, Redpanda, Azure Event Hubs, or self-managed Kafka using the Kafka wire protocol. And by integrating with the native Kafka driver, Atlas Stream Processing offers low-latency native performance at its foundation. In addition to our long-standing strategic partnership with Confluent, we are also excited to announce partnerships with AWS, Microsoft, Redpanda, and Google, at launch. Atlas Stream Processing then provides 3 key capabilities required to turn your firehose of streaming data into differentiated customer experiences. Let’s go through these one by one. Continuous processing First, developers can now use MongoDB’s aggregation framework to continuously process rich and complex streams of data from event streaming platforms such as Apache Kafka. This unlocks powerful new ways to continuously query, analyze, and react to streaming data without any of the delays inherent in batch processing. With the aggregation framework, you can filter and group data, aggregating high-velocity event streams into actionable insights over stateful time windows, powering richer, real-time application experiences. Continuous validation Next, Atlas Stream Processing offers developers robust and native mechanisms to handle incorrect data issues that can otherwise cause havoc in applications. Potential issues include passing inaccurate results to the app, data loss, and application downtime. Atlas Stream Processing solves these problems to ensure streaming data can be reliably processed and shared between event-driven applications. Atlas Stream Processing: Provides Continuous Schema Validation to check that events are properly formed before processing – for example rejecting events with missing fields or containing invalid value ranges Detects message corruption, and Detects late-arriving data that has missed a processing window. Atlas Stream Processing pipelines can be configured with an integrated Dead Letter Queue (DLQ) into which events failing validation are routed. This avoids developers having to build and maintain their own custom implementations. Issues can be quickly debugged while the risk of missing or corrupt data bringing down the entire application is minimized. Continuous merge Your processed data can then be continuously materialized into views maintained in Atlas database collections. We can think of this as a push query. Applications can retrieve results (via pull queries) from the view using either the MongoDB Query API or Atlas SQL interface. Continuously merging updates to collections is a really efficient way of maintaining fresh analytical views of data supporting automated and human decision-making and action. In addition to materialized views, developers also have the flexibility to publish processed events back into streaming systems like Apache Kafka. Creating a Stream Processor Let’s show you how easy it is to build a stream processor in MongoDB Atlas. With Atlas Stream Processing, you can use the same aggregation pipeline syntax for a stream processor that you’re familiar with from the database. Below we’re showcasing a simple stream processing instance from start to finish. It takes just a few lines of code. First, we’ll write an aggregation pipeline that defines a source for your data, performs validation ensuring data is not coming from the localhost/127.0.0.1 IP address, creates a tumbling window to collect grouped message data every minute, and then merges that newly processed data into a MongoDB collection in Atlas. Then, we’ll create our Stream Processor called “netattacks” specifying our newly defined pipeline p as well as dlq as arguments. This will perform our desired processing, and by using a Dead Letter Queue (DLQ), will store any invalid data safely for inspection, debugging, or re-processing later. Lastly, we can start it. That’s all it takes to build a stream processor in MongoDB Atlas. Request private preview We’re excited to get this product into your hands and see what you build with it. Learn more about Atlas Stream Processing and request private preview to participate in the private preview once it opens to developers. New to MongoDB? Get started for free today by signing up for MongoDB Atlas . Head to the MongoDB.local hub to see where we'll be showing up next. Safe Harbor The development, release, and timing of any features or functionality described for our products remains at our sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality.

June 22, 2023
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024