Atlas Online Archive

3 results

Atlas Data Federation and Online Archive Can Now Be Deployed in Azure

This post is also available in: Deutsch , Français , Español , Português , Italiano , 한국어 , 简体中文 . Exciting developments are on the horizon for users of Microsoft Azure, marking a significant leap in data management capabilities. First off, Atlas Data Federation is now Generally Available on Azure. This means you can now deploy it directly within Azure and even query data from Microsoft Azure Blob Storage. And that's not all. We've also launched the General Availability of Atlas Online Archive on Azure. These advancements usher in a new era of efficient archiving solutions for Azure-based data solutions. Both updates are big steps forward in making data management on Azure more powerful and flexible. Let's dive into what this means for you! Azure support in Atlas Data Federation (General Availability) With Atlas Data Federation, users can seamlessly query, transform, and create views across multiple Atlas databases and cloud object storage solutions, such as Amazon S3 and now Microsoft Azure Blob Storage. This feature, previously exclusive to AWS, is a game-changer, allowing direct deployment within Azure and the ability to tap into Microsoft Azure Blob Storage for data insights. Figure 1: Tap into Azure Blob Storage easily from the Atlas UI Key features of Atlas Data Federation Cloud flexibility: Choose between AWS and Azure for hosting federated database instances. Diverse data sources: Incorporate MongoDB Atlas clusters or Azure storage solutions (Azure Blob Storage and Azure Data Lake Storage Gen2) as data sources for comprehensive queries, including cross-region. Advanced aggregation: Comprehensive aggregation capabilities with operators inclusive of $match, $lookup, $queryHistory, $merge, $out, etc. Direct $out support for Azure Blob Storage and Azure Data Lake Storage Gen2. Atlas SQL queries on Azure: Execute SQL queries on Azure, integrating MongoDB data for a unified analysis experience. Atlas Data Federation simplifies accessing and analyzing complex data sets by combining data across multiple sources into a single, federated view, providing valuable insights for more informed business decisions. Explore Atlas Data Federation on Azure Today . Azure support in Atlas Online Archive (General Availability) Atlas Online Archive's expansion to Azure ensures that data tiering is not only efficient but also integrated, keeping archival data within the Azure ecosystem. This integration addresses the previous limitation of defaulting to AWS for storage, even for Azure-hosted clusters. Figure 2: Seamlessly select Azure when choosing a region Key features of Atlas Online Archive Provider choice: Opt for AWS or Azure to align with your cloud strategy. Automatic archiving: Set rules to move older data to cost-effective cloud storage automatically, eliminating manual offloading. Unified querying endpoint: Access all data through a single endpoint, ensuring quick insights without compromising data availability. Integrated MongoDB Atlas UI management: Manage your data tiering and archiving within the familiar Atlas interface, streamlining operations and maintenance. Seamlessly manage your MongoDB Atlas data tiering at scale with Atlas Online Archive. Atlas Online Archive empowers you to manage your data lifecycle efficiently, balancing cost and accessibility with ease. Finally, here are a few points to consider: Any newly created archive on an Azure cluster on or after 02/28 will default to Azure regions. Note that storage regions for Online Archive will default to Azure clusters only if there are no pre-existing AWS archives on that specific Azure cluster If there are any pre-existing AWS Online Archives on Azure clusters, then all newly created archives on that specific cluster will remain on AWS. Cloud providers or storage regions cannot be edited or modified once configured Embrace the Full Potential of Atlas Online Archive on Azure Today . We're thrilled to support your data management journey, offering enhanced control and flexibility over your data through these new Azure capabilities. As MongoDB Atlas continues to expand as a multi-cloud solution, we're here to ensure your data strategy is as dynamic and versatile as your business needs. For guidance on getting started, check out our documentation on Atlas Data Federation or Atlas Online Archive . Thank you for trusting MongoDB Atlas as your Developer Data Platform. Welcome to the future of multi-cloud data management!

February 29, 2024

New Online Archive with Performance Improvements and Enhanced Metrics

We are excited to announce several new features coming to Atlas Online Archive. With these improvements, customers will observe higher performance, be able to choose which region hosts their archival data, and have greater insight into the data stored in the archive through improved metrics. New storage engine We have optimized Online Archive through the introduction of a new storage engine that captures metadata during archival to power faster query performance. The new underlying storage service minimizes the overall data scan when querying the Online Archive. Additionally, the new storage engine delivers additional benefits in the form of storage optimizations like sorting as well as ongoing rebalancing to deliver consistent performance over time. With this feature, you will experience faster Online Archive querying performance while your overall costs are decreased when querying against the archives. Ability to choose a storage region With the new Online Archives, the users will be able to select the storage region at the time of creating an Online Archive. There will be a feature to choose one of the supported Data Federation regions from the dropdown during the creation process. In addition, we display the closest region so the users can make an informed decision about their storage region. Enhanced metrics With these new enhanced and improved metrics, we are helping customers better understand what data is in their archive such that they can connect to the appropriate endpoint for their use case and have insight into the performance of archival jobs. With these enhanced metrics, users can see a dashboard including the total # of documents in the archive and overall Archive data size (with a tooltip to explain metrics), what rate are archival jobs archiving, Min/Max Date Fields in the archive, and other statistics that are helpful and important for the users to increase the visibility of the archives. Newly created archives Note that only the newly created Online Archives on or after 06/07/2023 will see the performance improvements and enhancements initially. All existing archives created before 06/07/2023 will continue to function as-is (without the new improvements and enhanced metrics). Over the coming year, we will migrate all existing archives created before 06/07/2023 to the new backend storage service. See the documentation for additional information about creating new Online Archives and Enhanced Metrics .

June 13, 2023

MongoDB Atlas Online Archive for Data Tiering is now GA

We’re thrilled to announce that MongoDB Atlas Online Archive is now Generally Available. Online Archive allows you to seamlessly tier your data across Atlas clusters and fully managed cloud object stores, while retaining the ability to query it through a single endpoint. Reduce storage costs. Set the perfect price to performance ratio on your data. Automate data tiering. Eliminate the need to manually migrate or delete valuable data. Queryable archives. Easily federate queries across live and archival data using a unified connection string. With Online Archive, you can bring new use cases to MongoDB Atlas that were previously cost-prohibitive such as high volume time-series workloads, data archival for auditing purposes, historical log keeping and more. Manage your entire data lifecycle on MongoDB Atlas without replicating or migrating it across multiple systems. What is Atlas Online Archive? Online Archive is a fully managed data tiering solution that allows you to tier data across your "hot" database storage layer and "colder" cloud object storage to maintain queryability while optimizing on cost and performance. Online Archive is a good fit for many different use cases, including: Insert heavy workloads, where data is immutable and has lower performance requirements as it ages Historical log keeping and time-series datasets Storing valuable data that would have otherwise been deleted using TTL indexes We’ve received amazing feedback from the community over the past few months while the feature was in preview and we’re now confident in supporting your production workloads. Our users have put the feature through a variety of use cases in production and development workloads which has enabled us to make a wide range of improvements. Online Archive gives me the flexibility to store all of my data without incurring high costs, and feel safe that I won't lose it. It's the perfect solution. Ran Landau, CTO, Splitit Autonomous Archival Management It's easy to get started with Online Archive and it requires no ongoing maintenance once it’s been set up. In order to activate the feature, you can follow these simple steps: Navigate to the “Online Archive” tab on your cluster card and begin the setup flow. Set an archiving rule by selecting a date field, with dot-notation if it’s nested, or creating a custom filter. Choose commonly queried fields that you want your archival queries to be optimized for, with a few things in mind: Your data will always be “partitioned” by the date field in your archive, but can be partitioned by up to two additional fields as well. The fields that you most commonly query should be towards the top of the list (date can be moved to the top or bottom). Query fields should be chosen carefully as they cannot be changed after the fact and will have a large impact on query performance. Avoid choosing a field that has unique values as it will have negative performance impacts for queries that need to scan lots of data. And you’re done! MongoDB Atlas will automatically move data off of your cluster and into a more cost-effective storage layer that can still be queried with a single connection string that combines cluster and archive data, powered by Atlas Data Lake . What's Next? Along with announcing Online Archive as Generally Available, we’re excited to share a few additional product enhancements which should be available in the coming months: Private Link support when querying your archive Incremental deletes of data from your archive Support for BYO key encryption on your archival data Improved performance and stability Try Atlas Online Archive Online Archive allows you to right-size your Atlas clusters by storing hot data that is regularly accessed in live storage and moving colder data to a cheaper storage tier. Billing for this feature will include the cost to store data in our fully managed cloud object storage and usage based pricing for querying archive data. We can’t wait to see what new workloads you’ll bring onto MongoDB Atlas with the new flexibility provided by Online Archive! To get started, sign up for an Atlas account and deploy any dedicated cluster (M10 or higher). Have questions? Check out the documentation or head over to our community forums to get answers from fellow developers. And if we’re missing a feature you’d like to see, please let us know ! Safe Harbor Statement The development, release, and timing of any features or functionality described for MongoDB products remains at MongoDB's sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality. Except as required by law, we undertake no obligation to update any forward-looking statements to reflect events or circumstances after the date of such statements.

November 30, 2020