Querying the MongoDB Atlas Price Book with Atlas Data Federation
Rate this article
As a DevOps engineer or team, keeping up with the cost changes of a continuously evolving cloud service like MongoDB Atlas database can be a daunting task. Manual monitoring of pricing information can be laborious, prone to mistakes, and may result in delays in strategic decisions. In this article, we will demonstrate how to leverage Atlas Data Federation to query and visualize the MongoDB Atlas price book as a real-time data source that can be incorporated into your DevOps processes and application infrastructure.
Atlas Data Federation is a distributed query engine that allows users to combine, transform, and move data across multiple data sources without complex integrations. Users can efficiently and cost-effectively query data from different sources, such as your Atlas clusters, cloud object storage buckets, Atlas Data Lake datasets, and HTTP endpoints with the MongoDB Query Language and the aggregation framework, as if it were all in the same place and format.
While using HTTP endpoints as a data source in Atlas Data Federation may not be suitable for large-scale production workloads, it’s a great option for small businesses or startups that want a quick and easy way to analyze pricing data or to use for testing, development, or small-scale analysis. In this guide, we will use the JSON returned by https://cloud.mongodb.com/billing/pricing?product=atlas as an HTTP data source for a federated database.
Let's create a new federated database in MongoDB Atlas by clicking on Data Federation in the left-hand navigation and clicking “set up manually” in the "create new federated database" dropdown in the top right corner of the UI. A federated database is a virtual database that enables you to combine and query data from multiple sources.
The HTTP data source allows you to query data from any web API that returns data in JSON, BSON, CSV, TSV, Avro, Parquet, and ORC formats, such as the MongoDB Atlas price book.
Create a mapping between the HTTP data source and your federated database instance by dragging and dropping the HTTP data source into your federated database. Then, rename the cluster, database, and collection as desired by using the pencil icon.
Atlas Data Federation allows you to transform the raw source data by using the powerful MongoDB Aggregation Framework. We’ll create a view that will reshape the price book into individual documents, each to represent a single price item.
First, create a view:
Then, name the view and paste the following pipeline:
1 [ 2 { 3 "$unwind": { 4 "path": "$resource" 5 } 6 }, { 7 "$replaceRoot": { 8 "newRoot": "$resource" 9 } 10 } 11 ]
This pipeline will unwind the "resource" field, which contains an array of pricing data, and replace the root document with the contents of the "resource" array.
Now, let's save the changes and copy the connection string for our federated database instance. This connection string will allow you to connect to your federated database.
Atlas Data Federation supports connection methods varying from tools like MongoDB Shell and Compass, any application supporting MongoDB connection, and even a SQL connection using Atlas SQL.
Let’s now connect to the federated database instance using MongoDB Compass. By connecting with Compass, we will then be able to use the MongoDB Query Language and aggregation framework to start querying and analyzing the pricing data, if desired.
We’ll use MongoDB Atlas Charts for visualization of the Atlas price book. Atlas Charts allows you to create interactive charts and dashboards that can be embedded in your applications or shared with your team.
Once in Charts, you can create new dashboards and add a chart. Then, select the view we created as a data source:
As some relevant data fields are embedded within the sku field, such as NDS_AWS_INSTANCE_M50, we can use calculated fields to help us extract those, such as provider and instanceType:
Use the following value expression:
- Provider
{$arrayElemAt: [{$split: ["$sku", "_"]}, 1]} - InstanceType
{$arrayElemAt: [{$split: ["$sku", "_"]}, 3]} - additonalProperty
{$arrayElemAt: [{$split: ["$sku", "_"]}, 4]}
Now, by using Charts like a heatmap, we can visualize the different pricing items in a color-coded format:
- Drag and drop the “sku” field to the X axis of the chart.
- Drag and drop the “pricing.region” to the Y axis (choose “Unwind array” for array reduction).
- Drag and drop the “pricing.unitPrice” to Intensity (choose “Unwind array” for array reduction).
- Drag and drop the “provider”, “instanceType”, and “additionalProperty” fields to filter and choose the desired values.
The final result: A heatmap showing the pricing data for the selected providers, instance types, and additional properties, broken down by region. Hovering over each of the boxes will present its exact price using a tooltip. Thanks to the fact that our federated database is composed from an HTTP data source, the data visualized is the actual live prices returned from the HTTP endpoint, and not subjected to any ETL delay.
With Atlas Data Federation DevOps teams, developers and data engineers can generate insights to power real-time applications or downstream analytics. Incorporating live data from sources such as HTTP, MongoDB Clusters, or Cloud Object Storage reduces the effort, time-sink, and complexity of pipelines and ETL tools.
Have questions or comments? Visit our Community Forums.
Ready to get started? Try Atlas Data Federation today!
Rate this article