Joins and Other Aggregation Enhancements Coming in MongoDB 3.2 (Part 1 of 3) – Introduction

Andrew Morgan
October 30, 2015 | Updated: December 15, 2015

Series:

Part 1 – Introduction
Part 2 – Worked Examples
Part 3 - Adding Some Code Glue and Geolocation

This is the first of a three part blog series looking at the aggregation enhancements being introduced in MongoDB 3.2 – most notably $lookup which implements left-outer equi-joins in the MongoDB Aggregation Framework.

This post starts with an introduction to analyzing data with MongoDB. We then explain why joins are sometimes useful for MongoDB – in spite of the strengths of the document model – and how developers have been working without them.

The second post in the series works through examples of building aggregation pipelines – including using the operators added in MongoDB 3.2.

The third and final post shows how geolocation data can be included as well as what to do when you reach the limit of what can be done using a single pipeline – including adding wrapper code. That post also summarizes some of the limitations of the Aggregation Framework and reasons why you might supplement it with a full visualization solution such as Tableau together with MongoDB's Connector for BI (Business Intelligence) – also new in MongoDB 3.2.

Disclaimer

MongoDB's product plans are for informational purposes only. MongoDB's plans may change and you should not rely on them for delivery of a specific feature at a specific time.

Real-Time Analytics and Search

With the emergence of new data sources such as social media, mobile applications and sensor-equipped “Internet of Things” networks, organizations can extend analytics to deliver real-time insight and discovery into such areas as operational performance, customer satisfaction, and competitor behavior.

Time to value is everything. For example, having access to real-time customer sentiment or fleet tracking is of little benefit unless the data can be analyzed and reported in real-time.

MongoDB 3.2 aims to extend the options for performing analytics on the live, operational database – ensuring that answers are delivered quickly, and reflect current data. Work that would previously have needed to be done on the client side can now be performed by the database – freeing the developer to focus on new features.

The Case for Joins

MongoDB’s document data model is flexible and provides developers many options in terms of modeling their data. Most of the time all the data for a record tends to be located in a single document. For the operational application, accessing data is simple, high performance, and easy to scale with this approach.

When it comes to analytics and reporting, however, it is possible that the data you need to access spans multiple collections. This is illustrated in Figure 1, where the _id field of multiple documents from the products collection is included in a document from the orders collection. For a query to analyze orders and details about their associated products, it must fetch the order document from the orders collection and then use the embedded references to read multiple documents from the products collection. Prior to MongoDB 3.2, this work is implemented in application code. However, this adds complexity to the application and requires multiple round trips to the database, which can impact performance.

^{Figure 1: Application-Layer simulation of joins between documents}

MongoDB 3.2 introduces the $lookup operator that can now be included as a stage in an aggregation pipeline. With this approach, the work of combining data from the orders and products collections is implemented within the database, and as part of a broader aggregation pipeline that performs other processing in a single query. As a result, there is less work to code in the application, and fewer round trips to the database. You can think about $lookup as equivalent to a left outer equi-join.

Aside - What is a Left Outer Equi-Join?

A left outer equi-join produces a result set that contains data for all documents from the left table (collection) together with data from the right table (collection) for documents where there is a match with documents from the left table (collection). This is illustrated in Figure 2.

^{Figure 2: Left-Outer join between collections}

MongoDB's Aggregation Framework

The Aggregation Framework is a pipeline for data aggregation modeled on the concept of data processing pipelines. Documents enter a multi-stage pipeline that transforms the documents into aggregated results. The pipeline consists of stages; each stage transforms the documents as they pass through.

In general, each successive stage reduces the volume of data; removing information that isn't needed and combining other data to produce summarized results.

Figure 3 shows a conceptual model for the Aggregation Framework pipeline. This is what's happening at each stage:

On the left-hand side/start of the pipeline is the original collection contents – each record (document) containing a number of shapes (keys), each with a particular color (value)
The $match stage filters out any documents that don't contain a red diamond
The $project stage adds a new “square” attribute with a value computed from the value (color) of the snowflake and triangle attributes
The $lookup stage (new in 3.2 - more details later) performs a left-outer join with another collection, with the star being the comparison key. This creates new documents which contain everything from the previous stage but augmented with data from any document from the second collection containing a matching colored star (i.e., the blue and yellow stars had matching “lookup” values, whereas the red star had none).
Finally, the $group stage groups the data by the color of the square and produces statistics (sum, average and standard deviation) for each group.

^{Figure 3: MongoDB Aggregation Framework pipeline}

This is the full set of aggregation stages:

$match – Filter documents
$geoNear – Sort documents based on geographic proximity
$project – Reshape documents (remove or rename keys or add new data based on calculations on the existing data)
$lookup – Coming in 3.2 – Left-outer joins
$unwind – Expand documents (for example create multiple documents where each contains one element from an array from the original document)
$group – Summarize documents
$sample – Randomly selects a subset of documents
$sort – Order documents
$skip – Jump over a number of documents
$limit – Limit number of documents
$redact – Restrict sensitive content from documents
$out – Coming in 3.2* – store the results in a new collection

The details can be found in the documentation.

New Aggregation Operators in MongoDB 3.2

There are operators used within each stage and this set is being extended in MongoDB 3.2 to include:

Array operations

$slice, $arrayElemAt, $concatArrays, $isArray, $filter, $min, $max, $avg and $sum (some of these were previously available in a $group stage but not in $project

Standard Deviations

$stdDevSamp (based on a sample) and $stdDevPop (based on the complete population)

Square Root

$sqrt

Absolute (make +ve) value

$abs

Rounding numbers

$trunc, $ceil, $floor

Logarithms

$log, $log10, $ln

Raise to power

$pow

Natural Exponent

$exp

Further details on these new operators can be found in the MongoDB 3.2 Release Notes.

$lookup – Left Outer Equi-Joins

Figure 4 illustrates the syntax for performing the join:

leftCollection is the collection that the aggregation is being performed on and is the left collection in the join
from identifies the collection that it will be joined with – the right collection (rightCollection in this case)
localField specifies the key from the original/left collection – leftVal
foreignField specifies the key from the right collection – rightVal
as indicates that the data from the right collection should be embedded within the resulting documents as an array called embeddedData

^{Figure 4: $lookup – Left-Outer Joins for MongoDB}

In the follow-on blogs in this series, you’ll see how the data from a home sales collection (containing details of each home sale, including the property’s postal code) is joined with data from a postal code collection (containing postal codes and their geographical location). This produces documents that contain the original home sale information augmented with the coordinates of the property. In this case, the “homesales” collection is the left-collection and “postcodes” the right-collection; the “postcode” field from each collection is the localField which is matched with the foreignField.

Next Steps

To learn more about what's coming up in MongoDB 3.2, register for the What's new in MongoDB 3.2 webinar and review the MongoDB 3.2 development documentation.

To get the best understanding of the new features then you should experiment with the software, which is available in the MongoDB 3.2 release. It will be available in both the MongoDB Enterprise Advanced and Community Editions (GA coming soon).

The reason MongoDB releases development releases is to give the community a chance to try out the new software – and we hope that you'll give us feedback, whether it be by joining the MongoDB 3.2 bug hunt or commenting on this post.

Learn more about joins and other aggregation enhancements in MongoDB 3.2, watch Andrew's on-demand webinar: Watch the on-demand webinar

About the Author - Andrew Morgan

Andrew is a Principal Product Marketing Manager working for MongoDB. He joined at the start of this summer from Oracle where he’d spent 6+ years in product management, focussed on High Availability. He can be contacted @andrewmorgan or through comments on his blog (clusterdb.com).

← Previous

Document Validation – Part 2: Putting it all Together, a Tutorial

Introduction This is the second and final post in a series looking at document validation in MongoDB 3.2; if you haven’t already read the first blog in this series then you should read it now. The intent of this post is to step you through exactly how document validation can be introduced into an existing production deployment in such a way that there is no impact to your users. It covers: Setting up some test data (not needed for a real deployment) Using MongoDB Compass and the mongo shell to reverse engineer the de facto data model and identify anomalies in the existing documents Defining the appropriate document validation rules Preventing new documents being added which don’t follow the new rules Bring existing documents “up to spec” against the new rules Figure 1: Aligning document validation with application lifecycle Tutorial This section looks at taking an existing, deployed database which currently has no document validations defined. It steps through understanding what the current document structure looks like; deciding on what rules to add and then rolling out those new rules. As a pre-step add some data to the database (obviously, this isn't needed if working with your real deployment). use clusterdb; db.dropDatabase(); use clusterdb(); db.inventory.insert({ "_id" : 1, "sku" : "abc", "description" : "product 1", "instock" : 120 }); db.inventory.insert({ "_id" : 2, "sku" : "def", "description" : "product 2", "instock" : 80 }); db.inventory.insert({ "_id" : 3, "sku" : "ijk", "description" : "product 3", "instock" : 60 }); db.inventory.insert({ "_id" : 4, "sku" : "jkl", "description" : "product 4", "instock" : 70 }); db.inventory.insert({ "_id" : 5, "sku" : null, "description" : "Incomplete" }); db.inventory.insert({ "_id" : 6 }); for (i=1000; i<2000; i++) { db.orders.insert({ _id: i, item: "abc", price: i % 50, quantity: i % 5 }); }; for (i=2000; i<3000; i++) { db.orders.insert({ _id: i, item: "jkl", price: i % 30, quantity: Math.floor(10 * Math.random()) + 1 }); }; for (i=3000; i<3200; i++) { db.orders.insert({ _id: i, price: i % 30, quantity: Math.floor(10 * Math.random()) + 1 }); }; for (i=3200; i<3500; i++) { db.orders.insert({ _id: i, item: null, price: i % 30, quantity: Math.floor(10 * Math.random()) + 1 }); }; for (i=3500; i<4000; i++) { db.orders.insert({ _id: i, item: "abc", price: "free", quantity: Math.floor(10 * Math.random()) + 1 }); }; for (i=4000; i<4250; i++) { db.orders.insert({ _id: i, item: "abc", price: "if you have to ask....", quantity: Math.floor(10 * Math.random()) + 1 }); }; The easiest way to start understanding the de facto schema for your database is to use MongoDB Compass. Simply connect Compass to your mongod (or mongos if you're using sharding) and select the database/collection you'd like to look into. To see MongoDB Compass in action – view this demo video . As shown in Figure 2, there are typically four keys in each document from the clusterdb.orders table: _id is always present and is a number item is normally present and is a string (either "abc" or "jkl") but is occasionally null or missing altogether (undefined) price is always present and is in most cases a number (the histogram shows how the values are distributed between 0 and 49) but in some cases it's a string quantity is always present and is a number Figure 2: Viewing the Document Schema using MongoDB Compass For this tutorial, we'll focus on the price . By clicking on the string label, Compass will show us more information about the string content for price - this is shown in Figure 3. Figure 3: Drilling Down into string Values Compass shows us that: For those instances of price which are strings, the common values are "free" and "if you have to ask....". If you click on one of those values, a query expression is formed and clicking "Apply" runs that query and now Compass will show you information only for that subset of documents. For example, where price == "if you have to ask...." (see Figure 4). By selecting multiple attributes, you can build up fairly complex queries. The query you build visually is printed at the top so you can easily copy/paste into other contexts like the shell. Figure 4: Formulating Search Expressions with MongoDB Compass If applications are to work with the price from these documents then it would be simpler it it was always set to a numerical value, and so this is something that should be fixed. Before cleaning up the existing documents, the application should be updated to ensure numerical values are stored in the price field. We can do this by adding a new validation rule to the collection. We want this rule to: Allow changes to existing invalid documents Prevent inserts of new documents which violate validation rules Set up a very simple document validation rule that checks that price exists and contains a double – see the enumeration of MongoDB BSON types These steps should be run from the mongo shell: db.orders.runCommand("collMod", {validationLevel: "moderate", validationAction: "error"}); db.runCommand({collMod: "orders", validator: { price: {$exists: true}, price: {$type: 1} } }); The validation rules for this collection can now be checked: ```javascript db.getCollectionInfos({name:"orders"}) [ { "name": "orders", "options": { "validator": { "price": { "$type": 1 } }, "validationLevel": "moderate", "validationAction": "error" } } ] </code></pre> Now that this has been set up, it's possible to check that we can't add a new document that breaks the rule: <pre><code>db.orders.insert({ "_id": 6666, "item": "jkl", "price": "rogue", "quantity": 1 }); Document failed validation WriteResult({ "nInserted": 0, "writeError": { "code": 121, "errmsg": "Document failed validation" } }) </code></pre> But it's OK to modify an existing document that does break the rule: <pre><code>db.orders.findOne({price: {$type: 2}}); { "_id": 3500, "item": "abc", "price": "free", "quantity": 5 } > db.orders.update( {_id: 3500}, {$set: {quantity: 12}}); Updated 1 existing record(s) in 5ms WriteResult({ "nMatched": 1, "nUpserted": 0, "nModified": 1 })</code></pre> Now that the application is no longer able to store new documents that break the new rule, it's time to clean up the "legacy" documents. At this point, it's important to point out that Compass works on a random sample of the documents in a collection (this is what allows it to be so quick). To make sure that we're fixing **all** of the documents, we check from the `mongo` shell. As the following commands could consume significant resources, it may make sense to run them on a secondary): <pre><code>secondary> db.orders.aggregate([ {$match: { price: {$type: 2}}}, {$group: { _id: "$price", count: {$sum:1}}} ]) { "_id" : "if you have to ask....", "count" : 250 } { "_id" : "free", "count" : 500 } </code></pre> The number of exceptions isn't too high and so it is safe to go ahead and fix up the data without consuming too many resources: <pre><code>db.orders.update( {price:"free"}, {$set: {price: 0}}, {multi: true}); db.orders.update( {price:"if you have to ask...."}, {$set: {price: 1000000}}, {multi: true});</code></pre> At this point it's now safe to enter the strict mode where any inserts or updates will cause an error if the document being stored doesn't follow the rules: <pre><code>db.orders.runCommand("collMod", {validationLevel: "strict", validationAction: "error"});</code></pre> <h3 id="next-steps">Next Steps</h3> Hopefully this has given you a sense for what the Document Validation functionality offers and started you thinking about how it could be applied to your application and database. I'd encourage you to read up more on the topic and these are some great resources: <ul> <li>If you haven’t already read the <a href="https://www.mongodb.com/blog/post/document-validation-part-1-adding-just-the-right-amount-of-control-over-your-documents “Document Validation - Part 1: Adding Just the Right Amount of Control Over Your Documents"">first blog in this series</a> then you should read it now</li> <li><a href="https://docs.mongodb.org/master/release-notes/3.2/#document-validation" title="MongoDB 3.2 documentation for Document Validation">MongoDB 3.2 documentation for Document Validation</a></li> <li>The best way to really get a feel for the functionality is to try it out for yourself:<a href="https://www.mongodb.org/downloads#development" title="Download MongoDB 3.2">Download MongoDB 3.2</a></li> <li>Feedback is welcomed and we’d encourage you to join the <a href="https://www.mongodb.com/blog/post/announcing-the-mongodb-3-2-bug-hunt">MongoDB 3.2 bug hunt</a></li> <li><a href="http://www.eliothorowitz.com/blog/2015/09/11/document-validation-and-what-dynamic-schema-means/" title="Document Validation and What Dynamic Schema Means">Document Validation and What Dynamic Schema Means</a> – Eliot Horowitz. This blog post adds context to why this functionality is being introduced now.</li> <li><a href="https://www.mongodb.com/presentations/data-management-3-bulletproof-data-management" title="Bulletproof Data Management">Bulletproof Data Management</a> – Buzz Moschetti. Great presentation on how to look after your data - including in earlier versions of MongoDB</li> <li>Register for our upcoming webinar covering <a href="https://www.mongodb.com/webinar/whats-new-in-mongodb-3-2">what's new in MongoDB 3.2</a></li> </ul> <hr> Watch Andrew's webinar covering document validation in 3.2. <center><a class="btn btn-primary" href="https://www.mongodb.com/presentations/webinar-document-validation-in-mongodb-3-2?jmp=blog" target="_BLANK">Document Validation in MongoDB 3.2</a></center> <hr> About the Author - Andrew Morgan Andrew is a Principal Product Marketing Manager working for MongoDB. He joined at the start of this summer from Oracle where he’d spent 6+ years in product management, focussed on High Availability.

October 30, 2015

Next →

MongoDB’s 2024 Year in Review

It’s hard to believe that another year is almost over! 2024 was a transformative year for MongoDB, and it was marked by both innovation and releases that further our commitment to empowering customers, developers, and partners worldwide. So without further ado, let’s dive into MongoDB’s 2024 highlights. We’ll also share our executive team’s predictions of what 2025 might have in store. A look back at 2024 MongoDB 8.0: The most performant version of MongoDB ever In October we released MongoDB 8.0 , the fastest, most resilient, secure, and reliable version of MongoDB yet. Architectural optimizations in MongoDB 8.0 have significantly improved the database’s performance, with 36% faster reads and 59% higher throughput for updates. Our new architecture also makes horizontal scaling cheaper and faster. Finally, working with encrypted data is easier than ever, thanks to the addition of range queries in Queryable Encryption (which allows customers to encrypt, store, and perform queries directly on data). Whether you’re a startup building your first app, or you’re a global enterprise managing mission-critical workloads, MongoDB 8.0 offers unmatched power and flexibility, solidifying MongoDB’s place as the world’s most popular document database. Learn more about what makes 8.0 the best version of MongoDB ever on the MongoDB 8.0 page . Delivering customer value with the MongoDB AI Applications Program AI applications have become a cornerstone of modern software, and MongoDB is committed to equipping customers with the technology, tools, and support they need to succeed on their AI journey. That’s why we launched the MongoDB AI Applications Program (MAAP) in 2024, a comprehensive program designed to accelerate the development of AI applications. By offering customers resources like access to AI specialists, an ecosystem of leading AI and tech companies, and AI architectural best practices supported by integrated services, MAAP helps solve customers’ most pressing business challenges, unlocks competitive advantages, and accelerates time to value for AI investments. Overall, MAAP’s aim is to set customers on the path to AI success. Visit the MongoDB AI Applications Program page or watch our session from AWS re:Invent to learn more! Advancing AI with MongoDB Atlas Vector Search In 2024, MongoDB further cemented its role in the AI space with enhancements to MongoDB Atlas Vector Search . Recognized in 2024 (for the second consecutive year!) as one of the most loved vector databases , MongoDB continues to provide a scalable, unified, and secure platform for building cutting-edge AI use cases. Recent advancements like vector quantization in Atlas Vector Search help deliver even more value to our customers, enabling them to scale applications to billions of vectors at a lower cost. Head over to our Atlas Vector Search quick start guide to get started with Atlas Vector Search today, or visit our AI resources hub to learn more about how MongoDB can power AI applications. Search Nodes: Performance at scale Search functionality is indispensable in modern applications, and with Atlas Search Nodes, organizations can now optimize their search workloads like never before. By providing dedicated infrastructure for Atlas Search and Vector Search workloads, Search Nodes ensure high performance (e.g., a 40–60% decrease in query times), scalability, and reliability, even for the most demanding use cases. As of this year , Search Nodes are generally available across AWS, Google Cloud, and Microsoft Azure. This milestone underscores MongoDB’s commitment to delivering powerful solutions that scale alongside our customers’ needs. To learn more about Search Nodes, check out our documentation or watch our tutorial . Looking ahead: MongoDB’s 2025 predictions After the excitement of the past few years, 2025 will be defined by ensuring that technology investments deliver tangible value. Organizations remain excited about the potential AI and emerging technologies hold to solve real business challenges, but are increasingly focused on maintaining a return on investment. “Enterprises need to innovate faster than ever, but speed is no longer the only measure of success. Increasingly, organizations are laser-focused on ensuring that their technology investments directly address critical business challenges and provide clear ROI and competitive advantage—whether it’s optimizing supply chains, delivering hyper-personalized customer experiences, or scaling operations efficiently,” said Sahir Azam, Chief Product Officer at MongoDB. “In 2025, I expect to see organizations make significant strides in driving this innovation and efficiency by applying AI to more production use cases and by maturing the way they leverage their data to build compelling and differentiated customer experiences.” Indeed, we expect to see organizations make more strategic investments in emerging technologies like gen AI—innovating with a sharp focus on solving business challenges. “In 2025, we can expect the focus to shift from ‘what AI can do’ to ‘what AI should do,’ moving beyond the hype to a clearer understanding of where AI can provide real value and where human judgment is still irreplaceable,” said Tara Hernandez, VP of Developer Productivity at MongoDB. “As we advance, I think we’ll see organizations begin to adopt more selective, careful applications of AI, particularly in areas where stakes are high, such as healthcare, finance, and public safety. A refined approach to AI development will be essential—not only for producing quality results but also to build trust, ensuring these tools genuinely support human goals rather than undermining them.” With more capable, accessible application development tools and customer-focused programs like MAAP at developers’ fingertips, 2025 is an opportunity to make a data-driven impact faster than ever before. "Right now, organizations have an opportunity to leverage their data to reimagine how they do business, to more effectively adapt to a changing world, and to revolutionize our quality of life,” said Andrew Davidson, SVP of Products at MongoDB. “By harnessing our latest technologies, developers can build a foundation for a transformative future." Head over to our updates page to learn more about the new releases and updates from MongoDB in 2024. Keep an eye on our events page to learn what's to come from MongoDB in 2025!

December 19, 2024