Data modeling in MongoDB is the art and science of organizing data in a database to represent entities, attributes, and relationships, thereby optimally supporting business objectives.
There are three types of data models: conceptual, logical, and physical. Conceptually, for search, we have content, and it needs to be findable and rankable. Logically, content is generally segregated by types of data (e.g., books and library patron users are different types of entities in the system), each type having its own set of attributes and relationships.
The most primal level of data modeling is where the pedal meets the metal, almost literally — at the physical level. The physical model consists of a schema design, laying out how data is structured into databases, collections, shards, fields, field types, and indexes.
The structural and type flexibility of MongoDB’s document model leaves the exercise of schema design up to the developer to create the best layout for the workloads given physical, hardware, and software constraints. Design patterns have emerged by distilling and abstracting successful techniques for dealing with various common needs. And, as with any software, there are system constraints within which search operates.
This document serves as a blueprint for designing schemas for these common usage patterns, which leverage the power of the underlying search system and work within the boundaries imposed by the hardware and software environments.
There are 2 parts to this content. Here in Part 1, the practical considerations and foundational concepts and building blocks are detailed. In Part 2, common search use cases are presented with patterns that have emerged to tackle them including the current best practices.
Use case workloads
> Age-old data modeling adage: “The optimal grouping of objects into documents is determined by the workload.”
Our first stop in schema design for search is understanding the workloads that our application will demand of MongoDB Search. Here are the common use cases, given a search request:
Standard paged search results: return the top (10 or so) matching documents, in sorted or relevancy-ranked order, with or without facets.
Process all matching documents: docs may go through successive pipeline stages that need the entire set of results to continue, such as $group and $sort.
Return only facet counts, without any of the matching documents.
Relevant as-you-type suggestions.
Replace $match with $search to index and query on many more fields than can be indexed with database B-tree indexes.
“Find anything anywhere” - best practices; compared to $match and $elemMatch.
Leverage lexical and other types of vector pre-filtering.
Physical constraints
> Wise modeling master: “We model to face and answer to constraints.”
The architecture of MongoDB Search frames what’s possible. Given the way MongoDB Search is built, here’s a list of the main physical parameters that factor into schema design decisions:
The $search stage queries a single MongoDB Search index.
A MongoDB Search index is tied to a single Atlas collection.
A mapping defines which fields are to be indexed in what manner, mapped by field name and data type.
Search criteria is used to score and return matching collection documents (or individual embedded documents!)
It’s worth calling out that these particular architectural decisions are subject to change as MongoDB Search progresses. There are already several works in progress behind the scenes of the product that alter and relax, some of these architectural decisions, opening the door to some interesting ways of doing things that aren’t possible currently. These patterns will change as the system’s physical capabilities improve. As they say, we’re working on it… always! And in the OPEN!
The preceding list of architecturally controlled physical operating parameters fits within the aggregation pipeline framework as a $search stage, which makes calls to a search service built on top of the powerful search engine, Lucene. Let’s look at both ends of the architecture, taking into consideration the typical workloads demanded of $search and the constraints it works within. First, let’s see how $search fits into the aggregation pipeline. Then, we’ll look at how the underlying index structure operates at the lowest level of MongoDB Search.
Top down—how $search fits into an aggregation pipeline
The $search stage must be, because of pragmatic choices made during implementation, the first stage in the aggregation pipeline (or a $unionWith stage). There’s no input set of documents at this stage. The output of this stage consists of an ordered list of documents and some additional search metadata. The output of the $search stage can be further aggregated using any of the available stages that take a list of documents as input, such as $limit, $skip, $project, $addField, $facet, $match, $group, $lookup, and others, though post-search aggregating stages can have a drastic effect on performance, as explained here later.
> What about $searchMeta? This stage performs a search, but rather than return the matching documents, it returns search result metadata, such as the total count of matching documents and facets if they were requested. This stage is handy if all you need is this metadata, whereas $search returns all of this same metadata (as $$SEARCH_META) along with the matching documents.
Deep Dive
To use $search or $searchMeta: If you want matching documents with or without facets, use $search. If you only need the total count and/or facets, use $searchMeta.
When $search executes from the mongod (mongo “database”) node running the aggregation pipeline stages, it makes a remote call to a Java-based mongot (mongo “text”) process. The mongot process interfaces with the Lucene APIs for indexing, searching, ranking, highlighting, and faceting.
Important things to note as it relates to schema design: Matching collection documents are returned, in relevancy-ranked order. While a search may result in matching thousands, millions, or even billions of documents, the underlying search engine is designed to match, rank, and return the results quickly, providing the best matches first. The mongot service provides results in batches to the calling aggregation pipeline and will get invoked until the calling pipeline has exhausted the cursor and processed all of the matches or short-circuited from a stage like $limit.
Warning: Running aggregation pipelines in the Atlas UI or Compass will apply an implicit $limit to show only the first several resultant documents. Searches are quick. The same aggregation pipeline, if naively run from a driver-using code environment, may pull the entire cursors' worth of documents into an array data structure. Retrieving all search results into an array can be quite time- and memory-consuming. It’s usually the desired pattern to $limit search results and paginate through them as needed. If all goes well, the best are at the top, so users won’t need to see many. :)
These working conditions of the $search stage inform some design decisions: that it must be the first stage in the pipeline, that it returns documents in relevancy order, that it crosses process boundaries returning batches of matches, and, unless otherwise specifically limited, all matching search results will, eventually, be returned. That last point is an important one — the number of results fed through the pipeline after $search has a direct and increasingly dramatic impact on the performance. For the best result, design for limited results, leaning on relevancy tuning to get the best first. Pagination through search results is efficient and quick when using the searchAfter token mechanism.
When deployed in Atlas, the scalability of the search process can be controlled by adding and increasing the use of Search Nodes. You can set specific nodes in your cluster to run only the MongoDB Search mongot process. When mongot is run separately, availability improves, and workloads are balanced. To learn more, see the Search Nodes Architecture documentation.
Bottom up: Lucene and mongot
At the core of MongoDB Search lives Lucene, a mature, powerful, open source search engine library. Lucene’s solid abstractions shine through the calling layers, as well as its limitations.
“Index objects” are the individual units of data (Lucene also calls them “documents”, though they are referred to as “index objects” to avoid more terminology confusion since they aren’t necessarily one-to-one) added to the Lucene index, consisting of one index object per database collection document, plus one for each mapped embedded document. Embedded documents are covered separately below. How sub-documents nested within main collection documents are handled, depending on scale, could impact the physical model sharding design. This means using embedded documents will result in a higher number of indexed Lucene objects and needs to be considered for sizing. Lucene has a two-billion limit on index objects. Going beyond 2B docs requires partitioning or sharding.
Indexing
After an index configuration is created/updated, all documents in the collection are (re)indexed. All fields of each document are available for indexing, and mapped (or ignored) according to the index configuration. The physical storage space taken by the index structure depends entirely on the index configuration. A simple configuration adjustment can have a dramatic impact on index size (and thus speed to build, etc).
Searching
The mongot process powers the $search stage, handling successive requests to pull search results in batches. These remote calls for batches of results shape what’s performantly possible. Lucene is optimized to serve and page through the most relevant or top-sorted documents quickly, satisfying the most common search use case of where it is expected users will find what they are looking for in those top results. Queries often match vastly more documents than would be presented to a user at one time, and only the top ones may be of interest.
The successive batches of results requested by the $search stage allow for the actual documents returned to be cut off at a desired limit. Regardless of how many matching results are actually brought back into the aggregation pipeline, the exact count of matching documents can be provided, or a lower bound estimate for better performance, so the user can be presented with the magnitude of results available.
Summarizing the physical constraints
The key performance tips and system limitations are detailed on the MongoDB Search Index Performance page. Here are the main physical constraint highlights as they pertain to designing a schema for search use cases:
The database and search server are separate processes and potentially/ideally on separate nodes for more scalability.
Batching a large search result set across processes and the network can be a major bottleneck.
2.1 billion index objects per MongoDB Search index; index objects consist of one for each document, and one for each explicitly mapped embedded document.
Schema considerations
MongoDB Search maps document fields to indexes on Lucene index objects by field name and the data type. This direct connection to field name and type dictates what’s searchable and in what way, and is a crucial consideration in designing a schema for a search workload.
Consistent document structure matters. Field content is indexed based on the field name, data type, and structure of each collection document. It’s important to standardize document structure in MongoDB Search-backed collections:
_id: If the collection is merged/aggregated specifically for search from data structured differently in other collections, stable identifiers matter for updates.
Do not mix field types. For a given field name, its type should remain the same in all documents that have that field. Example: A price field is a floating point number price:37.99 in one document. It should not be a string price: “4.95” in another document.
If you plan to search or facet on any aspect of nested documents, a consistent layout is crucial. Index mappings are based on field name and structural placement (aka “path”).
Field name is the key. MongoDB Search architecture works best with a simple field_name=value structure, as index mapping configuration is first driven by the name of a field, and the internal Lucene index data structures are founded upon that field name.
Further, if field types are inconsistent, each will be indexed based on the field mapping setting for that particular type. Queries and faceting are also type-specific. In the above price example, a numeric range query would only be able to match the numeric price value, and string faceting would only show the string price value.
Partial indexing and transforming documents
What you have might not be exactly what you want to index, so there’s a handy way to use Views with MongoDB Search, providing a flexible mechanism to omit or morph a document during indexing. Here are some use cases where a View shines:
Computed (Search) Pattern: querying for array size
Partial indexing: Skipping docs from indexing
Unsearchable fields: Removing a sensitive field
Flattening Attribute Pattern
Vector dimension reduction
Textualize documents
A search index compatible View is a constrained pipeline that can include documents using a $match or, stated another way, you can exclude documents that should not be indexed/searchable, and can morph the resultant document using $addFields and $set stages.
Let’s go through the steps to create and index a view, using the Computed Pattern use case.
You’ve got documents that have a reasonable size-constrained array present on documents, something like this (to be generic):
And you need to query by the number of items in the values array. A core search mantra is to “index it like you want to find it”, so it would be ideal if we had a searchable num_values field. Sure, we can create and maintain this in our application at every document create and update touch-point.
Using the MongoDB shell, here’s how to create a View on a collection. Here, the collection of documents is called docs, and the view is simply called view:
The results from the view during the indexing job are then mapped through the index configuration, allowing you to index fields and values that are created or augmented in the view.
Without using the view, or in some way maintaining the number of values and indexing it, searching for a document with a particular number of values is a full collection or index scan at the B-Tree level and not even possible at the Search level.
Partial indexing
This View only indexes non-archived documents:
Unsearchable fields
This View removes a field from the indexing so that it will not be searchable, even if mapping was fully dynamic.
Flattening Attribute Pattern
Here’s a document that uses the Attribute Pattern for arbitrary keys/values. Those keys and values are not really suitable for indexing and searching with MongoDB Search.
The following View converts the Attribute Pattern into easily indexed uniquely named fields with values.
With that indexing transformation, attributes are queried by their transformed field name:
Vector dimension reduction
With Matroyshka vectors, dimensions can be truncated without adversely affecting distance computations by much. Take this 8-dimensional vector:
With the following transformation View, two additional fields are created, truncating the original 8-dimensional vector into 2 dimensions and also 4 dimensions. The associated index configuration would map and index these vector fields appropriately for querying.
Textualizing documents
Given a basic movie document, a full plain text concatenation and formatting of all relevant, searchable text can be very useful for vectorizing (or using as a single full-text searchable blob).
The following View builds a descriptive_text field, concatenating all of the structured fields into a single textual blurb suitable for autoembedding, full-text search on a single field, and possibly for display purposes as a summary:
The view specification may not be pretty, but it’s easy to develop and work through the nuances of these sophisticated expressions in tools like Atlas UI or Compass, where you can see the results or errors and refine before making a View and indexing from it.
View transformations are a solid tool for tuning a search index to suit your needs. Use it when it fits!
Field mappings
Whether and how a field is to be indexed or not depends on the index configuration mapping.
The logic of whether a field is to be indexed follows this flow:
An explicit field name mapping takes precedence.
If dynamic field mapping is enabled, and the field is not explicitly mapped, it will be mapped using the rules for that field's data type. Dynamic indexing rules are either an explicit typeSet specifying which types to index or an implicit “typeSet” that indexes all the basic field types, including strings as analyzed, indexed fields.
Dynamic field mapping can be enabled at a document's top level and applies to all nested sub-documents, or can be individually controlled at any particular nested document level.
The power of typeSet
The default MongoDB Search index configuration specifies a fully dynamic mapping:
Which is an implicit “typeSet” with these mappings:
While this default flattens and index-mostly-everything mapping is convenient to be able to immediately query powerfully across all common data types in the fields on the documents in a collection, rarely is it the best configuration for your querying needs.
It is strongly recommended to define a typeSet in your index configuration for controlling settings on strings. The default for string fields turns on all the settings bells and whistles, such as storing the field value for highlighting and computing/indexing term offsets for highlighting… for every string field mapped! That not only adds up, but it also multiplies up too! Huge savings can be made by turning down some of the default settings with a typeSet like this;
If you’ve been using dynamic: true for string fields, give that “lean_strings” configuration a try and compare index size. If you’re using highlighting, you’ll need to explicitly map that field to have store: true (a requirement for highlighting, but not a commonly used feature and certainly not on every string field of a document).
To treat a MongoDB Search index like a bunch of independent “B-tree”(-like) indexes so your queries can be served by fast indexes across any arbitrary number of fields and data types, simply ensure you’re indexing strings as token and include all other data types needed:
Note that storedSource is turned on to provide “index covered query” behavior, returning values directly from the search service, bypassing database lookup for each result document.
Sometimes you want to facet, sort, and equals/in query on a string value, but you also need full-text word-level search on it, too. This configuration illustrates how to index every string field as both a token type and an analyzed string type:
If you’re dynamically indexing string fields, a typeSet to at least set store: false and indexOptions: “positions” is greatly recommended - your index size will thank you! There is likely only a single field or two that need to be highlightable.
Attributes on nested documents can be handled with a sub-document level mapping to encapsulate a section or all MongoDB Search-able fields for a document.
Field mapping details are officially documented in the Define Field Mappings doc.
These are the field types supported by MongoDB Search alongside the search operators available for those types:
The * annotated types are only available on explicit field mappings and do not apply dynamically. Both document and embeddedDocument mappings apply to arrays of nested documents; the document mapping applies to dynamic mappings, whereas embeddedDocument only applies explicitly.
Basic data types
Indexing behavior is configurable within the options for the data type of that particular field instance. Numeric field types can be indexed for numeric equals, in, and range findability and range bucket faceting (counts per numeric range over a result set), and likewise for date field types.
Boolean values are indexed for equals and in findability.
Strings
String fields are special, as they are analyzed and broken down into terms that are indexed into an inverted index data structure designed for word/token-level matching. Query ability with fast, relevance-ranked results on a pile of full-text documents stems from wisely tokenizing and further analyzing and organizing string text into an inverted index.
Arrays
The data type of the values within an array is used to determine how the field is mapped and indexed. Arrays themselves aren’t their own field type. Nested documents by default are mapped as document type fields, and any fields within them are flattened and indexed as if they were fields on the main document with dotted name syntax, bringing all values of fields at the same relative path into an array of flattened values. There are good examples in the How to Index the Elements of an Array documentation.
Nested documents
The MongoDB mantra “data that is accessed together is stored together” builds upon the strengths of the document database, driving effective designs by nesting related and commonly accessed data into single values or arrays of attributes and sub-documents. MongoDB Search best practices adjust the mantra to “data that is searched together is indexed together.” By the nature of its architecture, a MongoDB Search index indexes all the mapped fields of a document together in a single structure so that any number of the indexed fields can be used (and intersected) in a single query for matching and ranking. There’s one index object per collection document, at least. Nested document structures present an interesting search challenge — are these structures to be intricately and independently matchable, or perhaps more straightforwardly considered a flat set of attributes used to match and return only the main document?
The results of a $search are typically only the matching top-level documents, not individual nested documents on their own. Top-level documents returned include the full document, including the nested structure; nested documents are, again, not typically returned independently or separately from their top-level parents. However, nested documents can be mapped as first-class index objects and matched, ranked, returned, and faceted independently of their ancestors and top-level containing structures.
There are two ways to handle indexing and searching content within nested documents: document and embeddedDocuments.
Flattened/document type: This is the default with dynamic mapping. It flattens sub-document structures into values on the root document. Queries match these flattened arrays and lose the nuance of the intra-embeddedDocument matching.
embeddedDocuments: You must configure this — it doesn’t come by default. It indexes each specified embeddedDocument as a separate Lucene “index object” tied to the parent index object. It allows for embeddedDocument-specific queries, similar to the $elemMatch stage.
Nested document structures can be several layers deep, and each level can be mapped in MongoDB Search as either document or embeddedDocuments—making it possible to index and intricately query multiple nested levels deep. Just because you can map things in this way, however, it doesn’t mean you should. Simpler and flatter is generally better, but some use cases will benefit greatly from indexing nested documents independently.
Let’s illustrate the pros and cons of the various ways to index nested structure with an example, Parents collection with a single “Mom” document, which has two kids. Kid1 is 5 years old. Kid2 is 11 years old. It looks like this in JSON:
Flattened index objects
Use document type mapping on paths containing nested documents (implicit document) to flatten the fields into a single index object in Lucene. A dynamic: true mapping implicitly uses document mapping for nested sub-document structures.
This flat structure allows us to query for Parents with a child 5 years old and also have a child named “Kid2”, matching this “Mom” document. However, we cannot query for Parents who have a 5-year-old child who is named “Kid2”. The original document, as far as the Lucene “index object” sees it, effectively looks like this, where the names and ages are not tied together:
Index object representation of document mapping of nested structure.
Embedded index objects
This same example document, with children mapped as an embeddedDocuments type, creates three index objects, one for each mapped embedded document and one for the main document.
This representation of what the example “Mom” document looks like as embeddedDocuments illustrates the difference from a document mapping:
In the index structure, these documents are co-located as a physical contiguous block. Any update to the document, in any piece of its structure, reindexes that document, which in turn reindexes the entire block of mapped embedded documents.
Operationally, an update to an index object in Lucene is atomically a delete and re-add. Reindexing a document with mapped embedded documents causes that entire block of index objects to be deleted and then re-added. Constantly updating documents with mapped embeddedDocuments thrashes the Lucene indexed objects structure; the more embedded documents, the heavier it is to add, update, or delete a collection document.
Searches that query embedded documents are able to query each independent embedded “index object” document separately, along with all indexed ancestor values, and factor in the associated score into the returned documents' computed relevancy ranking score. Embedded documents can be individually returned and faceted using returnScope within $search; documents returned using returnScope are returned directly from the search engine with no database lookup, a great benefit when the use case fits. See Query, Filter, and Retrieve Arrays of Objects for more details on this powerful feature.
To embed or not? It depends. Mapped embedded documents come at a cost and should only be used if the nuanced matching is needed and the explosion in the number of index objects is manageable. And even if that type of matching is needed, another deciding factor is the scale — a single Lucene index can contain a maximum of 2B documents. If you’re pushing up to that scale, partitioning and sharding are the next steps to consider. A collection of 200M database documents, each with 10 embedded documents, explodes to 2.2B (!) “index objects” ((10+1) * 200M), blowing up beyond what may be feasible or affordable.
Does the use case require matching across exact sub-document structure? Perhaps those sub-documents deserve to be first-class main documents in a search-based collection rather than contorting a nested structure to achieve something that fits best with an unwound, flatter structure. When considering embedded document mapping for search use cases, also consider turning the document structure inside-out so that the entities to be returned from a search request are first-class documents.
On to the next section
The practical considerations from physical and system constraints to field mappings and transformation views have been addressed above. These important details provide the basis for configuring a MongoDB Search index. Building on this foundation, Part 2 will cover common use cases and the patterns that have emerged to address them.
Next Steps
To learn more about MongoDB Search, visit our documentation.
Ready to start building? Get started with MongoDB Atlas for free today.