We’ve all experienced it. You type a keyword into a search box, hit enter, and then…disappointment. Sometimes, search engines will retrieve relevant documents, but other times, the answers are completely off. (And occasionally, you wonder if the search engine even read what you typed.)
Why does this happen? Most of the time, it’s because of how the search engine interprets user queries, and whether it’s using lexical or semantic search.
Lexical search is a traditional search method. It relies on exact word matches to produce a result. If you type “car,” you’ll only get responses for “car,” not “automobile” or “vehicle”—words with similar meaning that semantic search might find. Semantic search understands the intent behind the search term, so it may include these related concepts in its response.
If user queries don't return relevant results, it could stem from the system using keyword-based search instead of semantic search, or from lexical search that hasn’t been tuned for your specific use case. Getting relevant results requires fine-tuning in your search system.
In this article, we’ll explore what lexical search is, how it works, why it’s so popular, and where it can falter. We’ll also compare it to semantic search that powers systems like Google and Bing, as well as AI assistants, recommendation engines, and enterprise knowledge systems.
Table of contents
- Understanding how lexical search engines process queries
- How lexical search works behind the scenes
- Beyond lexical search: How semantic search clarifies keyword intent
- Lexical search vs semantic search
- Beyond exact word matches: How semantic search started
- How semantic search uses vector search
- Hybrid search systems: Combining lexical and semantic search technology
- Conclusion
- FAQs
Understanding how lexical search engines process queries
Before search engines could gauge intent, they were 100% literal—they only returned exact matches to the search term. Early Internet tools like Archie, Veronica, and Jughead worked this way using a technique called string matching, which is the same principle behind how many databases, including MongoDB, index strings today.
String matching
String matching in lexical search looks for exact matches for a word or phrase in filenames or directory entries. If the words match exactly, you get a result. If not, no result. It’s that simple. There is no guessing, no contextual understanding, just a letter-by-letter comparison.
This worked in the early days of the web but quickly ran into limits. Once the amount of online content exploded, it became clear filenames alone weren’t sufficient. With millions of documents now available, users couldn’t possibly know how files were named, so search engines needed a way to search the actual content of web pages. Full-text search solved this problem using a new technique: the inverted index.
The inverted index
An inverted index is a pre-built lookup table that maps each word to the document(s) where it lives. When you search for “apple pie,” the system looks up those words in the table instead of scanning through every document. This is why lexical search is so fast—all of the organizing work happens once, when the document first enters the system.
The inverted index is the foundation of traditional lexical search. From the early days of Google to today’s library databases, website search boxes, and enterprise software, it’s still one of the fastest ways for information retrieval.
Modern database platforms like MongoDB Atlas Search build on this foundation, providing developers with ready-to-use full-text search capabilities that handle tokenization and indexing automatically.
How lexical search works behind the scenes
Keyword, or lexical, search may be known as the most basic search method, but what happens after the user submits a given search query is rather complex.
Lexical search breaks the query into tokens
The lexical search process starts with tokenization, which is when the search engine breaks down your query into smaller pieces called tokens (usually words). Tokenization decides what actually counts as a “word” in your query.
For example:
- A query like “mother-in-law suite” may be treated as one exact phrase or broken into four words, or tokens.
- “Rock ’n roll history” could be kept as one phrase or split into separate words.
- Contractions like "don't" are typically kept as one token, though some systems can be configured differently.
Different search engines make different choices about how they handle things like hyphens, shortened words, contractions, and so on. Tokenization also handles punctuation, capitalization (should “Blue” match “blue”?), and even languages that don’t naturally separate words, like Chinese or Japanese. How these variations are handled can affect the search results.
Lexical search finds documents quickly
Once your query is tokenized, the search engine searches the inverted index we talked about earlier. This structured data method is lightning-fast, even across millions of documents. But just finding the right documents isn’t enough. The search engine still needs to rank the results to show the most useful ones first.
Lexical search ranks results with TF-IDF and BM25
Finding documents that contain your keywords is the first step. The next step is ranking the documents’ keywords to determine which documents are most relevant.
TF-IDF: Early ranking formula
One of the earliest and still most widely used ranking methods is TF-IDF, short for “term frequency-inverse document frequency.”
It sounds complex, but it’s just two methods combined:
Term frequency (TF) counts how many times a keyword appears in a document. Higher numbers mean the word is more likely to be important to that document.
Inverse document frequency (IDF) scores how rare the word is across the entire indexed collection. Words that appear everywhere (like “the”) get a low number; uncommon words (like “serendipity”) or specialized terms (like “API”) get a high number.
How it works: The search engine looks at how often a word appears in a document (TF) and how rare that word is across all documents (IDF). It combines those two numbers to create a score. That score tells the system how relevant the document is to the query. Different systems may adjust the formula in small ways, but the main idea stays the same.
Example: Let’s say you need to bring dessert to a family picnic and search for "apple pie recipe." The keyword search engine looks for pages where "apple," "pie," and "recipe" show up often—and prominently.
A blog post that reads, "The Ultimate Apple Pie Recipe for Summer Picnics," and also contains those words in the opening paragraphs, will rank high. However, a page about kitchen gadgets that mentions "apple pie" once ranks at the bottom. Even though the keywords are there, it’s not considered relevant because the page isn’t about making pie—it’s about blenders and mixers. With TF-IDF, the search engine can tell the difference, and returns the recipes you were looking for.
BM25: Refined ranking formula
TF-IDF ranks documents well in most search systems, but it does have limitations. In longer documents, a keyword may only show up a few times, so TD-IDF might rank it less relevant than it really is. In shorter documents, a keyword that’s mentioned just a few times can improve the TD-IDF score, which wouldn’t be accurate either. To balance this out, many search engines add Best Matching 25 (BM25) to the search retrieval process.
Search engines that include BM25 automatically:
Reduce the impact of repetitive keywords: Stuffing in the same word, like “recipe,” a hundred times doesn’t make a page a hundred times more relevant. BM25 reduces the impact of repeated words.
Normalize document length: Longer documents naturally contain more words but, because of BM25, they don’t score higher based just on that. BM25 weighs short blog posts and long reports equally.
Note: Different industries tune the formula to fit their needs. For example, a legal search engine may rank documents differently than one built for medical research.
Beyond lexical search: How semantic search clarifies keyword intent
Lexical search engines don’t just match words randomly. With Boolean operators like AND, OR, NOT, and NEAR, they can support more complex queries.
For example:
- Searching for "genetic sequencing" AND "disease prevention" will return only documents that discuss both ideas.
- Searching for "AI” NEAR "ethics" would find results where those two terms are near each other.
These advanced query techniques offer users more control, especially in academic research, legal databases, and enterprise search systems.
Lexical search vs semantic search
Lexical search and semantic search have different strengths. Lexical search works well when your query matches the exact phrase used in the content, but it can struggle when multiple words are used to express the same meaning.
One challenge with lexical search is the vocabulary mismatch. If you search for “airplane,” but the document only says “jet,” a basic lexical search won't find it automatically. That said, you can configure synonyms to connect related terms, which is a practical solution many systems use.
Lexical search can also have a hard time with:
Abbreviations: “Chitown” vs. “Chicago” or “SanFran” vs. “San Francisco”
Related concepts: "marketing campaign ideas" vs "promotional strategies"
Multiple meanings: where a word like "stream" might mean a small river or watching a video online
Here are a few more examples:
- A customer enters “billing support” in the search box, but misses help center articles that say “account charges assistance.”
- A software developer searches for “null pointer error” but doesn’t see critical bug reports labeled with “NPE” or “segfault.”
- A job seeker types “marketing lead” and overlooks job listings for “growth strategist” or “demand generation manager.”
In each case, the problem isn’t that the information isn’t available; it’s that lexical search can't recognize related concepts or connect slight differences in phrasing. This is why semantic search has gained momentum.
Beyond exact word matches: How semantic search started
About 25 years ago, developers became interested in how search engines often missed relevant and accurate results. To improve this, they began experimenting with new ways to get systems to understand user intent instead of just comparing strings of characters.
In the early 2010s, with the advent of machine learning (ML) and natural language processing (NLP), semantic search became possible. Instead of simply matching keywords, semantic search maps the relationships between words and concepts, enabling search engines to return results based on what you meant, not just what you typed. This approach provides deeper understanding of user intent through advanced vector search capabilities.
How semantic search uses vector search
Semantic search uses an embedding model and vector search technology. Instead of only matching exact words, it turns your words into numbers (called vectors) and stores them in a special database with other vectors. Words with similar meanings sit close together there. That’s why a search for “car” can also bring up “automobile” or “vehicle”—the vector search system knows they belong to the same neighborhood of meaning.
Here’s how vector search works:
- You type a search query.
- The embedded model turns your query into numbers (vectors).
- The system compares those vectors to the vectors already indexed there.
- It calculates how close they are and produces a similarity score (often a cosine similarity).
- The closest matches land at the top of the search results.
This vector search approach looks for synonyms, abbreviations, or different ways of saying the same thing, proving a deeper understanding of your query by finding related concepts.
As the vector search system grows, it can compare a query against millions of documents almost instantly. With MongoDB's semantic search, vector storage is built into the database, which means developers don't have to manage separate vector search systems when implementing semantic search. This helps close vocabulary gaps and return results that keyword search would miss.
For even more accuracy, embedding models can be trained for specific fields, like finance, healthcare, and law, so it understands the meaning behind specialized terms. Fine-tuning helps the system understand specialized terms and the broader context of a query.
Hybrid search systems combine lexical and semantic search, giving you both exact matches and meaning-based results.
Hybrid search systems: Combining lexical and semantic search technology
Many search engines now use hybrid search systems, which blend the two approaches. By combining lexical and semantic approaches, these hybrid systems can handle precise queries and broader intent at the same time.
Modern database platforms like MongoDB Atlas Search allows developers to use lexical search for precise keyword matching and semantic search for contextual understanding within the same system. Whether you’re searching for "NHL playoff scores" or "how to unclog a kitchen sink," hybrid systems can improve your chances of finding the right answer quickly.
Conclusion
The search engines we use everyday give us quick and accurate results from years of development and testing around lexical search, semantic search, and everything in between.
Lexical search excels at exact matches and enables fast information retrieval. Semantic search offers contextual understanding and semantic meaning. Hybrid search systems combine both for a flexible approach across search systems that returns more targeted results across different search scenarios.
Modern search represents a sophisticated balance of computational resources, vector search capabilities, and the reliability of traditional methods, delivering search experience users continue to use and appreciate.