Embedding models are a type of machine learning model, often based on neural networks like transformers that convert complex and unstructured data into dense numerical vectors called embeddings.
These embeddings capture the semantic meaning and contextual similarities of the data, enabling efficient comparison, clustering, and classification. This makes them highly useful in applications such as search engines, recommendation systems, retrieval-augmented generation (RAG), chatbots, and other AI applications.
Table of contents
Embedding models fundamentals
What are vector embeddings?
Embeddings refer to the technique of capturing semantic meaning and relationships in a numerical representation. At their core, vector embeddings are high-dimensional vectors, typically ranging from 256 to 4096 dimensions, although this can vary depending on the specific use case, data type, and model.
Each dimension contributes to capturing different aspects of the input data. Closely related content will produce similar vectors. For example, in language embeddings, some dimensions might represent formality, while others capture subject matter or sentiment.
Since embeddings are high-dimensional vectors, mathematical computations can easily determine their similarity, enabling efficient semantic search. To find similar vectors, two key methods are common, both relying on similarity metrics such as cosine similarity, Euclidean distance, or dot product:
Nearest neighbor search (NNS): Identifies the closest vectors based on the chosen similarity metric.
Hierarchical navigable small world (HNSW): An optimized approach that enhances NNS by organizing data into a multi-layer graph, where higher layers perform a fast, coarse search to filter candidates and lower layers refine the results for greater accuracy, enabling scalable and efficient approximate search.
How are embeddings generated?
Vector embeddings are generated by processing input data with embedding models, which are trained to recognize patterns and relationships from large datasets. In text data, for example, they capture semantic associations between words that frequently appear together, while in images, they identify distinct visual features. This enables embedding models to convert unstructured data such as text, images, videos, and documents into vector embeddings that encapsulate their meaning or key characteristics.
The typical process for generating embeddings involves a pre-processing step. This includes data preparation activities such as data cleansing, standardization, normalization, and chunking.
These preprocessing techniques are essential to improve data quality, ensuring that the generated embeddings effectively capture the meaning and context of the original data.
How are embedding models trained?
Embedding models are trained on large datasets using machine learning techniques to learn meaningful representations by recognizing patterns and relationships. Most commonly used embedding models are self-supervised, meaning they do not rely on labeled data for training but instead learn from structured patterns within the unlabeled training data. Self-supervised techniques used for training embedding models include masked language modeling (MLM) (e.g., BERT) and contrastive learning (e.g., SimCSE, SBERT).
For text embeddings, models like Word2Vec, GloVe, or BERT are trained by predicting missing words in sentences or understanding how words relate to each other. For example, Word2Vec captures semantic relationships, allowing it to infer analogies like “King - Man + Woman = Queen,” demonstrating how words are mapped based on their contextual meaning. This is possible because the model encodes gender as a consistent directional relationship across the vector space. The vector difference between "Man" and "Woman" captures the concept of gender. Similarly, it encodes the concept of royalty in the relationship between "King" and "Queen." These relationships emerge organically from the self-supervised process without explicit human labeling and explain why embedding models are so popular and effective at capturing deeper semantic relationships between concepts.
Similarly, in image embeddings, models such as convolutional neural networks (CNNs) or vision transformers (ViTs) learn to identify shapes, textures, and objects by analyzing large sets of labeled or unlabeled images.
During training, the model adjusts its parameters to ensure similar data points are mapped closer together in the embedding space while unrelated data points are kept farther apart. This process is guided by an optimization algorithm that improves accuracy over many iterations. Once trained, the model can generate embeddings for new data, capturing important qualities and relationships in a structured way.
Application of embedding models
Embedding models have a wide range of applications across various fields due to their ability to capture semantic meaning and relationships in data. The main applications are:
Natural language processing (NLP): Captures semantic meaning in text for tasks like sentiment analysis and machine translation.
Search and recommendation systems: Enhances personalization by matching user preferences with relevant content or products.
Computer vision: Analyzes images for tasks like image recognition and object detection.
Speech recognition: Converts audio signals into text for applications like voice assistants and transcription services.
Chatbots and conversational agents: Enables natural and contextually appropriate conversations in chatbots and virtual assistants.
Natural language processing
Embeddings generated by text embedding models are instrumental in understanding the meaning and intent behind text data. These embeddings are used in various use cases, such as:
Sentiment analysis: Determines the emotional tone of text, commonly used in social media monitoring and customer feedback analysis.
Text classification: Categorizes text into predefined labels, useful for spam detection, topic classification, and document organization.
Language translation: Facilitates machine translation by capturing contextual meanings across different languages.
Named entity recognition (NER): Identifies and classifies entities (such as names, locations, and dates) within text for applications like information extraction and automated summarization.
By capturing the contextual nuances of language, embeddings enable more accurate and meaningful interpretations of text.
Search and recommendation systems
Embeddings are heavily used in search engines to enhance their ability to understand user queries and retrieve relevant data. Specifically, embedding models are commonly used for semantic search, where the goal is to comprehend the intent behind a query and its context. This process involves converting both queries and documents into high-dimensional vectors in a semantic vector space. These vectors are typically stored in a vector database, allowing for efficient vector search operations. By performing a nearest neighbor search using measures such as cosine similarity or Euclidean distance between the query's embedding and the stored document embeddings, the most relevant documents can be identified and retrieved, significantly improving search accuracy and user satisfaction.
Similarly, in recommendation systems, embeddings can be leveraged to compare user queries with product descriptions, movie outlines, or other relevant content. By doing so, the system can identify and suggest the most pertinent recommendations. This approach enhances the relevance of recommendations, leading to a more personalized user experience.
Common examples of companies using this approach include Netflix for movie and TV show recommendations, Amazon for product suggestions, and Spotify for music and podcast recommendations.
Image processing and computer vision
In computer vision, embedding models transform visual data into numerical vectors, capturing essential features and patterns.
This enables applications like:
Image similarity search, where visually similar images are identified based on their embeddings
Image classification, categorizing images into predefined classes
Object detection, identifying and locating objects within images
These capabilities are typically used in fields such as medical imaging, autonomous driving, and security systems.
Retrieval-Augmented-Generation (RAG)
LLMs are trained on vast amounts of data, including much of the internet, forming what is known as their parametric knowledge. This allows them to respond to a wide range of questions across numerous topics. However, they lack specific knowledge of certain domains or an organization’s internal knowledge base. Retrieval-augmented generation (RAG) provides a cost-effective way to bridge this gap without the need for expensive retraining. By retrieving relevant external information, RAG enhances the accuracy and precision of LLM outputs, making them more context-aware, even for proprietary or private knowledge.
RAG combines document retrieval with large language models (LLMs) to produce more accurate and contextually relevant responses. This technique is widely used in LLM-based chatbots and other intelligent applications to incorporate external knowledge into LLMs. RAG uses semantic search to find documents relevant to the user's query. These documents are then included in the prompt, enabling the LLM to generate answers grounded in the provided data. As a result, embedding models are indispensable in the semantic search component necessary to enable RAG in LLM-based and agentic applications.
Types of embedding models
Embedding models come in various forms, each tailored to specific needs and applications. Understanding these different types of models is paramount for effectively leveraging their strengths and addressing specific use cases.
Embedding model sizes
As with other types of models, embedding models come in different sizes, balancing performance and computational efficiency. They are generally categorized as small, medium, or large, with the choice depending on the required accuracy in capturing semantic meaning and the available computational resources. The dimensionality of the output vectors (i.e., the number of data points in each vector) directly impacts retrieval performance, storage, and computational efficiency. Typically, these categories are defined as follows:
Small: 128 - 256 dimensions
Medium: 256 - 512 dimensions
Large: 768 - 2048+ dimensions
Domain-specific vs. general-purpose
Embedding models can also be categorized into general-purpose and domain-specific models.
General-purpose models: Designed for a wide range of tasks across different domains, offering broad applicability.
Domain-specific models: These models are fine-tuned for particular industries (e.g., legal, medical), providing specialized insights and improved accuracy in fields where small differences in terminology can significantly affect meaning. For example, in the medical field, the terms myocardial infarction and cardiac arrest are not interchangeable, even though both relate to heart conditions. A general model might treat them as similar, while a domain-specific model would recognize the critical distinction.
How are domain-specific embedding models trained?
Domain-specific embedding models are typically trained using transfer learning and fine-tuning. Developers start with a pre-trained general-purpose model and collect a corpus of domain-specific texts (e.g., medical journals, legal documents, or financial reports). The model is then fine-tuned on this specialized dataset, learning the nuanced terminology, relationships, and context-specific meanings unique to that field.
Fine-tuning for domain-specific embedding models generally employs self-supervised techniques, such as masked language modeling and contrastive learning, similar to those used in pre-training general models but applied to the specialized corpus. This approach is more efficient than training from scratch as it leverages the foundational knowledge of the general model while adapting it to recognize domain-specific patterns and semantic relationships.
Limitations of general-purpose embeddings
While general-purpose models work well for broad applications, they often fall short for specialized domains such as legal, medical, or finance.
These models may struggle with:
Limited vocabulary coverage for technical terminology
Difficulty capturing industry-specific nuances
Reduced accuracy when processing domain-specific terms
This usually leads to less relevant or imprecise results in semantic search or retrieval-augmented generation implementation in specialized applications. In practice, this means LLM-based applications are more likely to hallucinate and generate irrelevant responses in domain-specific areas.
Advantages of domain-specific embedding models
Domain-specific embedding models are trained on industry-specific data to address the limitations of general-purpose models and improve efficiency within a narrower scope. For example:
Legal embedding models enhance the analysis of legal documents.
Medical embeddings improve the understanding of medical terminology, processes, and formatting.
Financial embeddings specialize in interpreting financial contexts.
These models provide more precise and contextually relevant insights for their respective domains.
When to use general vs. domain-specific embeddings
General-purpose embedding models provide broad applicability and adapt easily to new tasks. They require less maintenance and generalize well across different domains. When versatility and efficiency outweigh the need for domain-specific precision, these models are a strong choice.
Domain-specific models, on the other hand, prioritize precision. They capture domain-specific nuances more effectively, leading to higher accuracy in specialized tasks. This makes them ideal for applications where detailed understanding and relevance are more critical than broad adaptability. For instance, a financial chat assistant leveraging a general embedding model for its knowledge base might treat the terms “liquidity” and “solvency” as interchangeable or close and retrieve documentation related to both, which would lead to confusing response generation. However, a domain-specific financial model would recognize the distinction and provide more accurate, relevant, and specific financial information.
The choice between these models depends on the level of domain specificity required. High-quality embeddings improve performance across both general and specialized use cases, but in highly technical fields like legal or medical applications, domain-specific models often provide a necessary improvement in accuracy.
Instruction-tuned embedding models
Embedding models, whether domain-specific or general-purpose, can also be instruction-tuned, meaning they are trained to respond to additional instructions usually prepended to the input text. This approach is particularly useful in retrieval and search tasks, where inputs can be labeled as documents or user queries, for example. By providing more context, instruction tuning optimizes embedding generation, ultimately improving retrieval performance.
Multimodal embedding models
One key characteristic of embedding models is the range of modalities they support. These modalities extend beyond text to include images, audio, and video. Early embedding models were typically unimodal, focusing on a single type of data, such as text, audio, or images. However, much of the world's knowledge exists in mixed formats, such as PowerPoint presentations that combine text and graphics, PDFs containing text, images, and charts, or screenshots that capture complex visual information.
With recent advancements, multimodal embedding models have emerged, allowing data from different formats to be encoded into vector embeddings within the same semantic space. These models can understand and relate information across multiple modalities, making them highly effective for tasks such as document retrieval, content recommendation, and cross-modal search.
Mixed modality search for RAG
Multimodal embedding models are particularly valuable for retrieval-augmented generation (RAG) use cases to enable mixed modality search over a knowledge base.
RAG is widely adopted in enterprises to integrate vast amounts of private and proprietary knowledge into LLM-based applications. Historically, RAG systems relied solely on text-based document elements. However, with multimodal embedding models, organizations can now incorporate multiple data types, such as images, figures, and charts, into the same semantic vector space, significantly enhancing the available knowledge, effectiveness of information retrieval, and consequently improving response accuracy.
Support for interleaved text and images
Traditional multimodal embedding models generate embeddings for either text or images separately and combine them in a shared representation. This approach requires heuristic-based document parsing to decompose complex documents into distinct elements like text, images, and charts. However, such preprocessing is often error-prone, particularly when dealing with intricate layouts, interspersed figures, and embedded images.
In contrast, multimodal embedding models capable of processing interleaved text and images eliminate the need for extensive preprocessing, offering greater flexibility and significantly improving retrieval accuracy. In addition to processing complex documents, interleaved multimodal embedding models are highly efficient for vectorizing screenshots, annotated images, and PDFs, significantly increasing the amount of knowledge that can be leveraged. By directly capturing relationships between textual and visual elements, these models enhance the effectiveness of RAG systems in handling rich, multimodal enterprise knowledge.
These advanced multimodal models (e.g., voyage-multimodal-3) leverage architectures similar to modern vision-language transformers, making them significantly more efficient than traditional multimodal embedding models. Voyage AI’s voyage-multimodal-3 or Colpali are examples of such architectures. Their ability to integrate multiple modalities into a unified representation improves retrieval performance while reducing the operational burden associated with preprocessing documents. These models generate unified embeddings for sequences containing both text and images, capturing cross-modal relationships within a single representation.
How to choose an embedding model
There are many factors influencing the choice of an embedding model. Let’s consider the following key factors:
Data type: What kind of data needs to be embedded? Options include text, images, audio and video, or fully multimodal embeddings such as interleaved text and images.
Cost: Many embedding models are priced per token. For large datasets, cost considerations can significantly impact feasibility.
Dimensionality: How many dimensions should the vector embedding have? Modern embedding models can reach up to 8,000 dimensions. The ideal choice depends on your use case:
Lower dimensionality leads to faster processing, lower storage costs, and cheaper embeddings.
Higher dimensionality captures more information but requires more storage and compute, increasing latency and costs.
The best approach is to experiment with your own data to determine the optimal balance between performance and efficiency.
Application use case: Whether for search, recommendations, classification, or retrieval-augmented generation (RAG), the embedding model should align with your specific application needs.
Domain specificity: Some applications, such as those in legal, medical, or code-related fields, require domain-specific embeddings to improve accuracy and relevance.
Quantization awareness: Quantization-aware embedding models are designed to optimize embeddings while considering the effects of quantization. This helps reduce memory usage and computational costs without significantly compromising accuracy. It's particularly useful for deploying models in resource-constrained environments.
Integration compatibility: Check the embedding model provider’s API availability (service level agreements and latency) and ensure compatibility with your AI stack.
Performance metrics: The MTEB leaderboard from HuggingFace benchmarks embedding models across multiple NLP tasks, including retrieval, classification, clustering, and reranking. It primarily assesses models in a zero-shot setting, meaning they are tested on tasks they weren’t explicitly fine-tuned for, helping gauge their generalization ability across unseen data. This makes MTEB a valuable resource for helping compare models in a wide range of applications.
The ideal embedding model strikes a balance between accuracy, efficiency, and practical implementation constraints based on your specific needs. Additionally, consider how frequently you plan to change embedding models, as regenerating embeddings for your entire knowledge base would be required each time.
Boost search and RAG accuracy with Voyage AI and MongoDB
Search and retrieval-augmented generation (RAG) systems can be optimized in numerous ways. MongoDB Atlas Vector Search offers developers advanced query capabilities such as hybrid search, metadata pre-filtering, and vector quantization. However, to achieve even greater accuracy and performance, leveraging best-in-class embedding models is paramount.
Next steps
Start building today. Simplify your developer experience and create more value with a fully managed, secure database integrated with a vast AI partner ecosystem. This ecosystem includes all major cloud providers, LLM model providers, and system integrators.
Relevant resources
Explore MongoDB Atlas, the modern, multi-cloud database for developing AI applications, with built-in search, vector search, and more capabilities. Register for free now.
To learn more about Voyage AI, read our blog announcement.
Conclusion
Embedding models represent an essential component in modern AI applications. While general-purpose embeddings provide broad utility, the trend toward domain-specific models offers enhanced precision for specialized applications. Whether for semantic search or retrieval-augmented generation, leveraging domain-specific embedding models can help improve accuracy and reduce hallucinations in AI systems. Moreover, success in implementing embedding models requires careful consideration of use cases, technical requirements, and the balance between precision, flexibility, and resources.