Announcing Hybrid Search Support for LlamaIndex

Prakul Agarwal and Silpa Ajjarapu
October 17, 2024 | Updated: October 18, 2024
#genAI#Vector Search

MongoDB is excited to announce enhancements to our LlamaIndex integration. By combining MongoDB’s robust database capabilities with LlamaIndex’s innovative framework for context-augmented large language models (LLMs), the enhanced MongoDB-LlamaIndex integration unlocks new possibilities for generative AI development.

Specifically, it supports vector (powered by Atlas Vector Search), full-text (powered by Atlas Search), and hybrid search, enabling developers to blend precise keyword matching with semantic search for more context-aware applications, depending on their use case.

Building AI applications with LlamaIndex

LlamaIndex is one of the world’s leading AI frameworks for building with LLMs. It streamlines the integration of external data sources, allowing developers to combine LLMs with relevant context from various data formats. This makes it ideal for building application features like retrieval-augmented generation (RAG), where accurate, contextual information is critical. LlamaIndex empowers developers to build smarter, more responsive AI systems while reducing the complexities involved in data handling and query management.

Advantages of building with LlamaIndex include:

  • Simplified data ingestion with connectors that integrate structured databases, unstructured files, and external APIs, removing the need for manual processing or format conversion.

  • Organizing data into structured indexes or graphs, significantly enhancing query efficiency and accuracy, especially when working with large or complex datasets.

  • An advanced retrieval interface that responds to natural language prompts with contextually enhanced data, improving accuracy in tasks like question-answering, summarization, or data retrieval.

  • Customizable APIs that cater to all skill levels—high-level APIs enable quick data ingestion and querying for beginners, while lower-level APIs offer advanced users full control over connectors and query engines for more complex needs.

MongoDB's LlamaIndex integration

Developers are able to build powerful AI applications using LlamaIndex as a foundational AI framework alongside MongoDB Atlas as the long term memory database. With MongoDB’s developer-friendly document model and powerful vector search capabilities within MongoDB Atlas, developers can easily store and search vector embeddings for building RAG applications. And because of MongoDB’s low-latency transactional persistence capabilities, developers can do a lot more with MongoDB integration in LlamIndex to build AI applications in an enterprise-grade manner.

LlamaIndex's flexible architecture supports customizable storage components, allowing developers to leverage MongoDB Atlas as a powerful vector store and a key-value store. By using Atlas Vector Search capabilities, developers can:

Diagram depicting the architecture. In the largest box, labeled storage context, there are four other boxes titled vector store interface, key value store, document store interface, and index store interface. The Key Value store box has arrows pointing to the document store interface and index store interface boxes. Finally, the overall storage context box has arrows pointing to three exterior boxes all labeled index.
Figure adapted from Liu, Jerry and Agarwal, Prakul (May 2023). “Build a ChatGPT with your Private Data using LlamaIndex and MongoDB”. Medium. https://medium.com/llamaindex-blog/build-a-chatgpt-with-your-private-data-using-llamaindex-and-mongodb-b09850eb154c

Adding hybrid and full-text search support

Developers may use different approaches to search for different use cases. Full-text search retrieves documents by matching exact keywords or linguistic variations, making it efficient for quickly locating specific terms within large datasets, such as in legal document review where exact wording is critical. Vector search, on the other hand, finds content that is ‘semantically’ similar, even if it does not contain the same keywords. Hybrid search combines full-text search with vector search to identify both exact matches and semantically similar content. This approach is particularly valuable in advanced retrieval systems or AI-powered search engines, enabling results that are both precise and aligned with the needs of the end-user.

It is super simple for developers to try out powerful retrieval capabilities on their data and improve the accuracy of their AI applications with this integration. In the LlamaIndex integration, the MongoDBAtlasVectorSearch class is used for vector search. All you have to do is enable full-text search, using VectorStoreQueryMode.TEXT_SEARCH in the same class. Similarly, to use Hybrid search, enable VectorStoreQueryMode.HYBRID. To learn more, check out the GitHub repository.

With the MongoDB-LlamaIndex integration’s support, developers no longer need to navigate the intricacies of Reciprocal Rank Fusion implementation or to determine the optimal way to combine vector and text searches—we’ve taken care of the complexities for you. The integration also includes sensible defaults and robust support, ensuring that building advanced search capabilities into AI applications is easier than ever. This means that MongoDB handles the intricacies of storing and querying your vectorized data, so you can focus on building!

We’re excited for you to work with our LlamaIndex integration. Here are some resources to expand your knowledge on this topic:

← Previous

Strengthen Data Security with MongoDB Queryable Encryption

MongoDB Queryable Encryption is a groundbreaking, industry-first innovation developed by the MongoDB Cryptography Research Group that allows customers to encrypt sensitive application data, store it securely in an encrypted state in the MongoDB database, and perform equality and range queries directly on the encrypted data—with no cryptography expertise required. Adding range query support to Queryable Encryption significantly enhances data retrieval capabilities by enabling more flexible and powerful searches. Queryable Encryption is available in MongoDB Atlas, Enterprise Advanced, and Community Edition. Encryption: Protecting data through every stage of its lifecycle Encryption is a critical security method for ensuring protection of sensitive data and compliance with regulations like GDPR, CCPA, and HIPAA. It involves rendering data unreadable to anyone without the decryption key. It can protect data in three ways: in-transit (over networks), at-rest (when stored), and in-use (during processing). While encryption in-transit and at-rest are standard for all databases and are well-supported by MongoDB , encryption in-use presents a unique challenge. Encryption in-use is difficult because encrypted data is unreadable—it looks like random characters and symbols. Traditionally, the database can’t run queries on encrypted data without decrypting it first to make it readable. However, if the database doesn’t have a decryption key, it has to send encrypted data back to the application or system (i.e., the client) that has the key so it can be decrypted before querying. This is a pattern that doesn’t scale well for real-world applications. This puts organizations in a difficult spot: in-use encryption is important for data privacy and regulatory compliance, but it's hard to implement. In the past, companies have either chosen not to encrypt sensitive data in-use or have employed less secure workarounds that complicate their operations. MongoDB Queryable Encryption: Safeguarding data in use without sacrificing efficiency MongoDB Queryable Encryption solves this problem. It allows organizations to encrypt their sensitive data, like personally identifiable information (PII) or protected health information (PHI), and to run equality and range queries directly on that data without having to decrypt it. Queryable Encryption was developed by the MongoDB Cryptography Research Group , drawing on their pioneering expertise in cryptography and encrypted search, and Queryable Encryption has been peer-reviewed by leading cryptography experts worldwide. Unmatched in the industry, MongoDB is the only data platform that allows customers to run expressive queries directly on non-deterministically encrypted data. This represents a groundbreaking advantage for customers, allowing them to maintain robust protection for their sensitive data without sacrificing operational efficiency or developer productivity by still enabling expressive queries to be performed on it. Organizations of all sizes, across all industries, can benefit from the impactful outcomes enabled by Queryable Encryption, such as: Stronger data protection: Data stays encrypted at every stage—whether in-transit, at-rest, or in-use—reducing the risk of sensitive data exposure or breaches. Enhanced regulatory compliance: Provides customers with the necessary tools to comply with data protection regulations like GDPR, CCPA, and HIPAA by ensuring robust encryption at every stage. Streamlined operations: Simplifies the encryption process without needing costly custom solutions, specialized cryptography teams, or complex third-party tools. Solidified separation of duties: Supports stricter access controls, where MongoDB and even a customer's database administrators (DBAs) don’t have access to sensitive data. Use cases for Queryable Encryption MongoDB Queryable Encryption has many use cases for organizations that host sensitive data, regardless of their size or industry. The recent addition of range query support to Queryable Encryption broadens those use cases even wider. Here are some examples to help illustrate how Queryable Encryption could be used to protect and query sensitive data: Financial Services Credit Scoring: Assess creditworthiness by querying encrypted data such as credit scores and income levels. For example, segment your customers based on credit scores between 600 and 750. Fraud Detection: Detect anomalies by querying encrypted transaction amounts for values that exceed typical spending patterns, such as transactions above $10,000. Insurance Risk Assessment: Personalize policy offerings by querying encrypted client data for risk levels within specified ranges, enhancing customer service without exposing sensitive information. Claims Processing: Automate claims processing by querying encrypted claims data for amounts within specific ranges or for claims within time periods, streamlining operations while safeguarding information. Healthcare Medical Research: Execute range-based searches on encrypted medical records, such as querying encrypted datasets for patients within specific age ranges or for abnormal lab results for medical research. Billing and Insurance Processing: Perform secure range queries on encrypted billing data to process insurance claims and payments while protecting patient financial details. Education Grading Systems: Process encrypted student scores to award grades within specific ranges, ensuring compliance with FERPA while protecting student privacy and maintaining data security. Financial Aid Distribution: Analyze encrypted income data within certain ranges to determine eligibility for scholarships and financial aid. Comprehensive data protection at every stage With Queryable Encryption, MongoDB offers unmatched protection for sensitive data throughout its entire lifecycle—whether in-transit, at-rest, or in-use. Now, with the addition of range query support, Queryable Encryption meets even more of the demands of modern applications, unlocking new use cases. To get started, explore the Queryable Encryption documentation .

October 16, 2024
Next →

Leveraging BigQuery JSON for Optimized MongoDB Dataflow Pipelines

We're delighted to introduce a major enhancement to our Google Cloud Dataflow templates for MongoDB Atlas. By enabling direct support for JSON data types, users can now seamlessly integrate their MongoDB Atlas data into BigQuery, eliminating the need for complex data transformations. This streamlined approach not only saves users time and resources, but it also empowers customers to unlock the full potential of their data through advanced data analytics and machine learning. Figure 1: JSON feature for user options on Dataflow Templates Limitations without JSON support Traditionally, Dataflow pipelines designed to handle MongoDB Atlas data often necessitate the transformation of data into JSON strings or flattening complex structures to a single level of nesting before loading into BigQuery. Although this approach is viable, it can result in several drawbacks: Increased latency: The multiple data conversions required can lead to increased latency and can significantly slow down the overall pipeline execution time. Higher operational costs: The extra data transformations and storage requirements associated with this approach can lead to increased operational costs. Reduced query performance: Flattening complex document structures in JSON String format can impact query performance and make it difficult to analyze nested data. So, what’s new? BigQuery's Native JSON format addresses these challenges by enabling users to directly load nested JSON data from MongoDB Atlas into BigQuery without any intermediate conversions. This approach offers numerous benefits: Reduced operating costs: By eliminating the need for additional data transformations, users can significantly reduce operational expenses, including those associated with infrastructure, storage, and compute resources. Enhanced query performance: BigQuery's optimized storage and query engine is designed to efficiently process data in Native JSON format, resulting in significantly faster query execution times and improved overall query performance. Improved data flexibility: users can easily query and analyze complex data structures, including nested and hierarchical data, without the need for time-consuming and error-prone flattening or normalization processes. A significant advantage of this pipeline lies in its ability to directly leverage BigQuery's powerful JSON functions on the MongoDB data loaded into BigQuery. This eliminates the need for a complex and time-consuming data transformation process. The JSON data within BigQuery can be queried and analyzed using standard BQML queries. Whether you prefer a streamlined cloud-based approach or a hands-on, customizable solution, the Dataflow pipeline can be deployed either through the Google Cloud console or by running the code from the github repository . Enabling data-driven decision-making To summarize, Google’s Dataflow template provides a flexible solution for transferring data from MongoDB to BigQuery. It can process entire collections or capture incremental changes using MongoDB's Change Stream functionality. The pipeline's output format can be customized to suit your specific needs. Whether you prefer a raw JSON representation or a flattened schema with individual fields, you can easily configure it through the userOption parameter. Additionally, data transformation can be performed during template execution using User-Defined Functions (UDFs). By adopting BigQuery Native JSON format in your Dataflow pipelines, you can significantly enhance the efficiency, performance, and cost-effectiveness of your data processing workflows. This powerful combination empowers you to extract valuable insights from your data and make data-driven decisions. Follow the Google Documentation to learn how to set up the Dataflow templates for MongoDB Atlas and BigQuery. Get started with MongoDB Atlas on Google Marketplace . Learn more about MongoDB Atlas on Google Cloud on our product page .

December 17, 2024