Pathfinder Labs Tames Data Chaos and Unleashes AI with MongoDB

Nick Bell
September 25, 2024 | Updated: October 14, 2024
#genAI

Pathfinder Labs develops software that specializes in empowering law enforcement agencies and investigators to apprehend criminals and rescue victims of child abuse.

The New Zealand-headquartered company is staffed by professionals with diverse backgrounds and expertise, including counter-terrorism, online child abuse investigations, industrial espionage, digital forensics and more, spanning both the government and private sectors.

Last July, I was thrilled to welcome Pathfinder Labs’ CEO Bree Atkinson, as well as co-founder and DevOps Architect, Peter Pilley to MongoDB .local Sydney where they shared more about the company’s innovative solutions powered by MongoDB.

Those solutions are deployed and utilized by prestigious organizations on a global scale, including Interpol.

Pathfinder Labs’ main product, Paradigm, has been built on MongoDB Atlas and runs on AWS. The tool—which relies on MongoDB’s developer data platform and document database model to sift through complex and continually growing numbers of data sets—helps collect, gather, and convert data into actionable decisions for law enforcement professionals.

Pilley explained that Paradigm was “made by investigators, for investigators.”

Paradigm is designed to present the information it helps gather in a way that will support a successful prosecution and outcome at trial.

MongoDB Atlas enables Pathfinder Labs to tame the chaos arising from the data sets created and gathered throughout an investigation. MongoDB’s scalability and automation capabilities are particularly helpful in this regard.

Powered by MongoDB Atlas, Paradigm can also easily identify similarities between cases, and uncover unique insights by bringing together information from disparate data sources. This could, for example, be about bringing together geolocalization data and metadata from an image, or identifying similar case patterns from law enforcement agencies operating in different states or countries.

Ultimately, Paradigm simplifies evidence gathering and analysis, integrates external data sources and vendors, future-proof investigation methods, and helps minimize overall costs. Its capabilities are unlocking a whole new generation of data-driven investigative capabilities.

During the presentation, Pilley used the example of the case of a missing female in the United States: it took a team of three investigators 12 months to solve the case. Using Paradigm, PathfinderLabs was able to solve that same case in less than an hour.

“With Paradigm, we were able to feed some extra information and solve the case in 40 minutes. MongoDB Atlas allowed us to make quick decisions and present information to investigators in the most efficient way.”

Pathfinder Labs also incorporates AI capabilities, including MongoDB Vector Search, which help identify which information is particularly relevant, select specific data points that can be used at a strategic point in time, connect data from one case to another, and identify what information might be missing.

MongoDB Atlas Vector Search helps Pathfinder match images and details in images (i.e. people, objects), classify documents and text, and to build better search experiences for users via semantic search.

“I was super excited when [Atlas Vector Search] came out. The fact that I can now have it as part of my standard workflow without having to deploy other kits all the time to support our vector searches has been an absolute game changer,” added Pilley.

Finally, the team has seen great value in MongoDB’s Performance Adviser and Schema Anti Patterns features: “The performance Adviser alone has solved many problems,” concluded Pilley.

To learn more and get started with MongoDB Vector Search, visit our Vector Search Quick Start page.

← Previous

Revolutionizing Sales with AI: Glyphic AI’s Journey with MongoDB

When connecting with customers, sales teams often struggle to understand and address the unique needs and preferences of each prospect, leading to ineffective pitches. Additionally, time-consuming admin tasks like data entry, sales tool updates, follow-up management, and maintaining personalized interactions across numerous leads can overwhelm teams, leaving less time for impactful selling. Glyphic AI, a pioneering AI-powered sales co-pilot, addresses these challenges. By analyzing sales processes and calls, Glyphic AI helps teams streamline workflows and focus on building stronger customer relationships. Founded by former engineers from Google DeepMind and Apple, Glyphic AI leverages expertise in large language models (LLMs) to work with private and dynamic data. "As LLM researchers, we discovered the true potential of these models lies in the sales domain, generating vast numbers of calls rich with untapped insights. Traditionally, these valuable insights were lost in digital archives, as extracting them required manually reviewing calls and making notes," says Devang Agrawal, co-Founder and Chief Technology Officer of Glyphic AI. “Our aim became to enhance customer centricity by harnessing AI to capture and utilize conversational and historical data, transforming it into actionable intelligence for ongoing and future deals.” Built on MongoDB, AWS, and Anthropic, Glyphic AI automatically breaks down sales calls using established methodologies like MEDDIC. It leverages ingested sales playbooks to provide tailored strategies for different customer personas and company types. By using data sources such as Crunchbase, LinkedIn, and internal CRM information, the tool proactively surfaces relevant insights before sales teams engage with customers. Glyphic AI employs LLMs to offer complete visibility into sales deals by understanding the full context and intent of real-time conversations. The system captures information at various points, primarily focusing on sales calls and recordings. These data are analyzed by LLMs tailored for sales tasks, summarizing content based on sales frameworks and extracting specific information requested by teams. MongoDB records serve as the main database for customer records, sales call data, and related metadata, while large video files are stored in AWS S3. MongoDB Atlas Search and Vector Search features are integrated, providing the ability to index and query high-dimensional vectors efficiently. Glyphic AI’s Global Search feature uses Atlas Vector Search to allow users to ask strategic questions and retrieve data from numerous sales calls. It matches queries with vector embeddings in MongoDB, utilizing metadata, account details, and external sources like LinkedIn and Crunchbase to identify relevant content. This content is processed by the LLM model for detailed conversational responses. Additionally, MongoDB's Atlas Vector Search continuously updates records, building a dynamic knowledge base that provides quick insights and proactively generates summaries enriched with data from various sources, assisting with sales calls and customer analysis. Figure 1: How Glyphic AI transforms sales call analysis Why Glyphic AI relies on advanced cloud solutions for efficient data management and innovation "I used MongoDB in the first app I ever built, and ever since it has consistently met our needs, no matter the project," says Agrawal. For Glyphic AI, MongoDB has seamlessly integrated into the company’s existing workflows. MongoDB Atlas has greatly simplified database management and analytics, initially involving the team implementing vector search from scratch. When MongoDB introduced Atlas Vector Search, Glyphic AI transitioned to this more streamlined and integrated solution. “If MongoDB's Atlas Vector Search had been available back then, we would have adopted it immediately for its ease of testing and deployment,” Agrawal reflects. While Agrawal appreciates the benefits of building from scratch, he acknowledges that maintaining complex systems, like databases or developing LLM models, becomes increasingly challenging over time. The AI feature enabling natural language queries in MongoDB Compass has been particularly beneficial for Glyphic AI, especially when extracting insights not yet available in dashboards or analyzing specific database elements. In the fast-paced AI industry, time to market is critical. MongoDB Atlas, as a cloud solution, offers Glyphic AI the flexibility and scalability needed to quickly test, deploy, and refine its applications. The integration of MongoDB Atlas with features like Atlas Vector Search has enabled the team to focus on innovation without being bogged down by infrastructure complexities, speeding up the development of AI-powered features. As a small, agile team, Glyphic AI leverages MongoDB's document model, which aligns well with object-oriented programming principles. This allows for rapid development and iteration of product features, enabling the company to stay competitive in the evolving generative AI market. By simplifying data management and reducing friction, MongoDB’s document model helps Glyphic AI maintain agility and focus on delivering impactful solutions. With vector search embedded in MongoDB, the team found relief in using a unified language and system. Keeping all data—including production records and vectors—in one place has greatly simplified operations. Before adopting MongoDB, the team struggled with synchronizing data across multiple systems and managing deletions to avoid inconsistencies. MongoDB’s ACID compliance has made this process far more straightforward, ensuring reliable transactions and maintaining data integrity. By consolidating production records and vectors into MongoDB, the team achieved the simplicity they needed, eliminating the complexities of managing disparate systems. Glyphic AI's next step: Refining LLMs for enhanced sales insights and strategic decision-making “Over the next year, our goal is to refine our LLMs specifically for the sales context to deliver more strategic insights. We've built a strong conversational intelligence product that enhances efficiency for frontline sales reps and managers. Now, we're focused on aggregating conversation data to provide strategy teams and CROs with valuable insights into their teams' performance,” says Agrawal. As sales analysis evolves to become more strategic, significant technical challenges will arise, especially when scaling from summarizing a handful of calls to analyzing thousands in search of complex patterns. Current LLMs are often limited in their ability to process large amounts of sales call data, which means ongoing adjustments and improvements will be necessary to keep up with new developments. Additionally, curating effective datasets, including synthetic and openly available sales data, will be a key hurdle in training these models to deliver meaningful insights. By using MongoDB, Glyphic AI will be able to accelerate innovation due to the reduced need for time-consuming maintenance and management of complex systems. This will allow the team to focus on essential tasks like hiring skilled talent, driving innovation, and improving the end-user experience. As a result, Glyphic AI will be able to prioritize core objectives and continue to develop and refine their products effectively. As Glyphic AI fine-tunes its LLMs for the sales context, the team will embrace retrieval-augmented generation (RAG) to push the boundaries of AI-driven insights. Leveraging Atlas Vector Search will enable Glyphic AI to handle large datasets more efficiently, transforming raw data into actionable sales strategies. This will enhance its AI’s ability to understand and predict sales trends with greater precision, setting the stage for a new level of sales intelligence and positioning Glyphic AI at the forefront of AI-driven sales solutions. As part of the MongoDB AI Innovators Program , Glyphic AI’s engineers gain direct access to MongoDB’s product management team, facilitating feedback exchange and receiving the latest updates and best practices. This collaboration allows them to concentrate on developing their LLM models and accelerating application development. Additionally, the provision of MongoDB Atlas credits helps reduce costs associated with experimenting with new features. Get started with your AI-powered apps by registering for MongoDB Atlas and exploring the tutorials in our AI resources center . If you're ready to dive into Atlas Vector Search, head over to the quick-start guide to kick off your journey. Additionally, if your company is interested in being featured in a story like this, we'd love to hear from you. Reach out to us at ai_adopters@mongodb.com .

September 24, 2024

Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024