PyMongoArrow Now Generally Available

Shelby Carpenter and Shubham Ranjan
July 5, 2023 | Updated: July 19, 2023

We are pleased to announce that PyMongoArrow, a Python library for data analysis with MongoDB, is now generally available.

PyMongoArrow allows you to efficiently move data in and out of MongoDB into other popular analytics tools in an easy and efficient manner. This library is built on top of PyMongo, MongoDB’s popular Python driver for synchronous programming.

Why we built PyMongoArrow

Today, PyMongoArrow is the recommended way to materialize MongoDB query result sets as contiguous-in-memory, typed arrays suited for in-memory analytical processing applications. It currently supports exporting MongoDB data into Pandas DataFrames, NumPy arrays, and Apache Arrow tables.

Before MongoDB created PyMongoArrow, it was possible to move data out of MongoDB into other analytics tools and systems, but there wasn’t a unified tool for working with the variety of data formats commonly used for analysis. Because different data analysts and developers may have different approaches and use different formats, this could sometimes interrupt collaboration and create a bottleneck in teams’ analytics pipelines.

PyMongoArrow solves these challenges for our users. While PyMongoArrow has been available in Public Preview since 2021, we have now made it generally available after adding additional features to ensure the best user experience.

Why use PyMongoArrow

The PyMongoArrow library can be easily integrated into your already existing analytics pipeline. Since it's built on top of the PyMongo library, it also extends all its functionality to let you work with MongoDB data in an easy and performant manner when operating at scale.

What can PyMongoArrow do?

Read data into Pandas DataFrame, NumPy Array, and Arrow Table Format

You can connect to your MongoDB instance through the PyMongoArrow library and use the following functions to output the query result sets into the desired data format:

find_pandas_all(): lets you output MongoDB query result sets as a Pandas DataFrame
find_arrow_all(): lets you output MongoDB query result sets as an Arrow Table
find_numpy_all(): lets you output MongoDB query result sets as a Numpy Array

Write to other data formats

Not only does PyMongoArrow allow you to output MongoDB query results sets as Pandas DataFrames, as NumPy arrays, and as Arrow tables, but it also allows you to write data to many other data formats. Once the MongoDB query result sets have been loaded as Arrow table type, it can be easily written to one of the other formats supported by PyArrow such as Parquet file, CSV, JSON etc.

Write data back to MongoDB

PyMongoArrow not only enables you to perform analytics tasks efficiently but lets you write the analyzed data back into the MongoDB database, ensuring permanent persistence for your valuable insights.

Result sets that have been loaded as Arrow’s table type, Pandas’ DataFrame type, or NumPy’s array type can be easily written to your MongoDB database using the write() function.

Use MongoDB's powerful aggregation pipeline with PyMongoArrow

In addition to basic find operations, you can also take advantage of MongoDB's powerful aggregation pipeline for even more complex analytical use cases.

Simply use the aggregate_pandas_all() function to query your MongoDB data using an aggregation pipeline and return the result sets as Pandas DataFrames. You can also use the aggregate_numpy_all() function and aggregate_arrow_all() functions to return the result sets as NumPy arrays and Arrow tables.

Get started today

We have plenty of resources available to guide you in your journey to quickly get started with the PyMongoArrow library. Here are some great resources:

Once you’ve given it a try, please share your feedback with us through the MongoDB Feedback Engine. Your feedback helps us understand what features will make the most impact for our users.

Head to the MongoDB.local hub to see where we'll be showing up next.

← Previous

ADS: Edge Server + Data Ingest

>> Announcement: Some features mentioned below will be deprecated on Sep. 30, 2025. Learn more . Maintaining data across an increasingly diverse set of devices – such as mobile phones, kiosks, IoT devices, sensors, and more – is becoming increasingly sophisticated. Requirements for low latency experiences, accurate visibility in real-time, management through hostile network conditions, and compatibility across an expanding set of device types all make this extremely challenging. We are thrilled to announce products that address this expanding challenge: Atlas Device Sync : Edge Server, Data Ingest, and C++ support. These capabilities are all key additions to MongoDB’s developer data platform, empowering teams with an out-of-the-box data synchronization layer that ensures uninterrupted operations and productivity across an organization’s ecosystem of distributed devices. Atlas Device Sync: Edge Server Traditionally, edge devices require cloud connectivity to sync with each other and reflect changes across users. This meant that if there was no internet connection, devices used for critical operations like inventory management or package tracking were not showing accurate data until the internet connection resumed. Many use cases require more reliability across local devices – for example, in a retail warehouse where tablets are used for real-time package management – for which some teams develop and implement their own local syncing solutions at their remote branch locations. Atlas Device Sync: Edge Server enables teams to leverage a pre-built local-first data synchronization layer. They can deploy a local Edge Server at their remote location which allows devices to sync directly with each other without the need for a roundtrip to the cloud. Once back online, the data is also synchronized with the cloud. This approach ensures swift and efficient synchronization, enhancing overall performance and enabling smooth operations even in unreliable network conditions. With MongoDB’s Atlas Device Sync: Edge Server, organizations can cut down the time it takes to build, test, and maintain a local sync solution from scratch, and instead focus on other pressing innovative business initiatives. The advantages of Atlas Device Sync extend beyond convenience and simplicity. Revisiting that retail warehouse example, in situations where the store operates in standalone mode, such as during a network outage caused by a natural disaster, our local Edge Server ensures that the in-store devices can sync with each other, providing a cohesive experience for both customers and employees. The applications of Atlas Device Sync are diverse, catering to a range of industries and scenarios. For example, mobile devices in an airplane can maintain a shared state across the cockpit and flight crew, facilitating efficient communication and collaboration. Cruise ship sales across multiple gift shops can keep a common inventory while at sea by syncing with the local server, thereby ensuring accurate stock management. Even medical records on a Navy ship can be updated during checkups and saved to the local Tiered Device Sync, ready to sync with the full backend once a network connection is established. Atlas Device Sync: Edge Server is now public preview. Sign up to get started. Atlas Device Sync: Data Ingest Data Ingest, now generally available, serves as a synchronization strategy tailored for applications that predominantly involve writing data on the client side, without requiring frequent reads. By enabling Data Ingest for one or more collections, businesses can experience accelerated write speeds while bypassing some of the processing involved in bi-directional sync. This feature supports writing data to any collection type, including Atlas time-series collections, making it suitable for a wide range of use cases. Consider an Internet of Things (IoT) application that continually logs sensor data, generating a significant workload in terms of data writes but with minimal read requirements. This IoT device may also experience prolonged periods of offline operation. With Data Ingest, the processing overhead associated with bi-directional synchronization is circumvented, resulting in significantly improved write speeds to an Atlas collection. This ensures that crucial sensor data is efficiently captured and stored, even under challenging network conditions. Data Ingest is not limited to IoT applications alone; it can be leveraged for various use cases where write operations dominate and conflict resolution is unnecessary. For instance, retail applications that generate invoices or log application events can benefit from the streamlined and accelerated data writing offered by Data Ingest. By eliminating the need for conflict resolution, businesses can optimize their processes, enhance performance, and improve overall operational efficiency. This feature can be selectively applied to individual collections, allowing your application to utilize Data Ingest for specific data sets while utilizing bi-directional Device Sync for other collections. This enables full flexibility, allowing businesses to tailor their synchronization approach based on their unique requirements. With the powerful capabilities of Atlas Device Sync: Edge Server and Data Ingest, our developer data platform enables enterprises to fully embrace the potential of edge computing. By eliminating the overhead of traditional methods of building these sophisticated synchronization mechanisms from scratch and instead leveraging pre-built solutions embodying industry best practices, teams can operate at peak performance levels, even in scenarios with limited connectivity or heavy data generation requirements. Atlas Device Sync: Data Ingest is now generally available. Read the docs and register for Atlas to get started today. C++ Support Lastly, we are proud to announce the beta release of our highly anticipated C++ support through our C++ client SDK . This addition further expands the reach of our synchronization solution, catering to applications running on embedded, lightweight, low-footprint devices, and Windows platforms. Developers can use this SDK to immediately add Atlas Device Sync to their C++ applications, enabling teams to leverage the full potential of edge computing without compromising on performance or ease of development. This announcement also includes improvements in schema definitions and API methods, providing a natural and intuitive experience for native C++ developers. The introduction of C++ Support is a testament to our commitment to providing comprehensive solutions that address the diverse needs of our customers. By expanding our compatibility to include C++ and Windows platforms, we aim to empower developers to create innovative and efficient applications that seamlessly synchronize data in edge environments. The C++ SDK is now in beta. Ready to get started? Use the C++ SDK by installing the SDK . Read our docs , and follow our repo . Then, register for Atlas to connect to Atlas Device Sync, a fully-managed mobile backend as a service. Leverage out-of-the-box infrastructure, data synchronization capabilities, network handling, and much more to quickly launch enterprise-grade mobile apps. Finally, let us know what you think, and get involved in our forums . See you there! Stay tuned for more updates as we continue to enhance our offerings and empower you with cutting-edge solutions.

July 3, 2023

Next →

AI-Powered Call Centers: A New Era of Customer Service

Customer satisfaction is critical for insurance companies. Studies have shown that companies with superior customer experiences consistently outperform their peers. In fact, McKinsey found that life and property/casualty insurers with superior customer experiences saw a significant 20% and 65% increase in Total Shareholder Return , respectively, over five years. A satisfied customer is a loyal customer. They are 80% more likely to renew their policies, directly contributing to sustainable growth. However, one major challenge faced by many insurance companies is the inefficiency of their call centers. Agents often struggle to quickly locate and deliver accurate information to customers, leading to frustration and dissatisfaction. This article explores how Dataworkz and MongoDB can transform call center operations. By converting call recordings into searchable vectors (numerical representations of data points in a multi-dimensional space), businesses can quickly access relevant information and improve customer service. We'll dig into how the integration of Amazon Transcribe, Cohere, and MongoDB Atlas Vector Search—as well as Dataworkz's RAG-as-a-service platform— is achieving this transformation. From call recordings to vectors: A data-driven approach Customer service interactions are goldmines of valuable insights. By analyzing call recordings, we can identify successful resolution strategies and uncover frequently asked questions. In turn, by making this information—which is often buried in audio files— accessible to agents, they can give customers faster and more accurate assistance. However, the vast volume and unstructured nature of these audio files make it challenging to extract actionable information efficiently. To address this challenge, we propose a pipeline that leverages AI and analytics to transform raw audio recordings into vectors as shown in Figure 1: Storage of raw audio files: Past call recordings are stored in their original audio format Processing of the audio files with AI and analytics services (such as Amazon Transcribe Call Analytics ): speech-to-text conversion, summarization of content, and vectorization Storage of vectors and metadata: The generated vectors and associated metadata (e.g., call timestamps, agent information) are stored in an operational data store Figure 1: Customer service call insight extraction and vectorization flow Once the data is stored in vector format within the operational data store, it becomes accessible for real-time applications. This data can be consumed directly through vector search or integrated into a retrieval-augmented generation (RAG) architecture, a technique that combines the capabilities of large language models (LLMs) with external knowledge sources to generate more accurate and informative outputs. Introducing Dataworkz: Simplifying RAG implementation Building RAG pipelines can be cumbersome and time-consuming for developers who must learn yet another stack of technologies. Especially in this initial phase, where companies want to experiment and move fast, it is essential to leverage tools that allow us to abstract complexity and don’t require deep knowledge of each component in order to experiment with and realize the benefits of RAG quickly. Dataworkz offers a powerful and composable RAG-as-a-service platform that streamlines the process of building RAG applications for enterprises. To operationalize RAG effectively, organizations need to master five key capabilities: ETL for LLMs: Dataworkz connects with diverse data sources and formats, transforming the data to make it ready for consumption by generative AI applications. Indexing: The platform breaks down data into smaller chunks and creates embeddings that capture semantics, storing them in a vector database. Retrieval: Dataworkz ensures the retrieval of accurate information in response to user queries, a critical part of the RAG process. Synthesis: The retrieved information is then used to build the context for a foundational model, generating responses grounded in reality. Monitoring: With many moving parts in the RAG system, Dataworkz provides robust monitoring capabilities essential for production use cases. Dataworkz's intuitive point-and-click interface (as seen in Video 1) simplifies RAG implementation, allowing enterprises to quickly operationalize AI applications. The platform offers flexibility and choice in data connectors, embedding models, vector stores, and language models. Additionally, tools like A/B testing ensure the quality and reliability of generated responses. This combination of ease of use, optionality, and quality assurance is a key tenet of Dataworkz's "RAG as a Service" offering. Diving deeper: System architecture and functionalities Now that we’ve looked at the components of the pre-processing pipeline, let’s explore the proposed real-time system architecture in detail. It comprises the following modules and functions (see Figure 2): Amazon Transcribe , which receives the audio coming from the customer’s phone and converts it into text. Cohere ’s embedding model, served through Amazon Bedrock , vectorizes the text coming from Transcribe. MongoDB Atlas Vector Search receives the query vector and returns a document that contains the most semantically similar FAQ in the database. Figure 2: System architecture and modules Here are a couple of FAQs we used for the demo: Q: “Can you explain the different types of coverage available for my home insurance?” A: “Home insurance typically includes coverage for the structure of your home, your personal belongings, liability protection, and additional living expenses in case you need to temporarily relocate. I can provide more detailed information on each type if you'd like.” Q: “What is the process for adding a new driver to my auto insurance policy?" A: “To add a new driver to your auto insurance policy, I'll need some details about the driver, such as their name, date of birth, and driver's license number. We can add them to your policy over the phone, or you can do it through our online portal.” Note that the question is reported just for reference, and it’s not used for retrieval. The actual question is provided by the user through the voice interface and then matched in real-time with the answers in the database using Vector Search. This information is finally presented to the customer service operator in text form (see Fig. 3). The proposed architecture is simple but very powerful, easy to implement, and effective. Moreover, it can serve as a foundation for more advanced use cases that require complex interactions, such as agentic workflows , and iterative and multi-step processes that combine LLMs and hybrid search to complete sophisticated tasks. Figure 3: App interface, displaying what has been asked by the customer (left) and how the information is presented to the customer service operator (right) This solution not only impacts human operator workflows but can also underpin chatbots and voicebots, enabling them to provide more relevant and contextual customer responses. Building a better future for customer service By seamlessly integrating analytical and operational data streams, insurance companies can significantly enhance both operational efficiency and customer satisfaction. Our system empowers businesses to optimize staffing, accelerate inquiry resolution, and deliver superior customer service through data-driven, real-time insights. To embark on your own customer service transformation, explore our GitHub repository and take advantage of the Dataworkz free tier .

November 27, 2024