Clear: Enabling Seamless Tax Management for Millions of People with MongoDB Atlas

MongoDB
March 6, 2023 | Updated: March 9, 2023
#Finance

Building India's largest tax and financial services software platform trusted by more than six million Indians

With India’s large population and growing middle class, the country’s tax-paying population has been rising steadily. At the end of the financial year 2021-22, about 5.83 crore (58.3 million ) individuals filed tax returns with the Indian Income Tax department. In addition, India also has about 13.8 million registered Goods and Services Tax (GST) taxpayers.

When juxtaposed with growing digitization in India, this opens up massive demand for a convenient and effective platform to manage tax returns. Clear realized this need early on and launched as a SaaS offering for ITR filing to individuals in 2011 that is currently trusted by more than six million Indians. It is second only to the Indian IT Department’s portal in terms of registered users.

More recently, Clear has been focused on expanding its B2B portfolio, including launching an e-invoicing system. Today, the system supports about 50,000 tax professionals, one million small businesses, and 4000 enterprises in GST filing.

How to ensure a seamless experience for all users at scale

Clear built the initial version of its B2B e-invoicing system on MySQL. However, as adoption grew, the team started to see the limits of the systems tested.

Certain batches of invoices were taking upwards of 25 minutes to process, an issue for the time-sensitive nature of tax filing. If any Clear customer failed to file in time, that customer could be given a penalty and labeled as non-compliant by the Indian government. The team knew they needed to take a step back and reevaluate the core structure of their system.

The Clear team started the system rework by outlining a set of required capabilities. The new database system would need to be able to scale up quickly to handle periods of peak demand and down when traffic was low to save on costs. Tax professionals need to be able to see multiple cuts of the data at different levels, so the database would need to be able to support quick and complex aggregations. Lastly, the team knew that didn’t want to be accountable for the management of the system themselves. They needed a fully-managed option.

MongoDB Atlas chosen for best in class scale and performance

The company ran a proof of concept (POC) study comparing MySQL’s performance with other competitive offerings, including MongoDB. It found that, in terms of the time taken to execute different batch sizes of data, MongoDB was considerably faster in all instances. For example, MongoDB’s processing time was 122% faster than the closest competitor and 767% faster than the farthest competitor.

Comparison of performance among databases

Given the document-based nature of invoices, the results of the POC made sense. With MongoDB, the Clear team could store invoice data together instead of splitting it across tables. This minimized the number of costly joins required to obtain data, leading to faster reads. MongoDB also allowed the team to easily split reads and writes in use cases where the system experienced high volumes of reads and where reading slightly stale data was permissible.

Clear’s aggregation needs were also easily met with MongoDB’s aggregation pipeline. The combination of aggregation support and MongoDB’s full-text search capabilities meant that the Clear team could easily build filterable and searchable dashboards on top of their invoice data.

Lastly, the team also loved the easy-to-use nature of MongoDB Atlas, MongoDB’s fully-managed developer data platform. With Atlas, the team could easily scale up and down their clusters on a schedule to match fluctuations in user traffic.

Achieving a 2900% jump in processing speed along with cost savings

After Clear replatformed from MySQL to MongoDB Atlas on AWS, their customers were shocked by the improvement. Pranesh Vittal, Director Of Engineering, ClearTax India said, “We have achieved considerable optimization with MongoDB. Our customers are often surprised by the pace of execution. There is a significant improvement in the performance, with as much as a 2900% jump in processing speed in some instances.”

Comparing the performance of the new MongoDB powered platform

On top of increased speeds, the team is also saving money. “We’ve generated over 20 crore invoices to date running on a single sharded cluster with a 4TB disc,” said Pranesh Vittal. “The ability to store older data in cold storage [with Online Archive] helped us achieve this.”

Atlas Triggers also help the team automatically scale down their clusters each night and scale them up in the morning. The triggers are fully-managed and schedule-based, so it’s as easy as setting them up and letting them run. This automatic right-sizing is saving the team upwards of $7000 each month ($700 per cluster for 10 clusters).

After seeing such positive results, the team has since decided to replatform multiple other products onto MongoDB. “Here, MongoDB’s live support and consultation have proved very useful,” said Pranesh Vittal. Now, Clear manages 25+ clusters and over 10TB of data on MongoDB Atlas.

← Previous

Build a ML-Powered Underwriting Engine in 20 Minutes with MongoDB and Databricks

The insurance industry is undergoing a significant shift from traditional to near-real-time data-driven models, driven by both strong consumer demand, and the urgent need for companies to process large amounts of data efficiently. Data from sources such as connected vehicles and wearables are utilized to calculate precise and personalized premium prices, while also creating new opportunities for innovative products and services. As insurance companies strive to provide personalized and real-time products, the move towards sophisticated and real-time data-driven underwriting models is inevitable. To process all of this information efficiently, software delivery teams will need to become experts at building and maintaining data processing pipelines. This blog will focus on how you can revolutionize the underwriting process within your organization, by demonstrating how easy it is to create a usage-based insurance model using MongoDB and Databricks. This blog is a companion to the solution demo in our Github repository . In the GitHub repo, you will find detailed step-by-step instructions on how to build the data upload and transformation pipeline leveraging MongoDB Atlas platform features, as well as how to generate, send, and process events to and from Databricks. Let’s get started. Part 1: the Use Case Data Model Part 2: the Data Pipeline Part 3: Automated Decision Support with Databricks Part 1: The use case data model Figure 1: Entity relationship diagram - Usage-based insurance example Imagine being able to offer your customers personalized usage-based premiums that take into account their driving habits and behavior. To do this, you'll need to gather data from connected vehicles, send it to a Machine Learning platform for analysis, and then use the results to create a personalized premium for your customers. You’ll also want to visualize the data to identify trends and gain insights. This unique, tailored approach will give your customers greater control over their insurance costs while helping you to provide more accurate and fair pricing. A basic example data model to support this use case would include customers, the trips they take, the policies they purchase, and the vehicles insured by those policies. This example builds out three MongoDB collections, as well two Materialized Views . The full Hackloade data model which defines all the MongoDB objects within this example can be found here . Part 2: The data pipeline Figure 2: The data pipeline - Usage-based insurance The data processing pipeline component of this example consists of sample data, a daily materialized view, and a monthly materialized view. A sample dataset of IoT vehicle telemetry data represents the motor vehicle trips taken by customers. It’s loaded into the collection named ‘customerTripRaw’ (1) . The dataset can be found here and can be loaded via MongoImport , or other methods. To create a materialized view, a scheduled Trigger executes a function that runs an Aggregation Pipeline. This then generates a daily summary of the raw IoT data, and lands that in a Materialized View collection named ‘customerTripDaily’ (2) . Similarly for a monthly materialized view, a scheduled Trigger executes a function that runs an Aggregation Pipeline that, on a monthly basis, summarizes the information in the ‘customerTripDaily’ collection, and lands that in a Materialized View collection named ‘customerTripMonthly’(3). For more info on these, and other MongoDB Platform Features: MongoDB Materialized Views Building Materialized View on TimeSeries Data MongoDB Scheduled Triggers Cron Expressions Part 3: Automated decisions with Databricks Figure 3: The data pipeline with Databricks - Usage-based insurance The decision-processing component of this example consists of a scheduled trigger and an Atlas Chart. The scheduled trigger collects the necessary data and posts the payload to a Databricks ML Flow API endpoint (the model was previously trained using the MongoDB Spark Connector on Databricks). It then waits for the model to respond with a calculated premium based on the miles driven by a given customer in a month. Then the scheduled trigger updates the ‘customerPolicy’ collection, to append a new monthly premium calculation as a new subdocument within the ‘monthlyPremium’ array. You can then visualize your newly calculated usage-based premiums with an Atlas Chart! In addition to the MongoDB Platform Features listed above, this section utilizes the following: MongoDB Atlas App Services MongoDB Functions MongoDB Charts Go hands on Automated digital underwriting is the future of insurance. In this blog, we introduced how you can build a sample usage-based insurance data model with MongoDB and Databricks. If you want to see how quickly you can build a usage-based insurance model, check out our GitHub repository and dive right in! Learn more about MongoDB and Insurance .

March 6, 2023
Next →

3 Ways MongoDB EA Azure Arc Certification Serves Customers

One reason more than 50,000 customers across industries choose MongoDB is the freedom to run anywhere—across major cloud providers, on-premises in data centers, and in hybrid deployments. This is why MongoDB is always working to meet customers where they are. For example, many customers choose MongoDB Atlas (which is available in more than 115 cloud regions across major cloud providers) for a fully managed experience. Other customers choose MongoDB Enterprise Advanced (EA) to self-manage their database deployments to meet specific on-premises or hybrid requirements. To that end, we’re pleased to announce that MongoDB EA is one of the first certified Microsoft Azure Arc-enabled Kubernetes applications, which provides customers even more choice of where and how they run MongoDB. Customer adoption of Azure Arc has grown by leaps and bounds. This new certification, and the launch of MongoDB EA as an Arc-enabled Kubernetes application on Azure Marketplace , means that more customers will be able to leverage the unparalleled security, availability, durability, and performance of MongoDB across environments with the centralized management of their Kubernetes deployments. We are very excited to have MongoDB available for our customers on the Azure Marketplace. By extending Azure Arc’s management capabilities to your MongoDB deployments, customers gain the benefit of centralized governance, enhanced security, and deeper insights into database performance. Azure Arc makes hybrid database management with MongoDB efficient and consistent. Collaboration between MongoDB and Microsoft represents an opportunity for many of our customers to further accelerate their digital transformation when building enterprise-class solutions with Azure Arc. Christa St Pierre, Partner Group Manager, Azure Edge Devices, Microsoft Here are three ways the launch of MongoDB EA on Azure Marketplace for Arc-enabled Kubernetes applications gives customers greater flexibility. 1. MongoDB EA supports multi-Kubernetes cluster deployments, simplifies management MongoDB Enterprise Advanced seamlessly integrates market-leading MongoDB capabilities along with robust enterprise support and tools for self-managed deployments at any scale. This powerful solution includes advanced automation, comprehensive auditing, strong authentication, reliable backup, and insightful monitoring capabilities, all of which work together to ensure security compliance and operational efficiency for organizations of any size. The relationship between MongoDB and Kubernetes is one of strong synergy. With Kubernetes, MongoDB EA really can run anywhere, such as a single deployment spanning on-premises and more than one public cloud Kubernetes cluster. Customers can use the MongoDB Enterprise Kubernetes Operator, a key component of MongoDB Enterprise Advanced, to simplify the management and automation of self-managed MongoDB deployments in Kubernetes. This includes tasks like creating and updating deployments, managing backups, and integrating with various Kubernetes services. The ability of the MongoDB Enterprise Kubernetes Operator to deploy and manage MongoDB deployments that span multiple Kubernetes clusters significantly enhances resilience, improves disaster recovery, and minimizes latency by allowing data to be co-located closer to where it is needed, ensuring optimal performance and reliability. 2. Azure Arc complements MongoDB EA, providing centralized management While MongoDB Enterprise Advanced is already among a select group of databases capable of operating across multiple Kubernetes clusters , it is now also supported in Azure Arc-enabled Kubernetes environments. Azure Arc enables the standardized management of Kubernetes clusters across various environments—including in Azure, on-premises, and even other clouds—while harnessing the power of Azure services. Azure Arc accomplishes this by extending the Azure control plane to standardize security and governance across a wide range of resources and locations. For instance, organizations can centrally monitor all of the Azure Arc-enabled Kubernetes clusters using Azure Monitor for containers , or they can enforce threat protection at scale using Microsoft Defender for Kubernetes. This centralized control significantly reduces the complexity of managing Kubernetes clusters running anywhere, as customers can oversee all resources and apply consistent security and compliance policies across their hybrid environment. 3. Customers can leverage the resilience of MongoDB EA and the centralized governance of Azure Arc Together, these solutions empower organizations to build robust applications across a wide array of environments, whether on-premises or in multi-cloud settings. The combination of MongoDB Enterprise Advanced and the MongoDB Enterprise Operator simplifies the deployment of MongoDB across Kubernetes clusters, allowing organizations to fully leverage enhanced resilience and geographic distribution that surpasses the capabilities of a single Kubernetes cluster. Azure Arc further enhances this synergy by providing centralized management for all of these Kubernetes clusters, regardless of where they are running; for customers running entirely in the public cloud, we recommend using MongoDB’s fully managed developer data platform, MongoDB Atlas. If you’re interested in learning more, we invite you to explore the Azure Marketplace listing for MongoDB Enterprise Advanced for Arc-enabled Kubernetes applications. Please note that aside from use for evaluation and development purposes, this offering requires the purchase of a MongoDB Enterprise Advanced subscription. For licensing inquiries, we encourage you to reach out to MongoDB at https://www.mongodb.com/contact to secure your license and to begin harnessing the full potential of these powerful solutions.

November 19, 2024