EventJoin us at AWS re:Invent 2024! Learn how to use MongoDB for AI use cases. Learn more >>

Nerve Solutions: 500% performance boost keeps up with financial markets

INDUSTRIES

Financial Services
Computer Software & SaaS

PRODUCT

MongoDB Atlas

USE CASE

Analytics

CUSTOMER SINCE

2020
INTRODUCTION

Managing massive amounts of data in
real time to help clients stay on top of
fast-moving financial markets

Imagine you are a financial advisor or investment broker who has had a tough day. You’ve missed on all of the big bets you’ve made that day and need to figure out a better way to stay on top of the markets.

There is a solution that can help. Nerve Solutions is a fintech and regtech consulting and development firm that specializes in real-time risk management and automated trading products for financial markets. Nerve Solutions wrestles with massive volumes of critical data in real time to help financial services professionals make critical decisions based on the freshest data. It also provides developer tools and APIs allowing brokers to offer tech-enabled products to their end clients.

Whereas some financial sites experience noticeable slowdowns or even crash on high-volume trading days, Nerve Solutions’ n.Prime product can perform more than 900,000 data runs per second while letting customers monitor complex options portfolios in a multi-client, multi-dealer, and multi-broker setup. Its n.Feed product lets authorized brokers consolidate feeds from across an entire market over a single connection to reduce complexity and help them focus on the most important market movements. n.Image is the company’s proprietary low-latency API that helps compute margin requirements. n.Tasc offers plug-and-play APIs for Strategy Creation, Scenario Analysis, and more to help with building an end-to-end Automated Trading and Analytics Platform.

Nerve Solutions Founder Mihir K. Malani has been an RDBMS expert for most of his career, and that experience taught him that an RDBMS was no match for the volume, velocity, and variety of data required to help some of the world’s most important financial services firms stay a step ahead of global financial markets. The relational model was never designed to manage today’s Web and mobile streaming data, because it was conceived and designed decades earlier, before those data sources even existed.

So a few years ago Malani made the move to MongoDB — and improved his company’s database performance, the time it takes to access market data, by 500%.

THE CHALLENGE

How to overcome legacy database limitations

Malani discussed a simple use case that illustrates both the challenges and the far more complex feats of data handling that Nerve Solutions and MongoDB perform every day: modeling stock data, such as the data appearing in a stock analyst’s candlestick chart for Apple (Nasdaq: $AAPL) in Figure 1.

Figure 1: Candlestick chart for AAPL, June-July 2020

Figure 1: Candlestick chart for AAPL, June-July 2020

Each day shown in the candlestick chart above brings together the data points shown in Figure 2: Stock Name, Date and Time, Open price, High and Low prices for the day, Closing price, and Volume of shares traded.
Figure 2: Daily stock data points

Figure 2: Daily stock data points

To design data for stocks in the relational world, users have two options: create a massive table containing data for all stocks, or create a separate table for each stock, as shown in Figure 3.
Figure 3: Two ways to model stock data in a relational database

Figure 3: Two ways to model stock data in a relational database

Another challenge: in the relational world, users have to map features into columns and every new data point needs to be stored in a row. A typical table for financial markets might contain hundreds of thousands of rows. All rows must contain the same number of columns, and if any rows vary from that, you have to divide them out into a separate table. Malani explained, “Embedded data sets? A table inside a table? In the relational world, that’s heresy!” Plus, creating JOINS to tie together all of that data scattered across all of those tables kills performance. “With most RDBMS community editions, as soon as your data starts to grow you start experiencing performance issues. That was one of the major reasons for us to move from RDBMS to MongoDB,” he added.

To obtain the real-time insights based on massive volumes of data, Malani said you must unlearn what you’ve learned over the years dealing with the challenges and limitations of relational databases. In short, he exclaimed: “Think different!”

"Embedded data sets? A table inside a table? In the relational world, that’s heresy! With....RDBMS...as soon as your data starts to grow you start experiencing performance issues. That was one of the major reasons for us to move from RDBMS to MongoDB."

Mihir K. Malani, Founder, Nerve Solutions

THE SOLUTION

Leveraging the power of MongoDB in
de-normalizing and embedding data

Malani discussed how MongoDB architecture contrasts with legacy databases and why that enables Nerve Solutions to provide its customers with the real-time processing of massive amounts of data they need to keep pace with today’s financial markets. “Relational databases require that you normalize data, and reference multiple fragments of the same record across multiple tables. MongoDB has no such restrictions. In MongoDB there are no columns. The different features of your data, in this case stock data, are simply Fields and Keys. Since there are no columns or rows in MongoDB, it follows that there are also no tables.” As shown in Figure 4, Malani said that instead of having to introduce complexity into the equation by modeling stock data across multiple tables, and most likely across multiple data stores — then joining them all together to approximate a single record — in MongoDB, Nerve Solutions can model all data for a given stock in a single document.
Figure 4: The ease and simplicity Nerve Solutions has experienced  by modeling stock data in MongoDB

Figure 4: The ease and simplicity Nerve Solutions has experienced by modeling stock data in MongoDB

As shown in Figure 4, the document contains just two fields, StockName and Data. Nerve Solutions inserts all data points for the stock by embedding them as multiple documents inside the Apple stock document. All documents for all stocks together comprise a Collection in MongoDB Atlas.
THE RESULTS

Quantum leaps with MongoDB: 50x more capacity and 500% faster

On disk it takes significantly more time for random fetches and significantly less time for sequential fetches, and as shown in Figure 5, that isn’t the only significant difference.

Action/FactorRDBMSMongoDB
How it stores dataData, even for a single stock,
is not stored together
All data for the same stock is stored together in a single document, and all data for all stocks is stored together in a collection
What you must do to delete a stock from the databaseDelete multiple records —
which can impact other records
Delete a single document
Average fetch time for a single record across two million records10.3 milliseconds (ms)2.4ms
Data set capacityInability to manage a data set
containing 10 million records
Ability to work with data sets
in excess of 500 million records

Action/Factor

RDBMS
How it stores dataData, even for a single stock,
is not stored together
What you must do to delete a stock from the databaseDelete multiple records —
which can impact other records
Average fetch time for a single record across two million records10.3 milliseconds (ms)
Data set capacityInability to manage a data set
containing 10 million records
MongoDB
How it stores dataAll data for the same stock is stored together in a single document, and all data for all stocks is stored together in a collection
What you must do to delete a stock from the databaseDelete a single document
Average fetch time for a single record across two million records2.4ms
Data set capacityAbility to work with data sets
in excess of 500 million records

Figure 5: Some advantages Nerve Solutions has gained using MongoDB vs RDBMS

By using MongoDB, Nerve Solutions accesses stock data 500% faster than it could with a relational database. That adds up to a quantum leap in performance across its business because it enables Nerve Solutions clients to move faster to capture areas of opportunity and avoid risk in the market. The ability to quickly access and, if necessary, delete individual records also enhances a company’s security & compliance readiness for things such a Right to be Forgotten request under the terms of the GDPR.

“In addition to the speed advantage we've gained, MongoDB has given us the added advantage of being able to increase the size of our data sets exponentially. The speed and retrieval issues of our previous database made it impossible to work with data sets consisting of even 10 million records. Moving to MongoDB has allowed us to work with data sets in excess of 500 million records with no signs of performance issues. This has greatly enhanced the scale and coverage of our products,” said Malani.

"The speed and retrieval issues of our previous database made it impossible to work with data sets consisting of even 10 million records. Moving to MongoDB has allowed us to work with data sets in excess of 500 million records with no signs of performance issues. This has greatly enhanced the scale and coverage of our products."

Mihir K. Malani, Founder, Nerve Solutions

Advice to others who want to reap the benefits of of MongoDB in their business

“For many developers and DBAs who, like me, have been dealing with the issues and constraints of relational databases, the door is open to a better way of working with data. To walk through it you need to avoid the tendency to simply replicate RDBMS design in MongoDB and follow data modeling strategies like the ones discussed here.” Malani added, “To obtain the drastic performance improvements you seek, think ‘Embed’ instead of ‘Reference.’ Spend some time with some of the great learning resources MongoDB offers. And most importantly: keep an open mind!”

Moving forward, Nerve Solutions plans to explore using MongoDB in its products that require complex computations on large scale in-memory datasets, said Malani. “We also intend to move remaining areas of our infrastructure from RDBMS to MongoDB, which we feel will help us greatly in working with one common data fabric for all our products. This will help us reduce the number of moving parts in our solutions, thereby making the product on-boarding process much easier for our clients.”

What will your story be?

MongoDB will help you find the best solution.