MongoDB Atlas optimizes national AI exam platform

THE CHALLENGE

Implementing a nationally important platform

Cambium is an Israel-based business that specializes in customized software development projects for clients ranging from start-ups to large retail groups, and even the Israeli government. Its portfolio of achievements is as broad as its customer base, including web development work for global retailers Foot Locker and Laline, and a high-impact Israeli government project relating to the country’s national elections. 

It’s a business that inspires high levels of confidence from its customers, so for Cambium, the right choice of tool is vital. It’s a key reason why MongoDB Atlas is Cambium’s database platform of choice for the large variety of projects it implements.

“We won’t recommend anything without researching it, and it all depends on the solution the client is seeking,” said Lev Buchel, Chief Technology Officer at Cambium. “But MongoDB does so much that impresses us that we regularly immediately choose it as our main solution.”

That was the case when Cambium started work on the Marvad AI project, an AI-enabled initiative that uses MongoDB Atlas as a key part of a solution that aims to highlight cheating and plagiarism in examinations.

“In Israel, every final high school examination paper is scanned and then uploaded to a portal,” explained Buchel. “Marvad AI’s job is to then check each one for duplications, and trigger an alert if a close match is detected between different submissions.”

Testing Marvad AI ahead of going into production had been relatively straightforward, but as the launch date approached and Cambium started working through significantly higher volumes of real-world data, unforeseen challenges emerged.

“It became unbearable: a single search query on a production workload across around 15 million vectors could take up to 55 minutes, and in some cases even led to timeouts” said David Salzer, Founder and Chief Executive Officer at Cambium. “That level of latency is not something you can live with when you are on your way to checking tens of thousands of documents almost in real time.”

Compounding the issue was a large number of false positives. Overall, Marvad AI was performing less well than the non-AI system it was replacing. And with the scheduled launch date just days away, time was running out—fast.     

Cambium knew that the underlying MongoDB Atlas technology that was driving Marvad AI was sound. What it couldn’t figure out was why it was underperforming so severely.

OUR SOLUTION

Scaling beyond conventional limits

“We had a lower success rate than the old system, and going through tens of thousands of papers was so slow. Our client was getting very frustrated and wanted much quicker results,” said Buchel. “The mission-critical urgency of a national launch acted as a powerful catalyst!”

Cambium tried what it saw as the obvious solutions, but adding more servers did very little to help the situation—and in some cases made things worse.

“At this point, we stopped,” said Salzer. “A moment like this, when you think you’ve taken out the heavy cannons and the target doesn’t move a millimeter, is both frustrating and eye-opening.”

“A combination of all those issues triggered us to seek help from MongoDB,” added Buchel. “And after just one or two hours talking to MongoDB Professional Services, we were able to understand how to scale our database, and configure it to get better and faster results.”

High page faults also meant the system had to constantly fetch index data from the disk, a process that is much slower than memory access. Working with Professional Services, Cambium implemented a combination of vertical scaling and memory management.

The issue of slow response times was resolved by optimizing index sizes and increasing memory on the search nodes, while resetting parameters and nodes around MongoDB’s vector search process fixed the concern around false positives. 

“We scaled vertically to larger search nodes with more RAM, moving to two search nodes at the S60 level, each with 64GB of memory,” said Salzer. “Overnight, we jumped from waiting almost an hour for a search response to having one come in three minutes.” 

“The speed improvement was almost instant,” added Buchel. “In the next few days, we improved it further. Other issues that didn’t relate to MongoDB took a week or two, but they were more on the software development side. Overall, it was really very fast.”

OUTCOME

The importance of having friends

Marvad AI is now live in production, and delivering response times that were originally envisaged when it was developed. It’s a highly important result, and one that Cambium attributes to the proactive and positive attitude of MongoDB Professional Services.

Cambium is now far more aware of the processes around optimizing large datasets with MongoDB Atlas. It also has a much greater understanding of the configuration and normalization of vector searches—much of which was achieved in just one relatively short session. 

“What you think is the obvious solution is sometimes just a habit that isn’t necessarily the right answer,” said Salzer. “Working with MongoDB Professional Services proved that there is nothing like experience! Sometimes you need to try something new.”

Equally important is the even greater level of trust that Cambium now has in MongoDB. The business had always valued the personal approach and ease of access to human-delivered support that MongoDB provides, and saving days or even weeks of hard work through one simple interaction has only reinforced that confidence.

“There is nothing like having friends to call on,” added Salzer. “Asking the people who work with this technology daily saved us days of trial and error.” 

It’s also set an exciting precedent that Cambium expects to follow in upcoming projects, and future interactions with MongoDB Professional Services are already in the pipeline. 

“Projects like this have a very big impact in Israel and beyond, and we’re really proud of them,” said Buchel. “The ability to leverage MongoDB’s expertise ensures that Cambium remains at the forefront of AI implementation, providing world-class reliability to our clients.”