Orbweaver & MongoDB: scaling with text search to digitize an industry

INTRODUCTION

Making suppliers and buyers more interconnected

Orbweaver focuses on bringing digital transformation solutions to the electronic components industry. Founded in 2012 in Bethlehem, Pennsylvania, the Orbweaver platform provides an end-to-end quoting, sales automation, and data integration solution for the electronic components industry to create a web of interconnectedness for industry participants, distributors, and OEMs of all sizes. The company helps streamline everything from quoting and sales order intake to customer and supplier integration, helping companies win in a competitive and concentrated market.

THE CHALLENGE

Facing scalability headwinds

In an industry that historically relied on sending email attachments back and forth, Orbweaver wanted to offer a truly digital-native solution. Their Parts Search API helps the process of procuring parts from a bill of materials. This process which includes finding those components, and ensuring they are correct against hundreds of others can be a manual and time consuming endeavor. In the electronics industry, part catalogs easily span into the millions with each part having up to 40 different attributes, creating a deluge of product data. Search plays an absolutely critical role in this process, serving as the starting point for manufacturers and buyers to quickly find what they need. Orbweaver’s DataHub platform specializes in the unique challenges brought upon by such quantities of data and building automation solutions to bring new levels of scale and efficiency.

The Orbweaver team responsible for this digital platform includes Wilmer Companioni, Director of Business Development, Jake Reeves, VP of Technical Operations, and Dave Antosh, the company’s Chief Architect. The team originally used several MySQL databases, but eventually switched to the combination of Amazon DocumentDB for their data and Elasticsearch for core search needs.

Dave Antosh recalls that the journey with Elasticsearch got “difficult and expensive” very quickly. There were noticeable problems specifically around imports, which continually caused a spike in errors. Antosh remembers, “It just started to feel like you really needed a whole team focused on a tool like Elasticsearch for it to work.” Jake Reeves experienced the same limitations, noting “We kept scaling Elasticsearch to try to compensate for the performance, but that was extremely expensive.”

With these concerns in mind, the team knew it was time to make a switch.

THE SOLUTION

Simplifying the tech stack with MongoDB Search on Atlas

The Orbweaver team evaluated alternate search solutions that offered an engaging end-user experience and greater scalability, while still keeping costs low. Dave had known about MongoDB for a while, and liked that MongoDB Search on Atlas was Lucene-based. He was also impressed with the product architecture, especially its simplicity, highlighting that “With MongoDB Atlas everything is just in a document and you search on it — it’s a much simpler model.”

The team also gained peace of mind with the integrated security and reduction in maintenance costs associated with switching to MongoDB Search on Atlas, as Jake recalls “We were also concerned about the resilience of Elasticsearch. It would have been very difficult to recover from any kind of issue.” The team set out a migration path, which on the data side meant rerunning past data with business logic from the old system into MongoDB Atlas. On the search side it took about two weeks time (including testing) to move to MongoDB Search on Atlas from Elasticsearch.

THE RESULTS

Scaling up while reducing cloud spend

Orbweaver saw immediate results upon leaving DocumentDB and Elasticsearch and migrating to the Atlas platform. The migration resulted in a cost reduction without any degradation in performance. In fact, Orbweaver was able to provide an improved performance and experience. “The migration resulted in a latency decrease of over 70%”, Jake notes.

Since MongoDB Search on Atlas delivers powerful text and semantic search as a native capability of Atlas, Orbweaver no longer had to run and maintain a separate ETL tool. Orbweaver has developed functionality, supported by MongoDB, to take in a flat file list of supplier parts and very quickly stand up a Parts API for them, saving time and digitizing what were previously more analog processes, The team also saw improvements in overall performance while saving engineering time. “Migrating to MongoDB Atlas has made the API faster and more stable. Overhead with ETL and Elasticsearch affected the performance of the API — this is something we don’t see anymore as Atlas handles both ends of this brilliantly,” Jake shares.

Dave was also thrilled with the productivity gains, recalling: “From a time-saving perspective, changes in Elasticsearch required creating a new index and copying all the data over. With MongoDB Atlas, creating a new index across gigabytes of data takes less than a minute, so the time savings are huge and the time to query is fast.” With the change over to MongoDB Atlas, Orbweaver was able to improve the experience for their customers by providing an API with much lower latency and faster response times. To the end user, the change was transparent but the improvements were felt and use of the APIs continues to grow.

With a successful migration completed, the team is now looking towards further growth, with search playing a major role. When asked to reflect back on their journey and lessons learned to share with those going through a similar transition, Dave gave two pieces of advice: the first is to avoid getting too stuck in any one specific environment if it’s not serving the team well. The second is to “Test early, and test often. MongoDB makes it easy to do with quick iteration time. It’s not a difficult technology to learn, so if you’re thinking about it, just try it.”