Leaf in the Wild: Haymarket Migrates from MySQL to MongoDB, Achieving 8x Higher Platform Efficiency

Mat Keep
October 27, 2015 | Updated: December 8, 2016
#Customer Stories

Leaf in the Wild posts highlight real world MongoDB deployments. Read other stories about how companies are using MongoDB for their mission-critical projects.

Classified Ads and New Analytics Hub Deployed to MongoDB for Faster Time to Market, Lower Costs and Embrace of Cloud

The media industry is undergoing a fundamental transformation as the move from print to web and mobile accelerates. I met with Peter Dignan, Platform Director at Haymarket Media Group to discuss how the use of modern database platforms enables media companies to embrace new channels and engage with a global audience.

Please start by telling us a little bit about your company. Haymarket Media Group is one of the world’s largest privately owned media companies, headquartered in London. We have over 70 brands delivered over print, web, events and mobile channels targeting the consumer, business, and professional sectors. Our properties are pretty diverse, including Asian Investor, Autocar, Autosport, Brand Republic, Campaign, Management Today, Marketing, Media Week, PistonHeads, The Clothes Show, and What Car.

I head the platform team at PistonHeads.com, the UK’s biggest online motoring community.

Please describe how you use MongoDB. We have multiple projects within PistonHeads.com. We started out with MongoDB powering our classified advertising product catalog back in 2011. We then deployed MongoDB as the backend database to our Single Sign On (SSO) systems that enable secure access to the site.

Our latest MongoDB project is PistonHub, a dedicated portal for dealers who advertise on our site. MongoDB stores all user engagements with each advert – tracking how many views the ad has received, emails, calls and responses the ad has generated – blending it with data from Google Analytics. MongoDB presents a consolidated, single view for the dealers so they can understand which ads and campaigns are performing the best. JavaScript routines process the raw data, and AngularJS has been used to create the UI (User Interface) and data visualizations. PistonHub not only helps dealers optimize their campaigns, it also helps our account management teams upsell premium advertising offerings.

PistonHub: Providing new analytics insight to users, powered by MongoDB

Beyond PistonHeads.com, many other Haymarket divisions use MongoDB today:

  • Autosport has migrated from MySQL to MongoDB to handle traffic spikes around deadline times for its Grand Prix Predictor game. It is also using the ability to scale MongoDB quickly to handle mass results processing, which in turn has driven a better user experience, more clicks, and higher revenue.
  • Our business to business properties are using MongoDB to store page layouts, working with Elasticsearch to assemble dynamic content.
  • Our medical journals track advert views, and run aggregation queries to generate reports. This allow us to provide more reliable statistics for our advertisers, and in turn increase our CPM (cost per thousand impressions).

What were you using before MongoDB? We built our first classified ads system with ASP.NET web framework, backed by MySQL. As our ad volume grew, we started to hit scalability challenges, so we did what many relational database users do in this situation – we added a caching layer (Squid in our case) in an attempt to avoid queries ever having to hit the database. It was a sticking plaster approach. It temporarily brought us more headroom, but as traffic continued to grow we were soon hitting the limits again.

By 2011, we decided to explore our options by evaluating multiple NoSQL databases as part of a replatforming exercise. Not only did we want to increase scalability, we also wanted to personalize the user experience, and so we needed something much more sophisticated than a simple cache. During our evaluations, we found the memory mapped storage engine design of MongoDB gave us much higher performance, and when we coupled it with the Solr search engine, we were able to build a highly functional “pseudo”-caching layer. MySQL remained in place as the system of record.

We provisioned the infrastructure on Amazon Web Services and began load testing. And that gave us a nasty shock when we discovered we would need a minimum of 24 web servers to handle our expected load. MySQL was proving the bottleneck due to slow connections with the web tier. As a result, we postponed the launch and went back to the design, moving more ad content into MongoDB, including page elements, reference data, dropdowns, and static information.

As a result of replacing MySQL with MongoDB, we were able to reduce the size of the web tier from 24 to just 3 servers, significantly reducing our cost and operational complexity, while also boosting performance and reliability.

From that initial 2012 launch, we have since moved all our catalog data into MongoDB. Performance is much higher and we are less I/O bound than when we were using MySQL. We live by the mantra that “if you hit disk, you are dead!”.

Can you describe your MongoDB deployment? Today we run MongoDB across two replica sets with three nodes each, distributed across multiple Amazon Web Services availability zones for disaster resilience. Each replica set is also configured with a hidden member, against which we perform backups. Each MongoDB node is provisioned onto Linux-based AWS R3 xlarge instances with SSDs. Our applications connect to MongoDB via the .NET driver.

MongoDB stores around 650,000 adverts per month, in addition to associated engagement statistics. So far this year, that totals 50 million individual documents.

We use AWS CloudFormation to build our environments, and New Relic for monitoring.

What best practices can you share with the MongoDB community? Firstly, profile your queries to understand access patterns to the database, and design your data model from there. MongoDB offers a rich array of secondary indexes, which are incredibly powerful in supporting complex queries. But it isn’t magic – you need to invest the time to understand your queries and optimize your schema.

Secondly, use SSDs. We’ve had major performance gains from moving up to SSD-equipped instances.

Thirdly, go along to your local MongoDB User Group (MUG), especially as you are getting started with MongoDB. There is a wealth of expertise on hand, from both the organizers and the attendees. This really helped to accelerate our project in the early days.

Consistent low latency from MongoDB drives responsive user experience

How are you measuring the impact of MongoDB on your business? Across multiple dimensions:

  • Time to market. We love the dynamic schema and JSON document model. It enables rapid development and feature evolution. We also love the performance we get out of the box with MongoDB. We’ve actually been able to reduce the amount of performance testing we do as we build our apps, which gives us more time to implement new features demanded by the business.
  • Cost savings. Consolidating our web tier as a result of migrating from MySQL to MongoDB has driven significant infrastructure simplification.
  • Embracing new platforms. As we have replatformed more of our infrastructure to the cloud, MongoDB has made the transition much simpler. It’s a better architectural fit to cloud platforms than relational databases ever were.

Do you have plans to use MongoDB for other applications?

MongoDB is now our default database. We wouldn’t think of using anything else.

What excites you about the MongoDB roadmap? MongoDB 3.2 looks pretty interesting. We are especially looking forward to the BI Connector demonstrated at MongoDB World. The business regularly requests custom, ad-hoc reports, so being able to connect a visualization tool to MongoDB and run standard SQL queries against our collections will help us to develop new reports faster, and with less development effort.

Peter, thanks for sharing your experiences with the MongoDB community.

If you are hitting the scalability limits of a relational database, read our RDBMS to MongoDB migration guide.
Read our RDBMS to MongoDB migration guide

About the Author - Mat Keep

Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

Document Validation - Part 1: Adding Just the Right Amount of Control Over Your Documents

This is the first in a two post series looking at Document Validation, a new feature in MongoDB 3.2. This post introduces the feature together with its benefits while the second will step through a tutorial on how to introduce validation to an existing, live MongoDB deployment.

October 27, 2015
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024