Utilizing AI and MongoDB to Power Content Personalization

Tom Wilson: Concured is a software company that leverages artificial intelligence techniques developed in the last five to 10 years. It is designed to help marketers know what to write about on a per-topic basis, to see what is working well from their own content, as well as that of their competitors, and within their industry. Finally, it maximizes the return on investment they make on any content that they create by personalizing the web visitor’s journey through the client’s website.

Concured has developed a recommender system, which is personalized to the individual visitor. It is a privacy friendly approach that doesn’t use third-party cookies or spy on the user, it is purely based on the web visitor’s behavior on the site, and it builds up an interest profile on that visitor as they click through successive web pages. As it builds a more precise picture of the interests and intent of the user, it tries to recommend content to them to read next, whether it’s a blog post, a product description, or any other kind of content.

Built with MongoDB: You mentioned artificial intelligence. How does Concured use artificial intelligence behind the scenes?

Wilson: We use artificial intelligence in a number of places. One of our unique selling propositions to our clients, is that, unlike other personalized systems, Concured doesn’t require a lengthy integration period, nor does it require ongoing maintenance on the part of the domain. Our approach is to scrape the content of the clients’ websites using AI powered bots to find the content that is relevant and to extract the text, the title, and any other relevant metadata, and to index that in an automatic way.

Our system leverages recent advances in natural language processing (NLP) in order to generate semantic metadata for each document, corresponding to a unique point within a multi-dimensional space. Another aspect is to understand the key entities that are in it, and also to understand the relationship of that particular article to the other ones within the same website. We create a lot of metadata automatically on each piece of content that we find using AI powered web crawlers and scrapers.

Built with MongoDB: Since AI isn’t always 100% accurate, what is the accuracy for the NLP that you’ve created at Concured?

Wilson: With recommender systems, it’s very hard to say what the best recommendation is, because it could be very different for the same person depending on the day, or the web session. If you think of some of the most famous recommender systems, such as Netflix, Amazon, or Spotify, they try to show us what we want to see next, but there’s actually no single right answer.

Because of that, it’s very hard to measure the performance, so the approach that we use is not to say that there is a 100% correct answer, but rather, to say, if we make changes to the algorithms, do the visitors on these websites click on more articles, and do they reach the target pages defined by the website’s owner, e.g. a product page or a sign-up form. The higher proportion of people performing that action at the end of the journey on the website relative to the number of the people who arrived on the website, the more successful the recommender system is, and we can compare the success rate that our clients have on their websites before and after they activate the personalization system from Concured. So far we’re looking at a two to three times uplift, and the algorithms are improving all the time.

Built with MongoDB: When did you join the team at Concured?

Wilson: At a time when the company had raised its first serious money from external investors, and one of the requirements was that they brought on a professional CTO, this often happens in early stage companies, the investors want to impose some structure, they want to know that their money is going to be spent wisely, and do things a little less shooting from the hip. And so, in some companies they joke about bringing in adult supervision. I don’t know if I’d say that about my role, since the team was already nicely established, but I was able to provide a lot more structure to ensure that we would meet our next milestones, as well as longer-term strategic planning and a technical vision.

Built with MongoDB: How did your team decide to build with MongoDB?

Wilson: The team that I joined was already using MongoDB at the time. Within a few months of joining, there was some discussion about whether to move to a structured database. That was a decision that had to be made, so that’s where I got involved and it became a conscious decision not to move away from MongoDB. This was the right fit for what we wanted going forward, and we absolutely made the right decision. We are also going to move away from our Community edition hosted on Google Cloud Platform to MongoDB Atlas Serverless. We’ll be happy not to manage machines any more, thanks to the serverless aspect, and we’re also excited to try out the text search features available on Atlas, since this could potentially simplify our tech stack. For where we are, as a business today, and where we want to be in five years, MongoDB is and will continue to be the right choice.

Built with MongoDB: What does the future of Concured look like?

Wilson: The future is being written as we speak. It’s bringing on more clients with similar needs to those of our largest clients at the moment, namely enterprises that have a huge amount of content, already in the archives, that they would like to continue to squeeze value out of, as well as publishing a lot; whether it’s big companies, such as consultancies or financial services industries, or traditional publishers, you want to make sure that they’re promoting the right content that’s going to have the biggest uplift in terms of whatever KPIs they’re measuring.

Built with MongoDB: What is the best piece of feedback that you’ve received?

Wilson: One nice piece of feedback I’ve received from my team is that I’ve always been there for them. If they have a problem, I will either fix it or remove obstacles for them so they can do the best that they can. It’s part of my philosophy that if you take care of the team, a lot of other things will take care of themselves.