Powerledger powers the future of renewable energy on MongoDB Atlas, 20 billion records at a time

INTRODUCTION

Powering the future of renewable energy with a unique blockchain-enabled platform

Founded in 2016, Australian-born Powerledger develops software solutions for the tracking, tracing and trading of renewable energy. Its unique blockchain-enabled energy trading platform aims to make energy markets more efficient. The business helps people trade energy peer-to-peer, trade environmental commodities, and provide flexibility services to the grid.

Powerledger’s modular platform includes a range of products and services across three main pillars:

• Energy trading and traceability, with products such as xGrid, uGrid and Vision which help track, trace and trade renewable energy so users can stay on top of the provenance and flow of their energy. This gives consumers choice over the energy they’re using and facilitates peer-to-peer trading.
• Flexible trading, with products including MODE Flex and MODE LEM which provide a marketplace that enables Distributed Energy Resource (DER) owners to provide flexibility services to the grid and monetise their assets.
• Environmental commodities trading with TraceX, a digital marketplace to trade environmental commodities such as renewable energy certificates in a more transparent, secure, and efficient way.

With more utilities, renewable energy generators, energy retailers and large corporate organisations with renewable energy targets entering its marketplace, Powerledger has experienced significant growth in the past few years.

It is aiming to have 100 million smart meters on its platform in the next five years, and in the years ahead Powerledger hopes to reach 1 billion users and create markets that remove any obstacles in achieving 24/7 carbon-free energy.

THE CHALLENGE

New thinking required to reach 1 billion users

In the past six years, the company and its customers have collected a massive amount of data, with the platform ingesting about 20 billion records every day.

As the company formulated its plan to on-board 100 million smart meters onto its platform in the next five years, it recognised the need to strengthen its existing tech foundations, and switch to a cloud database service. This would complement its existing software and blockchain technology and would support its ambitious goal.

“We needed to demonstrate our platform’s ability to ingest a much higher volume of data, and cater to the 1 billion users we aim to serve in the future. This meant a complete shift and re-evaluation of the data architecture that is servicing our different products. We needed a level of scalability and flexibility that our previous relational database couldn’t offer”, explained Francesco Pagetti, Solutions Architect at Powerledger.

As a provider of essential services and critical infrastructure, Powerledger cannot afford to have its platform down, even for a few minutes. Furthermore, the company operates in a highly regulated energy markets, which means that the billions of data sets it deals with, need to be secure and meet strict compliance requirements.

Whatever the company did, it was crucial that customers and users had complete trust in its systems. Moving to a new database was a great opportunity but also a daunting challenge. As a first step, Powerledger decided to move to the Google Cloud Platform (GCP) for its cloud infrastructure.

Pagetti said: “Our main priorities - and challenges - were to do a complete data platform migration, as well as add in scalability and flexibility without disrupting the platform or hinder any data security. We also needed to empower our development and engineering teams to be more productive and simplify things, not add more complexity to our architecture.”

THE SOLUTION

Migrating 6 years of data and 70 microservices to MongoDB Atlas

In 2021, Powerledger took on a comprehensive review process and built a Proof of Concept that matched the massive scale the platform needed to handle in the future. The team tried several databases , prioritising testing for scalability, security and ease of use for developers and engineers.

Load tests were conducted on MongoDB Atlas and other databases with MongoDB Atlas convincingly outperforming others in all categories.

Specifically, a load test for 100 million meters was conducted. Totalling 90 nodes on GCP, Atlas demonstrated capability for immense scale and the ability to support Powerledger’s ambitious growth goals.

The migration to MongoDB Atlas started shortly after, in close collaboration with the GCP team as well as experts from Confluent who helped make the transition smooth. Powerledger also took advantage of Confluent's managed data streaming layer to ingest meter data, then using that event data for separate services within the platform.

MongoDB Atlas is now supporting the microservices of three of Powerledger’s leading products: xGrid, TraceX and MODE LEM. Powerledger also deployed additional features including Backup, Monitoring and Real Time Monitoring.

The migration started with selected services to MongoDB Atlas. With the assistance of MongoDB’s Relational Migrator tool, Powerledger is continuing to migrate even more of its database and microservices to Atlas.

“Migrating an entire database is a pretty bold and risky endeavour. Amazingly, we didn’t experience any disruption or downtime. The Atlas tools allowed our development and engineering teams to do upgrading, downgrading, testing, changes of configuration and more without worrying about disrupting the service”, added Pagetti.

After the big load testing in December, the first microservice went live with MongoDB in January 2022, followed by the meter readings in February. All 70 microservices are expected to be live on MongoDB Atlas by August 2023.

THE RESULTS

Transformed scale and performance: Nine hours to four minutes

MongoDB Atlas demonstrated capability for immense scale, but Powerledger was also amazed by the cost-effectiveness of the platform as well as its flexibility and reliability.

The team has recorded a processing capacity of 1 million records in seven minutes, which Powerledger had never experienced before. Most recently the team was able to upload data from 25,000 different meters in just four minutes, which previously would have taken eight to nine hours.

Powerledger also found MongoDB’s data compression capabilities impressive compared to competitors. With its previous database, Powerledger wasn’t able to scale up and down as it wished, which would have been a major impediment to the business and the velocity of the team in the long run. Now with Atlas, Powerledger can easily expand and reduce capacity in response to demand.

Pagetti added: “The scalability and flexibility of MongoDB’s Data Platform are important because it means Powerledger can operate efficiently in a very volatile and fast-changing market.”

The development and engineering teams’ productivity has also improved, with the new data platform enabling them to develop new features faster than they would have been able to previously.

Powerledger has also seen the extra value of the MongoDB platform through features such as Atlas Search , field level encryption, embedded security, and more. As data is a core part of Powerledger’s business, the company is also now further investigating data analytics and machine learning capabilities.

MongoDB has been instrumental in helping Powerledger build strong foundations for its growth. It has helped future-proof the business and empowers the company to keep adapting to new opportunities.

“When we set this goal to onboard 100 million new smart meters in five years, this raised a lot of questions in terms of how we would be able to support this using the right cloud database service. Now that we have switched to MongoDB Atlas and Google Cloud we really feel confident that the technology will be an enabler and not a roadblock”, concluded Pagetti.