From batch to real-time ‘Data In Flight’

THE CHALLENGE

Overcoming 30 years of legacy system complexity

Bpifrance is known for its vision of ‘serving the future’. It fulfils a crucial public-interest mission to assist companies with financing, investment, and business development support, both domestically and internationally. 

Over its 30-year history, including around 15 mergers, Bpifrance has accumulated a highly complex information system. Its current environment includes 700 production applications and over 1,500 production databases, managed by more than 240 agile teams.

“The mergers brought all the redundancy, legacy, and complexity you can imagine into our information system,” explained Christophe Jalady, Architect at Bpifrance. “To give you an idea, we have a single Kafka platform hosting 2,700 topics, and a central API gateway exposing about 300 APIs.”

Historically, Bpifrance relied on a fragile batch data architecture that it now refers to as the “Stone Age of data processing”. The system required extracting nightly data dumps into a massive 400 TB shared database, resulting in delays of up to five days before data could be consumed. The batch job chains were so fragile that if one failed, the entire chain collapsed.

“Real-time processing was impossible due to the architecture,” added Nizar Salhaji, Head of Domain Architecture at Bpifrance. “The operational complexity, with huge batch chains, led to a high risk of failure.”

As part of a wider modernization project, Bpifrance looked to build a data warehouse architecture using an enterprise-wide data model. However, as Mr Salhaji noted, this proved to be a “false good idea”.

“It forced unique business specifics into a single, rigid model,” he added. “That caused downstream data loss and required extensive, expensive maintenance to manage complex pipelines.” 

Centralized teams also remained a massive bottleneck, and because the ingestion process still relied on nightly batch processing, true real-time data access remained impossible.

“It was also extremely expensive,” said Mr Salhaji. “It costs millions just to maintain the architecture due to its complexity.”

OUR SOLUTION

A decentralized, real-time data architecture

Bpifrance needed a new approach. To address its hurdles and eliminate its deep technical debt, the organization transitioned to a target architecture known as ‘Data In Flight’, relying on standard, reversible technologies such as MongoDB Atlas, Java, and Kafka. 

This modern approach operates on three powerful principles: a central Confluent Kafka instance serves as the single channel for data distribution. Producers publish data in real time using their own business models, and consumers ingest and adapt that data locally.

“Producers are responsible for publishing business events using their own models. There is no single enterprise standard,” explained Mr Jalady. “Contracts are modelled differently in financing, investment, and innovation—and that’s okay.”

To augment the ‘Data In Flight’ concept, Bpifrance built a transversal service called ‘Knowledge In Flight’. Data from Kafka flows directly into MongoDB, where all fields are immediately searchable without filtering or transformation, thereby preserving end-to-end immutability. 

A complementary graph database is used alongside MongoDB to store historical versions and relationship links, while Kafka serves as the immutable event log, providing long-term retention.

With a single Kafka platform now hosting 2,700 topics and a central Gateway exposing 300 APIs, Bpifrance has successfully created a highly scalable infrastructure.

OUTCOME

A 360-degree view empowers autonomous teams

The integration of MongoDB and Confluent has established a robust, high-performance platform that perfectly supports strategic use cases, ranging from analytics for better decision-making to real-time transaction processing. 

More importantly, Bpifrance’s agile teams have gained total autonomy to produce and consume data independently.

“Producers know best how to model their own data,” explained Mr Jalady. “Working with MongoDB gives them the autonomy to do that.”

The organization has now achieved a global, detailed 360-degree view of its operations, encompassing 450 million MongoDB documents and 1.5 billion nodes in its graph database. And despite the vast scale of managing 7 million contracts and 3.5 million ratings, the latest data fits easily within a highly efficient 250 GB footprint in MongoDB.

“With MongoDB, all fields are searchable immediately,” Mr Jalady continued. “And no transformation is applied—real business data is ingested exactly as it’s produced.”

This seamless architecture has unlocked powerful operational capabilities. Customer interactions, contracts, and payments can now be instantly queried to produce a comprehensive CRM timeline. Additionally, Bpifrance’s prospecting datasets are updated in real time, allowing users to easily and efficiently query up to 5,000 companies across 50 criteria.

The system’s immutable tracing even natively meets complex regulatory requirements, such as the GDPR ‘right-to-be-forgotten’, by simply deleting a user’s specific encryption key.

“Our recommendation engine is now much more accurate as all data is finally accessible,” Mr Salhaji concluded. “And with our CRM system, one query retrieves every interaction, including contracts, payments, calls, and so on. The integration of Confluent and MongoDB gives us outstanding performance, reliability, and scalability.”