Raphael Schor

3 results

Saving Energy, Smarter: MongoDB and Cedalo for Smart Meter Systems

The global energy landscape is undergoing a significant transformation, with energy consumption rising 2.2% in 2023, surpassing the 2010-2019 average of 1.5% per year. This increase is largely due to global developments in BRICS member countries—Brazil, Russia, India, China, and South Africa. As renewable sources like solar power and wind energy become more prevalent (in the EU, renewables accounted for over 50% of the power mix in the first quarter of 2024 ), ensuring a reliable and efficient energy infrastructure is crucial. Smart meters, the cornerstone of intelligent energy networks, play a vital role in this evolution. According to IoT analyst firm Berg Insight, the penetration of smart meters is skyrocketing, with the US and Canada expected to reach nearly 90% adoption by 2027, whereas China is expected to account for as much as 70–80% of smart electricity meter demand across Asia in the next few years. This surge is indicative of a growing trend towards smarter, more sustainable energy solutions. In Central Asian countries, the Asian Development Bank is supporting the fast deployment of smart meters to save energy and improve the financial position of countries' power utilities. This article will delve into the benefits of smart meters, the challenges associated with managing their data, and the innovative solutions offered by MongoDB and Cedalo. The rise of smart meters Smart meters, unlike traditional meters that require manual readings, collect and transmit real-time energy consumption data directly to energy providers. This digital transformation offers numerous benefits, including: Accurate Billing: Smart meters eliminate the need for estimations, ensuring that consumers are billed precisely for the energy they use. Personalized Tariffs: Energy providers can offer tailored tariffs based on individual consumption patterns, allowing consumers to take advantage of off-peak rates, special discounts, and other cost-saving opportunities. Enhanced Grid Management: Smart meter data enables utilities to optimize grid operations, reduce peak demand, and improve overall system efficiency. Energy Efficiency Insights: Consumers can gain valuable insights into their energy usage patterns, identifying areas for improvement and reducing their overall consumption. With the increasing adoption of smart meters worldwide, there is a growing need for effective data management solutions to harness the full potential of this technology. Data challenges in smart meter adoption Despite the numerous benefits, the widespread adoption of smart meters also presents significant data management challenges. To use smart metering, power utility companies need to deploy a core smart metering ecosystem that includes the smart meters themselves, the meter data collection network, the head-end system (HES), and the meter data management system (MDMS). Smart meters collect data from end consumers and transmit it to the data aggregator via the Local Area Network (LAN). The transmission frequency can be adjusted to 15 minutes, 30 minutes, or hourly, depending on data demand requirements. The aggregator retrieves the data and then transmits it to the head-end system. The head-end system analyzes the data and sends it to the MDMS. The initial communications path is two-way, signals or commands can be sent directly to the meters, customer premise, or distribution device. Figure 1: End-to-end data flow for a smart meter management system / advanced metering infrastructure (AMI 2.0) When setting up smart meter infrastructure, power, and utility companies face several significant data-related challenges: Data interoperability: The integration and interoperability of diverse data systems pose a substantial challenge. Smart meters must be seamlessly integrated with existing utility systems and other smart grid components often requiring extensive upgrades and standardization efforts. Data management: The large volume of data generated by smart meters requires advanced data management and analytics capabilities. Utilities must implement robust data storage, processing, and analysis solutions to handle real-time time series data streams storage, analysis for anomaly detection, and trigger decision-making processes. Data privacy: Smart meters collect vast amounts of sensitive information about consumer energy usage patterns, which must be protected against breaches and unauthorized access. Addressing these challenges is crucial for the successful deployment and operation of smart meter infrastructure. MQTT: A cornerstone of smart meter communication MQTT , a lightweight publish-subscribe protocol, shines in smart meter communication beyond the initial connection. It's ideal for resource-constrained devices on low-bandwidth networks, making it perfect for smart meters. While LoRaWAN or PLC handle meter-to-collector links, MQTT bridges Head-End Systems (HES) and Meter Data Management Systems (MDMS). Its efficiency, reliable delivery, and security make it well-suited for large-scale smart meter deployments. Cedalo MQTT platform and MongoDB: A powerful combination Cedalo , established in 2017, is a leading German software provider specializing in MQTT solutions. Their flagship product, the Cedalo MQTT Platform, offers a comprehensive suite of features, including the Pro Mosquitto MQTT broker and Management Center . Designed to meet the demands of large enterprises, the platform delivers high availability, audit trail logging, persistent queueing, role-based access control, SSO integration, advanced security, and enhanced monitoring. To complement the platform's capabilities, MongoDB's Time Series collections provide a robust and optimized solution for storing and analyzing smart meter data. These collections leverage a columnar storage format and compound secondary indexes to ensure efficient data ingestion, reduced disk usage, and rapid query processing. Additionally, window functions enable flexible time-based analysis, making them ideal for IoT and analytical applications. Figure 2: MongoDB as the main database for the meter data management system where it receives meter data via Pro Mosquitto MQTT broker. Let us revisit Figure 1 and leverage both the Cedalo MQTT Platform and MongoDB in our design. In Figure 2, the Head-end System (HES) can use MQTT to filter, aggregate, and convert data before storing it in MongoDB. This data flow can be established using the MongoDB Bridge plugin provided by Cedalo. Since the MQTT payload is JSON, it is ideal to store it in MongoDB as the database stores data in BSON (Binary JSON). The MongoDB Bridge plugin offers advanced features such as flexible data import settings (specifying target databases and collections, choosing authentication methods, and selecting specific topics and message fields to import) and advanced collection mapping (mapping multiple MQTT topics to one or more collections with the ability to choose specific fields for insertion). MongoDB's schema flexibility is crucial for adapting to the ever-changing structures of MQTT payloads. Unlike traditional databases, MongoDB accommodates shifts in data format seamlessly, eliminating the constraints of rigid schema requirements. This helps with interoperability challenges faced by utility companies. Once the data is stored in MongoDB, it can be analyzed for anomalies. Anomalies in smart meter data can be identified based on various criteria, including sudden spikes or drops in voltage, current, power, or other metrics that deviate significantly from normal patterns. Here are some common types of anomalies that we might look for in smart meter data: Sudden spikes or drops: These include voltage, current, or power spikes or drops. A sudden increase or decrease in voltage beyond expected limits. Outliers: Data points that are significantly different from the majority of the data. Unusual patterns: Unusually high or low energy consumption compared to historical data or inconsistent power factor readings. Frequency anomalies: Frequency readings that deviate from the normal range. MongoDB's robust aggregation framework can aid in anomaly detection. Both anomaly data and raw data can be stored in time series collections, which offer reduced storage footprint and improved query performance due to an automatically created clustered index on timestamp and _id. The high compression offered addresses the challenge of data management at scale. Additionally, data tiering capabilities like Atlas Online Archive can be leveraged to push cold data into cost-effective storage. MongoDB also provides built-in security controls for all your data, whether managed in a customer environment or MongoDB Atlas, a fully managed cloud service. These security features include authentication, authorization, auditing, data encryption (including Queryable Encryption ), and the ability to access your data security with dedicated clusters deployed in a unique Virtual Private Cloud (VPC). End-to-end solution Figure 3: End-to-end data flow Interested readers can clone this repository and set up their own MongoDB-based smart meter data collection and anomaly detection solution. The solution follows the pattern illustrated in Figure 3, where a smart meter simulator generates raw data and transmits it via an MQTT topic. A Mosquitto broker receives these messages and then stores them in a MongoDB collection using the MongoDB Bridge. By leveraging MongoDB change streams , an algorithm can retrieve these messages, transform them according to MDMS requirements, and perform anomaly detection. The results are stored in a time series collection using a highly compressed format. The Cedalo MQTT Platform with MongoDB offers all the essential components for a flexible and scalable smart meter data management system, enabling a wide range of applications such as anomaly detection, outage management, and billing services. This solution empowers power distribution companies to analyze trends, implement real-time monitoring, and make informed decisions regarding their smart meter infrastructure. We are actively working with our clients to solve IoT challenges. Take a look at our Manufacturing and Industrial IoT page for more stories.

September 4, 2024

Transforming Industries with MongoDB and AI: Manufacturing and Motion

This is the first in a six-part series focusing on critical AI use cases across several industries . The series covers the manufacturing and motion, financial services, retail, telecommunications and media, insurance, and healthcare industries. The integration of artificial intelligence (AI) within the manufacturing and automotive industries has transformed the conventional value chain, presenting a spectrum of opportunities. Leveraging Industrial IoT, companies now collect extensive data from assets, paving the way for analytical insights and unlocking novel AI use cases, including enhanced inventory management and predictive maintenance. Inventory management Efficient supply chains can control operational costs and ensure on-time delivery to their customers. Inventory optimization and management is a key component in achieving these goals. Managing and optimizing inventory levels, planning for fluctuations in demand, and of course, cutting costs are all imperative goals. However, efficient inventory management for manufacturers presents complex data challenges too, primarily in forecasting demand accurately and optimizing stock levels. This is where AI can help. Figure 1: Gen AI-enabled demand forecasting with MongoDB Atlas AI algorithms can be used to analyze complex datasets to predict future demand for products or parts. Improvement in demand forecasting accuracy is crucial for maintaining optimal inventory levels. AI-based time series forecasting can assist in adapting to rapid changes in customer demand. Once the demand is known, AI can play a pivotal role in stock optimization. By analyzing historical sales data and market trends, manufacturers can determine the most efficient stock levels and even reduce human error. On top of all this existing potential, generative AI can help with generating synthetic inventory data and seasonally adjusted demand patterns. It can also help with creating scenarios to simulate supply chain disruptions. MongoDB Atlas makes this process simple. At the warehouse, the inventory can be scanned using a mobile device. This data is persisted in Atlas Device SDK and synced with Atlas using Device Sync, which is used by MongoDB customers like Grainger . Atlas Device Sync provides an offline-first seamless mobile experience for inventory tracking, making sure that inventory data is always accurate in Atlas. Once data is in Atlas, it can serve as the central repository for all inventory-related data. This repository becomes the source of data for inventory management AI applications, eliminating data silos and improving visibility into overall inventory levels and movements. Using Atlas Vector Search and generative AI, manufacturers can easily categorize products based on their seasonal attributes, cluster products with similar seasonal demand patterns, and provide context to the foundation model to improve the accuracy of synthetic inventory data generation. Predictive maintenance The most basic approach to maintenance today is reactive — assets are deliberately allowed to operate until failures actually occur. The assets are maintained as needed, making it challenging to anticipate repairs. Preventive maintenance, however, allows systems or components to be replaced based on a conservative schedule to prevent commonly occurring failures — although predictive maintenance is expensive to implement due to frequent replacement of parts before end-of-life. Figure 2: Audio-based anomaly detection with MongoDB Atlas. Scan the QR code to try it out yourself. AI offers a chance to efficiently implement predictive maintenance using data collected from IoT sensors on machinery trained to detect anomalies. ML/AI algorithms like regression models or decision trees are trained on the preprocessed data, deployed on-site for inference, and continuously analyzed sensor data. When anomalies are detected, alerts are generated to notify maintenance personnel, enabling proactive planning and execution of maintenance actions to minimize downtime and optimize equipment reliability and performance. A retrieval-augmented generation (RAG) architecture can be deployed to generate or curate the data preprocessor removing the need for specialized data science knowledge. The domain expert can provide the right prompts for the large language model. Once the maintenance alert is generated by an AI model, generative AI can come in again to suggest a repair strategy, taking spare parts inventory data, maintenance budget, and personal availability into consideration. Finally, the repair manuals can be vectorized and used to power a chatbot application that guides the technician in performing the actual repair. MongoDB documents are inherently flexible while allowing data governance when required. Since machine health prediction models require not just sensor data but also maintenance history and inventory data, the document model is a perfect fit to model such disparate data sources. During the maintenance and support process of a physical product, information such as product information and replacement parts documentation must be available and easily accessible to support staff. Full-text search capabilities provided by Atlas Search can be integrated with the support portal and help staff retrieve information from Atlas clusters with ease. Atlas Vector Search is a foundational element for effective and efficiently powered predictive maintenance models. Manufacturers can use MongoDB Atlas to explore ways of simplifying machine diagnostics. Audio files can be recorded from machines, which can then be vectorized and searched to retrieve similar cases. Once the cause is identified, they can use RAG to implement a chatbot interface that the technician can interact with and get context-aware, step-by-step guidance on how to perform the repair. Autonomous driving With the rise of connected vehicles, automotive manufacturers have been compelled to transform their business models into software-first organizations. The data generated by connected vehicles is used to create better driver assistance systems, paving the way for autonomous driving applications. However, it is challenging to create fully autonomous vehicles that can drive safer than humans. Some experts estimate that the technology to achieve level 5 autonomy is about 80% developed — but the remaining 20% will be extremely hard to achieve and will take a lot of time to perfect. Figure 3: MongoDB Atlas’s Role in Autonomous Driving AI-based image and object recognition in automotive applications face uncertainties, but manufacturers must utilize data from radar, LiDAR, cameras, and vehicle telemetry to improve AI model training. Modern vehicles act as data powerhouses, constantly gathering and processing information from onboard sensors and cameras, generating significant Big Data. Robust storage and analysis capabilities are essential to manage this data, while real-time analysis is crucial for making instantaneous decisions to ensure safe navigation. MongoDB can play a significant role in addressing these challenges. The document model is an excellent way to accommodate diverse data types such as sensor readings, telematics, maps, and model results. New fields to the documents can be added at run time, enabling the developers to easily add context to the raw telemetry data. MongoDB’s ability to handle large volumes of unstructured data makes it suitable for the constant influx of vehicle-generated information. Atlas Search provides a performant search engine to allow data scientists to iterate their perception AI models. Finally, Atlas Device Sync can be used to send configuration updates to the vehicle's advanced driving assistance system Other notable use cases AI plays a critical role in fulfilling the promise of Industry 4.0. Numerous other use cases of AI can be enabled by MongoDB Atlas, some of which include: Logistics Optimization: AI can help optimize routes resulting in reduced delays and enhanced efficiency in day-to-day delivery operations. Quality Control and Defect Detection: Computer or machine vision can be used to identify irregularities in the products as they are manufactured. This ensures that product standards are met with precision. Production Optimization: By analyzing time series data from sensors installed on production lines, waste can be identified and reduced, thereby improving throughput and efficiency. Smart After Sales Support: Manufacturers can utilize AI-driven chatbots and predictive analytics to offer proactive maintenance, troubleshooting, and personalized assistance to customers. Personalized Product Recommendations: AI can be used to analyze user behavior and preferences to deliver personalized product recommendations via a mobile or web app, enhancing customer satisfaction and driving sales. The integration of AI in manufacturing and automotive industries has revolutionized traditional processes, offering a plethora of opportunities for efficiency and innovation. With industrial IoT and advanced analytics, companies can now harness vast amounts of data to enhance inventory management and predictive maintenance. AI-driven demand forecasting ensures optimal stock levels, while predictive maintenance techniques minimize downtime and optimize equipment performance. Moreover, as automotive manufacturers work toward autonomous driving, AI-powered image recognition and real-time data analysis become paramount. MongoDB Atlas emerges as a pivotal solution, providing flexible document modeling and robust storage capabilities to handle the complexities of Industry 4.0. Beyond the manufacturing and automotive sectors, the potential of AI-enabled by MongoDB Atlas extends to logistics optimization, quality control, production efficiency, smart after-sales support, and personalized customer experiences, shaping the future of Industry 4.0 and beyond. Learn more about AI use cases for top industries in our new white paper, “ How Leading Industries are Transforming with AI and MongoDB Atlas .” Head over to our quick-start guide to get started with Atlas Vector Search today.

March 19, 2024

How MongoDB Enables Digital Twins in the Industrial Metaverse

The integration of MongoDB into the metaverse marks a pivotal moment for the manufacturing industry, unlocking innovative use cases across design and prototyping, training and simulation, and maintenance and repair. MongoDB's powerful capabilities — combined with Augmented Reality (AR) or Virtual Reality (VR) technologies — are reshaping how manufacturers approach these critical aspects of their operations, while also enabling the realization of innovative product features. But first: What is the metaverse, and why is it so important to manufacturers? We often use the term, "digital twin" to refer to a virtual replication of the physical world. It is commonly used for simulations and documentation. The metaverse goes one step further: Not only is it a virtual representation of a physical device or a complete factory, but the metaverse also reacts and changes in real time to reflect a physical object’s condition. The advent of the industrial metaverse over the past decade has given manufacturers an opportunity to embrace a new era of innovation, one that can enhance collaboration, visualization, and training. The industrial metaverse is also a virtual environment that allows geographically dispersed teams to work together in real-time. Overall, the metaverse transforms the way individuals and organizations interact to produce, purchase, sell, consume, educate, and work together. This paradigm shift is expected to accelerate innovation and affect everything from design to production across the manufacturing industry. Here are some of the ways the metaverse — powered by MongoDB — is having an impact on manufacturing. Design and prototyping Design and prototyping processes are at the core of manufacturing innovation. Within the metaverse, engineers and designers can collaborate seamlessly using VR, exploring virtual spaces to refine and iterate on product designs. MongoDB's flexible document-oriented structure ensures that complex design data, including 3D models and simulations, is efficiently stored and retrieved. This enables real-time collaboration, accelerating the design phase while maintaining the precision required for manufacturing excellence. Training and simulation Taking a digital twin and connecting it to physical assets enables training beyond traditional methods and provides immersive simulations in the metaverse that enhance skill development for manufacturing professionals. VR training, powered by MongoDB's capacity to manage diverse data types — such as time-series, key-values and events — enables realistic simulations of manufacturing environments. This approach allows workers to gain hands-on experience in a safe virtual space, preparing them for real-world challenges without affecting production cycles. Gamification is also one of the most effective ways to learn new things. MongoDB's scalability ensures that training data, including performance metrics and user feedback, is efficiently handled to continuously enlarge the training modules and the necessary resources for the ever-increasing amount of data. Maintenance and repair Maintenance and repair operations are streamlined through AR applications within the metaverse. The incorporation of AR and VR technologies into manufacturing processes amplifies the user experience, making interactions more intuitive and immersive. Technicians equipped with AR devices can access real-time information overlaid onto physical equipment, providing step-by-step guidance for maintenance and repairs. MongoDB's support for large volumes of diverse data types, including multimedia and spatial information, ensures a seamless integration of AR and VR content. This not only enhances the visual representation of data from the digital twin and the physical asset but also provides a comprehensive platform for managing the vast datasets generated during AR and VR interactions within the metaverse. Additionally, MongoDB's geospatial capabilities come into play, allowing manufacturers to manage and analyze location-based data for efficient maintenance scheduling and resource allocation. The result is reduced downtime through more efficient maintenance and improved overall operational efficiency. From the digital twin to metaverse with MongoDB The advantages of a metaverse for manufacturers are enormous, and according to Deloitte many executives are confident the industrial metaverse “ will transform research and development, design, and innovation, and enable new product strategies .” However, the realization is not easy for most companies. Challenges include managing system overload, handling vast amounts of data from physical assets, and creating accurate visualizations. The metaverse must also be easily adaptable to changes in the physical world, and new data from various sources must be continuously implemented seamlessly. Given these challenges, having a data platform that can contextualize all the data generated by various systems and then feed that to the metaverse is crucial. That is where MongoDB Atlas , the leading developer data platform, comes in, providing synchronization capabilities between physical and virtual worlds, enabling flexible data modeling, and providing access to the data via a unified query interface as seen in Figure 1. Figure 1: MongoDB connecting to a physical & virtual factory Generative AI with Atlas Vector Search With MongoDB Atlas, customers can combine three systems — database, search engine, and sync mechanisms — into one, delivering application search experiences for metaverse users 30% to 50% faster . Atlas powers use cases such as similarity search, recommendation engines, Q&A systems, dynamic personalization, and long-term memory for large language models (LLMs). Vector data is integrated with application data and seamlessly indexed for semantic queries, enabling customers to build easier and faster. MongoDB Atlas enables developers to store and access operational data and vector embeddings within a single unified platform. With Atlas Vector Search , users can generate information for maintenance, training, and all the other use cases from all possible information that is accessible. This information can come from text files such as Word, from PDFs, and even from pictures or sound streams from which an LLM then generates an accurate semantic answer. It’s no longer necessary to keep dozens of engineers busy, just creating useful manuals that are outdated at the moment a production line goes through first commissioning. Figure 2: Atlas Vector Search Transforming the manufacturing industry with MongoDB In the digital twin and metaverse-driven future of manufacturing, MongoDB emerges as a linchpin, enabling cost-effective virtual prototyping, enhancing simulation capabilities, and revolutionizing training processes. The marriage of MongoDB with AR and VR technologies creates a symbiotic relationship, fostering innovation and efficiency across design, training, and simulation. As the manufacturing industry continues its journey into the metaverse, the partnership between MongoDB and virtual technologies stands as a testament to the transformative power of digital integration in shaping the future of production. Learn more about how MongoDB is helping organizations innovate with the industrial metaverse by reading how we Build a Virtual Factory with MongoDB Atlas in 5 Simple Steps , how IIoT data can be integrated in 4 steps into MongoDB, or how MongoDB drives Innovations End-To-End in the whole Manufacturing Chain .

March 12, 2024