Community

Nenad Milosavljevic

IoT (Internet of Things) systems are increasingly becoming a part of our daily lives, offering smart solutions for homes and businesses. 

This article will explore a practical case study on household power consumption, showcasing how MongoDB's time series collections can be leveraged to store, manage, and analyze data generated by IoT devices efficiently.

## Time series collections

[Time series collections](https://www.mongodb.com/docs/manual/core/timeseries-collections/) in MongoDB effectively store time series data — a sequence of data points analyzed to observe changes over time.

Time series collections provide the following benefits:

- Reduced complexity for working with time series data
- Improved query efficiency
- Reduced disk usage
- Reduced I/O for read operations
- Increased WiredTiger cache usage

Generally, time series data is composed of the following elements:

- The timestamp of each data point
- Metadata (also known as the source), which is a label or tag that uniquely identifies a series and rarely changes
- Measurements (also known as metrics or values), representing the data points tracked at increments in time — generally key-value pairs that change over time

## Case study: household electric power consumption

This case study focuses on analyzing the [data set](https://archive.ics.uci.edu/dataset/235/individual+household+electric+power+consumption) with over two million data points of household electric power consumption, with a one-minute sampling rate over almost four years.

The dataset includes the following information:

- **date**: Date in format dd/mm/yyyy 
- **time**: Time in format hh:mm:ss 
- **global_active_power**: Household global minute-averaged active power (in kilowatt) 
- **global_reactive_power**: Household global minute-averaged reactive power (in kilowatt) 
- **voltage**: Minute-averaged voltage (in volt) 
- **global_intensity**: Household global minute-averaged current intensity (in ampere) 
- **sub_metering_1**: Energy sub-metering No. 1 (in watt-hour of active energy); corresponds to the kitchen, containing mainly a dishwasher, an oven, and a microwave (hot plates are not electric but gas-powered) 
- **sub_metering_2**: Energy sub-metering No. 2 (in watt-hour of active energy); corresponds to the laundry room, containing a washing machine, a tumble drier, a refrigerator, and a light. 
- **sub_metering_3**: Energy sub-metering No. 3 (in watt-hour of active energy); corresponds to an electric water heater and an air conditioner

## Schema modeling

To define and model our time series collection, we will use the [Mongoose](https://www.npmjs.com/package/mongoose) library. Mongoose, an Object Data Modeling (ODM) library for MongoDB, is widely used in the Node.js ecosystem for its ability to provide a straightforward way to model our application data.

The schema will include:

- **timestamp:** A combination of the “date” and “time” fields from the dataset.
- **global_active_power**: A numerical representation from the dataset.
- **global_reactive_power**: A numerical representation from the dataset. 
- **voltage**: A numerical representation from the dataset. 
- **global_intensity**: A numerical representation from the dataset.
- **sub_metering_1**: A numerical representation from the dataset. 
- **sub_metering_2**: A numerical representation from the dataset.
- **sub_metering_3**: A numerical representation from the dataset.

To configure the collection as a time series collection, an additional “**timeseries**” configuration with “**timeField**” and “**granularity**” properties is necessary. The “**timeField**” will use our schema’s “**timestamp**” property, and “**granularity**” will be set to “minutes” to match the dataset's sampling rate.

Additionally, an index on the “timestamp” field will be created to enhance query performance — note that you can query a time series collection the same way you query a standard MongoDB collection.

The resulting schema is structured as follows:

```javascript
const { Schema, model } = require('mongoose');

const powerConsumptionSchema = new Schema(
    {
        timestamp: { type: Date, index: true },
        global_active_power: { type: Number },
        global_reactive_power: { type: Number },
        voltage: { type: Number },
        global_intensity: { type: Number },
        sub_metering_1: { type: Number },
        sub_metering_2: { type: Number },
        sub_metering_3: { type: Number },
    },
    {
        timeseries: {
            timeField: 'timestamp',
            granularity: 'minutes',
        },
    }
);

const PowerConsumptions = model('PowerConsumptions', powerConsumptionSchema);

module.exports = PowerConsumptions;
```

For further details on creating time series collections, refer to [MongoDB's official time series documentation](https://www.mongodb.com/docs/manual/core/timeseries/timeseries-procedures/).

## Inserting data to MongoDB

The [dataset](https://archive.ics.uci.edu/dataset/235/individual+household+electric+power+consumption) is provided as a .txt file, which is not directly usable with MongoDB. To import this data into our MongoDB database, we need to preprocess it so that it aligns with our database schema design.

This can be accomplished by performing the following steps:

1. Connect to MongoDB.
2. Load data from the .txt file.
3. Normalize the data and split the content into lines.
4. Parse the lines into structured objects.
5. Transform the data to match our MongoDB schema model.
6. Filter out invalid data.
7. Insert the final data into MongoDB in chunks.

Here is the Node.js script that automates these steps:

```javascript
// Load environment variables from .env file
require('dotenv').config();

// Import required modules
const fs = require('fs');
const mongoose = require('mongoose');
const PowerConsumptions = require('./models/power-consumption');

// Connect to MongoDB and process the data file
const processData = async () => {
    try {
        // Connect to MongoDB using the connection string from environment variables
        await mongoose.connect(process.env.MONGODB_CONNECTION_STRING);

        // Define the file path for the data source
        const filePath = 'Household_Power_Consumption.txt';

        // Read data file
        const rawFileContent = fs.readFileSync(filePath, 'utf8');

        // Normalize line endings and split the content into lines
        const lines = rawFileContent.replace(/\r\n/g, '\n').replace(/\r/g, '\n').trim().split('\n');

        // Extract column headers
        const headers = lines[0].split(';').map((header) => header.trim());

        // Parse the lines into structured objects
        const parsedRecords = lines.slice(1).map((line) => {
            const values = line.split(';').map((value) => value.trim());
            return headers.reduce((object, header, index) => {
                object[header] = values[index];
                return object;
            }, {});
        });

        // Transform and prepare data for insertion
        const transformedRecords = parsedRecords.map((item) => {
            const [day, month, year] = item.Date.split('/').map((num) => parseInt(num, 10));
            const [hour, minute, second] = item.Time.split(':').map((num) => parseInt(num, 10));
            const dateObject = new Date(year, month - 1, day, hour, minute, second);

            return {
                timestamp: dateObject.toISOString(),
                global_active_power: parseFloat(item.Global_active_power),
                global_reactive_power: parseFloat(item.Global_reactive_power),
                voltage: parseFloat(item.Voltage),
                global_intensity: parseFloat(item.Global_intensity),
                sub_metering_1: parseFloat(item.Sub_metering_1),
                sub_metering_2: parseFloat(item.Sub_metering_2),
                sub_metering_3: parseFloat(item.Sub_metering_3),
            };
        });

        // Filter out invalid data
        const finalData = transformedRecords.filter(
            (item) =>
            item.timestamp !== 'Invalid Date' &&
            !isNaN(item.global_active_power) &&
            !isNaN(item.global_reactive_power) &&
            !isNaN(item.voltage) &&
            !isNaN(item.global_intensity) &&
            !isNaN(item.sub_metering_1) &&
            !isNaN(item.sub_metering_2) &&
            !isNaN(item.sub_metering_3)
        );

        // Insert final data into the database in chunks of 1000
        const chunkSize = 1000;
        for (let i = 0; i < finalData.length; i += chunkSize) {
            const chunk = finalData.slice(i, i + chunkSize);
            await PowerConsumptions.insertMany(chunk);
        }

        console.log('Data processing and insertion completed.');
    } catch (error) {
        console.error('An error occurred:', error);
    }
};

// Call the processData function
processData();
```

Before you start the script, you need to make sure that your environment variables are set up correctly. To do this, create a file named “.env” in the root folder, and add a line for “MONGODB_CONNECTION_STRING”, which is your link to the MongoDB database. 

The content of the .env file should look like this:

```javascript
MONGODB_CONNECTION_STRING = 'mongodb+srv://{{username}}:{{password}}@{{your_cluster_url}}/{{your_database}}?retryWrites=true&w=majority'
```

For more details on constructing your connection string, refer to the [official MongoDB documentation](https://www.mongodb.com/docs/manual/reference/connection-string/).

## Visualization with MongoDB Atlas Charts

Once the data has been inserted into our MongoDB time series collection, [MongoDB Atlas Charts](https://www.mongodb.com/products/charts) can be used to effortlessly connect to and visualize the data.

In order to connect and use MongoDB Atlas Charts, we should:

1. Establish a connection to the time series collection as a data source.
2. Associate the desired fields with the appropriate X and Y axes.
3. Implement filters as necessary to refine the data displayed.
4. Explore the visualizations provided by Atlas Charts to gain insights.

![Atlas Charts Visualization for Household Power Consumption][1]

In the above example, we visualized the power consumption from various sources within a single day. The visualization revealed distinct usage patterns: Kitchen equipment was primarily used in the morning and evening, laundry room equipment was active around noon, and the water heater and air conditioner showed continuous use from morning to evening.

For the displayed visualization, we used a query to filter the data for a specific date:

```
{ timestamp: { $gt: ISODate('2007-01-17T00:00:00.000-00:00'), $lt: ISODate('2007-01-18T00:00:00.000-00:00') } }
```

If you want to change what is shown in the charts, you can apply different filters or aggregation pipelines to the data, tailoring the results according to your needs.

## Conclusion

This article demonstrates the powerful capabilities of MongoDB when integrated with IoT systems. By leveraging MongoDB's time series collection, we can efficiently store, manage, and analyze the large volumes of time-series data generated by IoT devices.

The case study on household power consumption not only showcases the practical applications of IoT in our daily lives but also highlights how MongoDB can help us get a deeper understanding of IoT data sets.

Through visualization with MongoDB Atlas Charts, we have gained significant insights into power consumption patterns. This not only helps in making informed decisions but also opens the door for significant improvements in energy efficiency and cost savings.

As we have explored the capabilities of MongoDB in handling IoT data and visualizing it with Atlas Charts, I hope it gets you excited to work more on your own data projects. I invite you to join the [MongoDB Community Forums](https://www.mongodb.com/community/forums/) to share your experiences, ask questions, and collaborate with fellow enthusiasts. Whether you are seeking advice, sharing your latest project, or exploring innovative uses of MongoDB, the community is a great place to continue the conversation.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt50c50df158186ba2/65f8b9fad467d26c5f0bbf14/image1.png

Explore a case study on household power consumption and how MongoDB's time series collections are leveraged to store and analyze data from IoT devices.

IoT and MongoDB: Powering Time Series Analysis of Household Power Consumption

One of the most distinctive projects I've worked on is an application named Entangled. Developed in partnership with the Princeton Engineering Anomalies Research lab (PEAR), The Global Consciousness Project, and the Institute of Noetic Sciences, Entangled aims to test human consciousness. 

The application utilizes a quantum random number generator to measure the influence of human consciousness. This quantum generator is essential because conventional computers, due to their deterministic nature, cannot generate truly random numbers. The quantum generator produces random sequences of 0s and 1s. In large datasets, there should be an equal number of 0s and 1s.

For the quantum random number generation, we used an in-house [Quantis QRNG USB](https://www.idquantique.com/random-number-generation/products/quantis-software) device. This device is plugged into our server, and through specialized drivers, we programmatically obtain the random sequences directly from the USB device.

Experiments were conducted to determine if a person could influence these quantum devices with their thoughts, specifically by thinking about more 0s or 1s. The [results](https://www.youtube.com/watch?v=TRkAaiJv6zw) were astonishing, demonstrating the real potential of this influence. 

To expand this test globally, we developed a new application. This platform allows users to sign up and track their contributions. The system generates a new random number for each user every second. Every hour, these contributions are grouped for analysis at personal, city, and global levels. We calculate the standard deviation of these contributions, and if this deviation exceeds a certain threshold, users receive notifications. 

This data supports various experiments. For instance, in the "Earthquake Prediction" experiment, we use the contributions from all users in a specific area. If the standard deviation is higher than the set threshold, it may indicate that users have predicted an earthquake.

If you want to learn more about Entangled, you can check the [official website](https://entangled.org).

## Hourly-metrics schema modeling 

As the lead backend developer, and with MongoDB being my preferred database for all projects, it was a natural choice for Entangled.

For the backend development, I chose Node.js (Express), along with the Mongoose library for schema definition and data modeling. Mongoose, an Object Data Modeling (ODM) library for MongoDB, is widely used in the Node.js ecosystem for its ability to provide a straightforward way to model our application data.

Careful schema modeling was crucial due to the anticipated scaling of the database. Remember, we were generating one random number per second for each user. 

My initial instinct was to create hourly-based schemas, aligning with our hourly analytics snapshots. The initial schema was structured as follows:

- User: a reference to the "Users" collection 
- Total Sum: the sum of each user's random numbers; either 1s or 0s, so their sum was sufficient for later analysis 
- Generated At: the timestamp of the snapshot 
- Data File: a reference to the "Data Files" collection, which contains all random numbers generated by all users in a given hour

```javascript
const { Schema, model } = require("mongoose");

const hourlyMetricSchema = new Schema({
    user: { type: Schema.Types.ObjectId, ref: "Users" },
    total_sum: { type: Number },
    generated_at: { type: Date },
    data_file: { type: Schema.Types.ObjectId, ref: "DataFiles" }
});

// Compound index forr "user" (ascending) and "generated_at" (descending) fields
hourlyMetricSchema.index({ user: 1, generated_at: -1 });

const HourlyMetrics = model("HourlyMetrics", hourlyMetricSchema);

module.exports = HourlyMetrics;
```

Although intuitive, this schema faced a significant scaling challenge. We estimated over 100,000 users soon after launch. This meant about 2.4 million records daily or 72 million records monthly. Consequently, we were looking at approximately 5GB of data (including storage and indexes) each month. 

This encouraged me to explore alternative approaches.

## Daily-metrics schema modeling 

I explored whether alternative modeling approaches could further optimize storage requirements while also enhancing scalability and cost-efficiency. 

A significant observation was that out of 5GB of total storage, 3.5GB was occupied by indexes, a consequence of the large volume of documents. 

This led me to experiment with a schema redesign, shifting from hourly to daily metrics. The new schema was structured as follows:

```javascript
const { Schema, model } = require("mongoose");

const dailyMetricSchema = new Schema({
    user: { type: Schema.Types.ObjectId, ref: "Users" },
    date: { type: Date },
    samples: [
        {
            total_sum: { type: Number },
            generated_at: { type: Date },
            data_file: { type: Schema.Types.ObjectId, ref: "DataFiles" }
        }
    ]
});

// Compound index forr "user" (ascending) and "date" (descending) fields
hourlyMetricSchema.index({ user: 1, date: -1 });

const DailyMetrics = model("DailyMetrics", dailyMetricSchema);

module.exports = DailyMetrics;
```

Rather than storing metrics for just one hour in each document, I now aggregated an entire day's metrics in a single document. Each document included a "samples" array with 24 entries, one for each hour of the day. 

It's important to note that this method is a good solution because the array has a fixed size — a day only has 24 hours. This is very different from the [anti-pattern of using big, massive arrays in MongoDB](https://www.mongodb.com/developer/products/mongodb/schema-design-anti-pattern-massive-arrays).

This minor modification had a significant impact. The storage requirement for a month's worth of data drastically dropped from 5GB to just 0.49GB. This was mainly due to the decrease in index size, from 3.5GB to 0.15GB. The number of documents required each month dropped from 72 million to 3 million. 

Encouraged by these results, I didn't stop there. My next step was to consider the potential benefits of shifting to a monthly-metrics schema. Could this further optimize our storage? This was the question that drove my next phase of exploration.

## Monthly-metrics schema modeling 

The monthly-metrics schema was essentially identical to the daily-metrics schema. The key difference lay in how the data was stored in the "samples" array, which now contained approximately 720 records representing a full month's metrics.

```javascript
const { Schema, model } = require("mongoose");

const monthlyMetricSchema = new Schema({
    user: { type: Schema.Types.ObjectId, ref: "Users" },
    date: { type: Date },
    samples: [
        {
            total_sum: { type: Number },
            generated_at: { type: Date },
            data_file: { type: Schema.Types.ObjectId, ref: "DataFiles" }
        }
    ]
});

// Compound index forr "user" (ascending) and "date" (descending) fields
monthlyMetricSchema.index({ user: 1, date: -1 });

const MonthlyMetrics = model("MonthlyMetrics", monthlyMetricSchema);

module.exports = MonthlyMetrics;
```

This adjustment was expected to further reduce the document count to around 100,000 documents for a month, leading me to anticipate even greater storage optimization. However, the actual results were surprising. 

Upon storing a month's worth of data under this new schema, the storage size unexpectedly increased from 0.49GB to 0.58GB. This increase is likely due to the methods MongoDB's WiredTiger storage engine uses to compress arrays internally.

## Summary

Below is a detailed summary of the different approaches and their respective results for one month’s worth of data:

|                                  | **Hourly Document**                             | **Daily Document**                  | **Monthly Document**    |
| -------------------------------- | ----------------------------------------------- | ----------------------------------- | ----------------------- |
| **Document Size**                | 0.098 KB                                        | 1.67 KB                             | 49.18 KB                |
| **Total Documents (per month)**  | 72,000,000 (100,000 users * 24 hours * 30 days) | 3,000,000 (100,000 users * 30 days) | 100,000 (100,000 users) |
| **Storage Size**                 | 1.45 GB                                         | 0.34 GB                             | 0.58 GB                 |
| **Index Size**                   | 3.49 GB                                         | 0.15 GB                             | 0.006 GB                |
| **Total Storage (Data + Index)** | 4.94 GB                                         | 0.49 GB                             | 0.58 GB                 |

## Conclusion

In this exploration of schema modeling for the Entangled project, we investigated the challenges and solutions for managing large-scale data in MongoDB.

Our journey began with hourly metrics, which, while intuitive, posed significant scaling challenges due to the large volume of data and index size. 

This prompted a shift to daily metrics, drastically reducing storage requirements by over 10 times, primarily due to a significant decrease in index size. 

The experiment with monthly metrics offered an unexpected twist. Although it further reduced the number of documents, it increased the overall storage size, likely due to the internal compression mechanics of MongoDB's WiredTiger storage engine. 

This case study highlights the critical importance of schema design in database management, especially when dealing with large volumes of data. It also emphasizes the need for continuous experimentation and optimization to balance storage efficiency, scalability, and cost.

If you want to learn more about designing efficient schemas with MongoDB, I recommend checking out the [MongoDB Data Modeling Patterns](https://www.mongodb.com/blog/post/building-with-patterns-a-summary) series.

Learn how to reduce your storage in MongoDB by optimizing your data model through various techniques.

Nenad Milosavljevic

Content by Nenad Milosavljevic

IoT and MongoDB: Powering Time Series Analysis of Household Power Consumption

Entangled: A Story of Data Re-modeling and 10x Storage Reduction