Real-Time ESG Data Management

Wei You Pan
February 14, 2023 | Updated: November 19, 2024

ESG (Environmental, Social, and Governance) data collection and reporting has become a corporate priority, with over 96% of S&P 500 companies publishing sustainability reports in 2021, according to research from the Governance and Accountability Institute.

There are several factors driving the adoption and use of ESG data; ranging from consumer preference for companies with positive ESG information, to employees, who increasingly believe environmental, social, and governance metrics are important indicators when choosing an employer.

Many government bodies and regulators either have, or are considering, mandatory ESG data collection and ESG data reporting requirements for corporations under their jurisdiction. The European Union is taking the lead here, with several key pieces of legislation either already enacted, or coming soon. In the US, the SEC has also announced proposed rule changes for securities reporting, mandating companies make detailed climate-related disclosures in their filings.

In addition to companies that report on their own data, financial firms, including the private equity industry, use ESG data and research to weigh risks and identify opportunities for the companies they invest in.

Faced with growing scrutiny around ESG reporting and scoring, companies are struggling to meet ever more detailed and comprehensive reporting requirements. At the heart of the problem is the sheer volume and variety of data companies are expected to ingest and analyze to produce the scores that investors, consumers, and government entities demand. And with real-time data making its way into reports, ESG data management is becoming even harder.

ESG data collection and analysis

The volume and variety of ESG data makes collection and analysis difficult. The data collection problem can be broken down as follows:

Variety

Unlike financial datasets, which are mostly numerical, ESG metrics can include both structured and unstructured datasets, like an email or a media report. If a company wants to analyze satellite data to derive their own climate dataset, they may even need to analyze images and videos. Given these variables, companies need to employ a data model that can support many different types of data.

Velocity

As companies increasingly integrate real-time data sources into their ESG scoring systems, the velocity of data collected and analyzed increases exponentially. One example is loan due diligence in the financial sector. As customers demand faster loan approval turnaround times, financial institutions that currently rely on quarterly ESG data to make those decisions now need the information in real-time to instantly approve loans in an ESG compliant manner.

Volume

The increased variety of data sources, coupled with the growing velocity of data being collected leads to an increase in the sheer volume of data requiring analysis. Currently, ESG ratings and scores are derived from a blend of human judgment and model driven quantitative rating. But as the volume of data increases, along with the need for instant analysis of that data, real-time analytics and an increased use of AI/ML tools will become an ever greater part of ESG ratings and reporting.

On top of this, there are also no universally applicable ESG standards, leaving companies having to deal with multiple different standards, with different data requirements, depending on which jurisdictions they operate in.

Real-time ESG data analytics

Companies are increasingly incorporating real-time data into their ESG analysis, reporting, and scoring. Harnessing technologies such as cloud computing, AI, and machine learning, those that utilize real-time data can, for instance, instantly parse breaking news stories for ESG-related data on their investments, or incorporate up-to-the-minute satellite data into reports on a firm’s environmental impact.

The financial services industry in particular is taking a lead on integrating real-time ESG data into investment decisions.

Asset and fund managers use real-time data platforms that allow them to calculate up-to-date ESG scores to aid investment decisions and risk calculations. For example, a bank looking to invest in an electric vehicle company would be alerted to a breaking news story about a hazardous accident at the manufacturer’s battery plant, with follow up data from social media or analyst reports quantifying the size of the public reaction and the level of negative market sentiment around the accident.

MongoDB and ESG data management

MongoDB Atlas is an ideal data foundation for ESG platforms. MongoDB Atlas uses the document data model, giving users the ability to ingest data from almost any source, consolidate data from a number of siloed data sets, enable the easy search of that data, and with a few clicks, create customized views of the data without the need for additional ETL operations to other databases or tools.

MongoDB Atlas also future-proofs your ESG data platform with a flexible data schema that can easily adapt to rapidly changing ESG requirements and standards.

Graphic titled MongoDB optimizes ESG data & insights. The graphic is broken into four sections. The first section is titled cost optimization and says easily query and combine ESG data across data sources using Atlas Data Federation and optimize storage cost with Atlas Online Archive. The second section is titled compute intelligence and says use Atlas's aggregation framework to group values from multiple documents together, and perform a variety of operations on the grouped data to return a single result. The third section is titled search efficiency and says with Atlas text and vector search, quickly find the right ESG data without the need for a seperate search system alongside your database. Finally, the fourth section is titled analysis simplicity and says use Atlas charts to analyze and visualize ESG data, metrics, and KPIs without moving data elsewhere.

See why Hydrus chose MongoDB Atlas as the basis for its ESG reporting platform.

FAQ

ESG data definition

ESG (Environment, Social, and Governance) data comes from a growing list of sources, all of which help “score” a corporation based on how well positioned it is to handle the risks and opportunities presented by the environment, societal stakeholders, and corporate governance.

Environment - What are a company's greenhouse gas emissions? How about its stewardship over natural resources? And how well positioned is it to weather physical climate risks, like global warming, flooding, drought, fire etc.

Social - How does a company measure up against prevailing fair wage and employee engagement metrics? What impact does a company have on the communities where it operates?

Governance - How well is a company managed? How responsive is a company to shareholders? How accountable is leadership? What safeguards are in place to ensure transparency?

The growing interest around ESG data science and data analytics has prompted the rise of a new industry of ESG data companies and ESG data management software vendors.

What are the different ESG data sources?

ESG data may come from two primary sources; 'inside-out' and 'outside-in'.

Inside-out data is supplied by companies, used for analysis, and usually lags 6-12 months due to annual ESG-related disclosures.

Outside-in data is more regularly updated, sometimes even in real time. Most financial institutions, including banks who often have access to a lot of financial and company data from their customers, do not rely solely on their own data. ESG data analysis requires a broad range of inputs and data that the bank does not possess or can obtain even from their customers.

For example, a bank may want to assess the risk of flooding for a chip manufacturing company that has factories in several provinces in China. The bank would need to collect the flood data from the different operating locations in order to score the risk.

As banks don’t typically collect flood data themselves, the bank would purchase data from third-party climate data vendors. At this nascent stage of climate risk assessment within the banking industry, it is likely that the bank would not even attempt to collect the raw climate data and create the risk models to score the risk, relying instead on third-party risk scoring vendors.

The bank would then make use of these scores and combine in models which they have strong competencies eg. credit risk to come up with flood risk-adjusted credit risk scores for loan approvals.

Why is ESG data essential for investors?

ESG data is used by asset managers and investors for market analysis, supporting asset allocation and risk management, and in providing insights into the long-term sustainability of investments in various corporations.

← Previous

Honoring Black History Month: How These MongoDB Employees Defied the Odds

February is Black History Month. It’s a time to reflect on and celebrate the struggles and triumphs of the black community and remember the importance of elevating black voices. Each year at MongoDB, we ask members of our employee resource group BEAM (Black Employees At MongoDB) if they’d like to share a personal story about their experiences and what this month means to them. This year, hear from Administrative Assistant Rita Henderson and Regional Director Daniel Hawthorne to learn more about their journeys into tech. Rita Henderson: Breaking Down Barriers and Owning Technology for Social Justice As we celebrate Black History Month, I am grateful for those who have paved the way for us to have a voice and fight for our rights. I am reminded of the struggles and achievements of black leaders throughout history. The fight for equal rights and justice is ongoing, and technology plays a crucial role in this fight. It is important to empower and uplift underrepresented communities in the tech industry to create a more inclusive and equitable future. I am a proud member of the Afro-Latinx community from North Philadelphia. Growing up in a neighborhood called Badlands, I witnessed first-hand the impact and struggles of poverty, high crime rates, and drugs. I am the youngest of six children, with parents who worked two jobs to make ends meet. Despite my parents' hard work and dedication to provide for their children, life was still a struggle for my family. At the age of 17, after completing my junior year of High School, I became a teen mom. Unfortunately, society tries to shame young mothers, especially teen moms of color. Many people reminded me that teen pregnancy is closely linked to single parenthood and that growing up in single-parent families remains the largest factor in increased poverty among children. Me (middle) and my sibling with our dad. Yes, I photoshopped myself in. As a teen mom, I was determined to break through the barriers society placed on me. With $200 in my pocket, I moved my daughter and I to western Pennsylvania and enrolled in Indiana University of Pennsylvania. There, I earned my bachelor's degree in Criminology and studied the school-to-prison pipeline in black communities. After the murder of the young unarmed black teenager, Mike Brown, and the Ferguson uprising, my sister and I collaborated with organizers in the Ferguson community to launch a free technology program to empower community organizers, educators, and youth with skill sets to create technology tools for social and economic justice. Graduating from Indiana University of Pennsylvania. Pictured with my firstborn London Rae and my mom. I am influenced by the work of the Black Panther Party; specifically, the 10th Point of the party’s 10-Point Platform, “Community Control of Modern Technology”. For 45 years, the Black Panther Party included the right to learn, access, and control technology as a right. Huey said, "Knowing how to struggle is the essence of winning. Recognizing ills is fundamental; recognizing how to overcome ills is mandatory." That is why I believe it is critical for black and latinx people to understand the role technology plays in our society and the economy if we want to understand social justice and create tools for liberation. When I hear people talk about technology in black, latinx, and working-class communities, they often use it as a scare tactic. The fear of data and control and the feeling that technology is too advanced and that we lack the knowledge and tools to participate can be overwhelming. However, it is crucial for our community to claim our place in the tech world. We need to change our thinking and know there is a place for us, just like there is for anyone else. I am grateful for MongoDB's value "embrace the power of differences" and creating a platform where underrepresented communities can share their stories, bring their ideas to the forefront, and be heard in the tech industry. As we celebrate Black History Month, I am grateful for those who have paved the way for us to have a voice and fight for our rights. I am also thankful for the opportunities I have been given to make a difference in my community and empower others to do the same. With education and technology, we can continue breaking down barriers and striving for equality, justice, and liberation. In 2022, my partner and I welcomed our baby girl Lara Sky. Daniel Hawthorne: Building a Career as a Black Man in Tech Sales I was brought into the world with the odds against me, a black boy born in South Central Los Angeles in the 80s. However, I never felt that I was on my own. Throughout my entire life, God has choreographed my every step. At a very young age, my parents decided to move us to Austin, Texas where my grandparents were moving their church ministry. I was raised in Austin along with my two older brothers (Dante and Derrell), my younger sister (Amber), and my younger brother (Joseph). My siblings called me the “golden child” because I was a mama’s boy and kept to myself. The elementary and middle schools that I attended in Austin were fairly diverse, and I seldomly experienced racism. In the 7th grade, my family moved to a suburb of North Austin that wasn’t as diverse, and racist experiences became much more frequent. It was then that I began to acknowledge that being black brought different treatment. There were moments I embraced my blackness, but others where I was more focused on adapting myself into someone I thought those in my non-diverse environment wanted me to be. In middle school, the place to hang out was the Rec Center. I would run into kids from other schools, and we’d have the basketball gym to ourselves for a bit. Eventually, the older guys would take over the court, but I was good enough that I typically got to play with them. I remember observing them as they entered the gym. They’d be dressed in nice work clothes with Dell badges hanging from their shirts - the Rec Center was only five minutes from the Dell HQ - and that became an early image of what success looked like for me. In high school and college, I started my career in sales with a few small gigs. I enjoyed it because I was typically one of the top sellers no matter what I sold. I even sold women’s shoes at one point! After graduating with my M.B.A, I had no idea what my next move would be. But then, that image of success popped into my head. I focused my attention on getting a sales job at Dell. Despite not having any experience in tech, I knew I could excel. Who knew that 10 years after my days on the Rec Center courts, I would land my first job in tech. I joined the inside sales development team at Dell, and it was one of the most pivotal moments of my career. The job was intense. After a week of training, it was clear that I was the least technical in every room. But, I was determined to not let anyone outwork me. We were required to make over 100 outbound calls per day, but I quickly figured out how to achieve the true objective (10 scheduled virtual demonstrations in a week) in fewer calls. Through my efficiency, I helped form new standards and began to make a name for myself. Being in sales development wasn’t my end goal. I knew I wanted to get into outside sales, so I began building relationships with some of the Dell outside sellers I worked with. During a coaching session with one of my mentors, who was also a minority, he shared some guidance that I wasn’t ready for. He told me that if I truly wanted to be in outside sales, I needed to lose my earrings because professional men didn’t wear them. Even though he and I understood that earrings didn’t define me, his guidance was that being a person of color meant I was already playing from behind, and that I should exhaust all things within my control to create as level a playing field as I possibly could. This theme would continue throughout my career. Similar to when I was a kid in the non-diverse suburbs of Austin, as a black man in tech, I’ve felt heavy pressure to be a certain way to appease others. When I was first getting started, I hardly encountered sales folks that looked like me. I’d attend internal trainings and events where there might be one or two other black sellers out of 200+ people. In many ways, I felt that I was on an island and had to live through trial and error. I had a fear that being ‘too black’ would put me at an even greater disadvantage. I walked the line and was careful about what I said or did. I hardly engaged in extracurricular activities with co-workers, and when I did, I kept my guard up. So much of my energy and effort was exhausted into protecting my brand and trying to avoid negative stereotyping because of the color of my skin. I often think about how much more successful I could’ve been had I not felt obligated to focus on the things that never should’ve mattered. My wife and our two daughters at the apple orchards outside of St. Louis, Missouri. As I stated before, God has led my path in life. Numerous times when I was unsure of the next turn to make, He introduced someone to provide direction. I’m truly grateful for the people who may not have looked like me, but provided me with valuable coaching that helped guide my career in tech. I joined MongoDB to help customers with their data transformations, but I didn’t expect that I would go through a transformation myself. I’ve never felt more empowered to just be myself, and through that, I’ve reached new levels of individual and team accomplishments. I was a direct seller for my first two years with the company, and after receiving coaching from peers and leaders around me, I stepped into management a year ago. This wasn’t necessarily a milestone or goal that I had set out for myself, but I came to the realization that there was tremendous value in helping other sellers (and their families) achieve new levels of success. What better company to step into leadership than at MongoDB. Every company has employee resource groups nowadays, but the intentionality behind those groups at MongoDB is different. Our leadership team has leaned into those difficult, vulnerable discussions, sometimes simply to listen because they knew they didn’t have the answers. Even in those scenarios, they’d come up with relevant action that they could personally be responsible for. Despite the comfort zone I had created over the past 10+ years of watering down my blackness, our Sales team encourages individuality and has brought out the best version of me. It’s helped lift a giant weight off my back. I know I’m no longer starting from behind, and I don’t fear that folks are going to judge me. As I wrap-up my first year in sales leadership, I’ve noticed significant transformation in my personal development, and I’m excited that I get to continue taking on new challenges that will bring discomfort, but instill confidence that I can persevere. As we celebrate Black History Month, I think about the opportunity I have to expose other members of the black community to a profession in sales. Our experiences and our perspectives are highly valued and necessary in order to build a better tech-centric future. We’re passionate about cultivating a culture where people of all backgrounds, identities, and experiences feel valued and heard. Find your next career opportunity at MongoDB.

February 14, 2023
Next →

AI-Powered Call Centers: A New Era of Customer Service

Customer satisfaction is critical for insurance companies. Studies have shown that companies with superior customer experiences consistently outperform their peers. In fact, McKinsey found that life and property/casualty insurers with superior customer experiences saw a significant 20% and 65% increase in Total Shareholder Return , respectively, over five years. A satisfied customer is a loyal customer. They are 80% more likely to renew their policies, directly contributing to sustainable growth. However, one major challenge faced by many insurance companies is the inefficiency of their call centers. Agents often struggle to quickly locate and deliver accurate information to customers, leading to frustration and dissatisfaction. This article explores how Dataworkz and MongoDB can transform call center operations. By converting call recordings into searchable vectors (numerical representations of data points in a multi-dimensional space), businesses can quickly access relevant information and improve customer service. We'll dig into how the integration of Amazon Transcribe, Cohere, and MongoDB Atlas Vector Search—as well as Dataworkz's RAG-as-a-service platform— is achieving this transformation. From call recordings to vectors: A data-driven approach Customer service interactions are goldmines of valuable insights. By analyzing call recordings, we can identify successful resolution strategies and uncover frequently asked questions. In turn, by making this information—which is often buried in audio files— accessible to agents, they can give customers faster and more accurate assistance. However, the vast volume and unstructured nature of these audio files make it challenging to extract actionable information efficiently. To address this challenge, we propose a pipeline that leverages AI and analytics to transform raw audio recordings into vectors as shown in Figure 1: Storage of raw audio files: Past call recordings are stored in their original audio format Processing of the audio files with AI and analytics services (such as Amazon Transcribe Call Analytics ): speech-to-text conversion, summarization of content, and vectorization Storage of vectors and metadata: The generated vectors and associated metadata (e.g., call timestamps, agent information) are stored in an operational data store Figure 1: Customer service call insight extraction and vectorization flow Once the data is stored in vector format within the operational data store, it becomes accessible for real-time applications. This data can be consumed directly through vector search or integrated into a retrieval-augmented generation (RAG) architecture, a technique that combines the capabilities of large language models (LLMs) with external knowledge sources to generate more accurate and informative outputs. Introducing Dataworkz: Simplifying RAG implementation Building RAG pipelines can be cumbersome and time-consuming for developers who must learn yet another stack of technologies. Especially in this initial phase, where companies want to experiment and move fast, it is essential to leverage tools that allow us to abstract complexity and don’t require deep knowledge of each component in order to experiment with and realize the benefits of RAG quickly. Dataworkz offers a powerful and composable RAG-as-a-service platform that streamlines the process of building RAG applications for enterprises. To operationalize RAG effectively, organizations need to master five key capabilities: ETL for LLMs: Dataworkz connects with diverse data sources and formats, transforming the data to make it ready for consumption by generative AI applications. Indexing: The platform breaks down data into smaller chunks and creates embeddings that capture semantics, storing them in a vector database. Retrieval: Dataworkz ensures the retrieval of accurate information in response to user queries, a critical part of the RAG process. Synthesis: The retrieved information is then used to build the context for a foundational model, generating responses grounded in reality. Monitoring: With many moving parts in the RAG system, Dataworkz provides robust monitoring capabilities essential for production use cases. Dataworkz's intuitive point-and-click interface (as seen in Video 1) simplifies RAG implementation, allowing enterprises to quickly operationalize AI applications. The platform offers flexibility and choice in data connectors, embedding models, vector stores, and language models. Additionally, tools like A/B testing ensure the quality and reliability of generated responses. This combination of ease of use, optionality, and quality assurance is a key tenet of Dataworkz's "RAG as a Service" offering. Diving deeper: System architecture and functionalities Now that we’ve looked at the components of the pre-processing pipeline, let’s explore the proposed real-time system architecture in detail. It comprises the following modules and functions (see Figure 2): Amazon Transcribe , which receives the audio coming from the customer’s phone and converts it into text. Cohere ’s embedding model, served through Amazon Bedrock , vectorizes the text coming from Transcribe. MongoDB Atlas Vector Search receives the query vector and returns a document that contains the most semantically similar FAQ in the database. Figure 2: System architecture and modules Here are a couple of FAQs we used for the demo: Q: “Can you explain the different types of coverage available for my home insurance?” A: “Home insurance typically includes coverage for the structure of your home, your personal belongings, liability protection, and additional living expenses in case you need to temporarily relocate. I can provide more detailed information on each type if you'd like.” Q: “What is the process for adding a new driver to my auto insurance policy?" A: “To add a new driver to your auto insurance policy, I'll need some details about the driver, such as their name, date of birth, and driver's license number. We can add them to your policy over the phone, or you can do it through our online portal.” Note that the question is reported just for reference, and it’s not used for retrieval. The actual question is provided by the user through the voice interface and then matched in real-time with the answers in the database using Vector Search. This information is finally presented to the customer service operator in text form (see Fig. 3). The proposed architecture is simple but very powerful, easy to implement, and effective. Moreover, it can serve as a foundation for more advanced use cases that require complex interactions, such as agentic workflows , and iterative and multi-step processes that combine LLMs and hybrid search to complete sophisticated tasks. Figure 3: App interface, displaying what has been asked by the customer (left) and how the information is presented to the customer service operator (right) This solution not only impacts human operator workflows but can also underpin chatbots and voicebots, enabling them to provide more relevant and contextual customer responses. Building a better future for customer service By seamlessly integrating analytical and operational data streams, insurance companies can significantly enhance both operational efficiency and customer satisfaction. Our system empowers businesses to optimize staffing, accelerate inquiry resolution, and deliver superior customer service through data-driven, real-time insights. To embark on your own customer service transformation, explore our GitHub repository and take advantage of the Dataworkz free tier .

November 27, 2024