Overcome the Biggest Obstacle to True Customer 360: with MongoDB Atlas and Dataworkz

The more you know about your customers, the better you can attract them and increase their lifetime value to your business. The ability of modern systems to collect, store, and leverage customer data across disparate systems has raised the bar for what it means to understand your customer. To the winner goes the spoils, and the spoils go to the companies that navigate the last mile in the journey to attain true Customer 360: they leave no customer data behind. They hunt down every relevant source of customer data in their reach and turn them into valuable insight – without breaking a sweat.

How customer acquisition costs and customer lifetime value impacts business success

Two calculations are critical to understanding modeling how to grow your business.

  1. Customer Acquisition Costs (CAC): the amount of money a company spends to get a new customer

  2. Customer Lifetime Value (CLV): the net profit attributed to the customer relationship for the duration of the time they use a product

Note that “customer” is front and center in both terms. Capturing the data behind these two calculations correctly and continuously on a timely basis across the whole customer journey – from first touch to renewals – and making sense of it, is essential to understanding how to grow and retain your customers. These calculations must be performed on multiple customers and their interactions over a span of years. Analytical and AI models that calculate CLV and LTV need to be fed up-to-date, contextual, reliable data to identify relevant patterns.

Historical customer data guides future CLV and influences decisions on how to optimize investments in sales, marketing, and customer service to positively impact the customer’s future purchase and product adoption behaviors. Similarly, customer data analysis guides decisions on how to optimize sales and marketing investments to dial-in CAC for the fastest and most profitable conversion of prospects to customers.

How successful a company is at leveraging CLV, and LTV calculations depends on their ability to link different data sources and formats, and integrate all of the internal and external systems required for modeling. Until now, companies have struggled to accomplish this because data joining technologies have fallen short for one or more of these reasons: cost, complexity, and scalability.

Forrester1 declared Activation Of Unstructured “Dark” Data a 2022 mega trend. According to Forrester”

"The question is no longer about collecting enough data — it’s about ensuring that the data is usable and liberated from company silos to create new streams of value. In 2022, being “data driven” must become more than just a slogan — the next wave of business transformation will hinge on activating “dark” data (data that organizations collect but do not effectively use) to drive differentiated experiences for both customers and employees."

Creating that holistic 360° view of your customer is a business imperative for every organization–but creating a unified view or golden record of your customers is hard and expensive, especially if you want to capture and make sense of the structured and coherent unstructured data and include unstructured “dark” data.

Why creating a customer 360 is a technical challenge

Customer 360 data resides in your ecommerce platforms, Customer Relationship Management (CRM), Sales Force Automation (SFA), ERP, Customer Service systems, Partner Relationship Management programs, loyalty programs, payment portals, web apps, mobile apps and more–across business units and functional departments. Cross-object reporting in a single application (like Salesforce) can be a challenge but cross application reporting (SFDC + Zoom + Zendesk + Parquet files in a data lake) present an even bigger set of challenges with no common identifiers in the data collected from various applications. The data not only comes from a variety of sources but also in different formats significant time and data engineering effort is required to combine data to create the elusive golden record.

To compound the challenge, different teams require different views of the same data which demands different flavors of customer 360:


The power to harness customer data where it belongs: Business users

The traditional approach to creating a customer 360 takes the view that data from disparate sources like Prospecting (outreach.io), CRM(Salesforce), Marketing Automation (Marketo), Web Meetings (Zoom) and Support (Zendesk) needs to be combined, converted into a star or snowflake schema and deposited in an enterprise data warehouse. Or buy a Customer Data Platform (CDP) that makes yet another copy of your data that is a black box with no visibility into how data is processed. Keep in mind, this approach requires months of back and forth with the IT organization, resulting in cost and time overruns that lead to costly, stale customer 360 data that is no longer relevant.

Dataworkz gives anyone from marketing ops to sales leaders the ability to assemble a complete, reliable customer 360 in minutes or hours. Unlike a legacy CDP, Dataworkz eliminates data security concerns as a barrier because it does the job without storing customer data. Users configure their sources and destinations, then Dataworkz acts as the “personal data engineer” bringing together data, processing, and ML with a visual no-code interface for business users – with the option to use SQL.

A visual, easy-to-use with a no-code approach, to seamlessly gain a 360 view of your entire customer journey is the most innovative and efficient way to gain the contextual and quick customer insights necessary to grow your business.

No more kinks in your Customer 360 data: Dataworkz and MongoDB Atlas

Getting the best of all breeds is the mantra of smart, future-proof solutions. Capitalizing on Dataworkz and MongoDB Atlas together is a “no brainer.”

Connecting Dataworkz and Atlas

  • Add the Dataworkz cluster IP to the list of white listed IP addresses in the “network access” section of the Atlas console.

  • Navigate to the “Databases'' section in the Dataworkz configuration section and specify the Atlas cluster IP address, username and password.

Here are three ways this partnership can help:

  1. Get out of your RDBMS: Combining data from different sources, in different formats, with changing schema, is exceptionally hard, if not impossible with a rigid relational model. But MongoDB Atlas uses a JSON data model with a flexible schema which makes it ideal for building a customer 360.

  2. Work with a flexible, adaptive construct: Agility is a key tenet for organizations constantly adapting to their customer’s ever changing needs. With dynamic schemas, you are no longer worried about changes in source data breaking downstream applications.

  3. In minutes, not months: Dataworkz and MongoDB Atlas are SaaS services with usage based pricing. You get started without an upfront investment or hiring a team of data engineers and scientists and get to value in a matter of minutes and not weeks or months.

Customer case study - Gunderson Dettmer

Gunderson Dettmer is the preeminent international law firm with an exclusive focus on the innovation economy. In order to serve their clients, they need to leverage the latest and greatest information about corporate innovators anywhere in the world, from a wide variety of disparate data sources.

To accomplish this, they had to transform and integrate multiple data sources into a coherent, evolving data warehouse and attempt to match financial, legal, and personnel information across “un-mappable” data sets without any common identifiers or keys. The process had to be scalable, reliable, flexible, and continuous since new information and new data sources are added daily.

Gunderson needed to compare attributes for venture-backed companies from multiple internal and external datasets, but there were no globally unique identifiers available to join the datasets in most cases, and there were no off-the-shelf, end-to-end solutions to solve the problem. Working with Dataworkz, we developed an automated, scalable fuzzy matching pipeline that feeds likely matches to our internal infrastructure for manual review and approval, ultimately flowing into our MongoDB data warehouse.

Maria Concordia, Data Operations and Analytics Architect

The objective

Gunderson's data warehouse and analytics team were tasked with creating a brand new process for unlocking data insights by weaving together previously un-mappable data sets and serving them up to the organization to create an evolving, holistic view of the Venture Capital universe. The resulting data warehouse would be leveraged by downstream users, including data analysts, upper management, marketing, and other firm stakeholders.

The challenge with no single source of truth

Gunderson Dettmer needed to compare attributes for companies and venture capital investors across multiple internal and external datasets, but there were no globally unique identifiers available to join the datasets in most cases, and there were no off-the-shelf, end-to-end solutions to solve the problem. Data sources included Gunderson’s on-premise proprietary data held in a SQL-based relational database, CSV data dumps from Pitchbook, Crunchbase, and various PDF files, and other potential sources, as available. Although the data sets overlapped, there were no common keys or identifiers across them.

How they used power of MongoDB Atlas + Dataworkz

Before the data warehouse team at Gunderson got involved, the de facto approach was to create interim data sets for the entity matching process by extracting data from the relational database, performing multiple Excel spreadsheet lookups to enrich it, and then manually matching the exported data to the external data sources. This process typically took 3-4 weeks in order to create a unified data set that could be used for analytics. The process was static and needed to be repeated (complete with manual record matching) when any of the data sources were updated.

Not thrilled with this process, the data warehouse team created python scripts using Request and Fuzzy-Wuzzy modules to call a REST API endpoint, programmatically joining the datasets and then performing fuzzy logic for matching. Unfortunately, the REST API and Fuzzy matched datasets created were also one-time-only events and could not be leveraged for automating future analysis or ongoing data integration.

This is where MongoDB Atlas was introduced to change the one-time only dataset into a JSON collection that could be reused. It required transforming the resulting datasets into rich, nested JSON objects that could be searched and queried using MongoDB Atlas. While this reduced the data set preparation time to about one week, there was still a large human review process that required Subject Matter Experts (SMEs) for difficult matches that fell out of the simplistic, fuzzy match parameters.

For the final step in this project, they replaced the python scripts for transforming raw data into the JSON data model with Dataworkz to ingest data into MongoDB Atlas. The Dataworkz platform provided crucial features that vastly simplified the process of weaving the disparate “unmappable” data sets together into a coherent whole.

Using Dataworkz allowed Gunderson’s team to eliminate their python scripts and enabled:

  • Discovery, introspection, and manipulation of Parquet, BSON, JSON, CSV, structured, and unstructured data sources by bringing them all onto the same playing field for transformation,

  • A clearer understanding of the possible challenges to creating easy-to-match collections,

  • Automatically perform JSON format transformations without code,

  • The ability to describe and execute no-code multi-stage pattern matching

  • Automate their ingestion, transformation, and data storage pipeline in a day (rather than the months that it took to write the original python scripts),

  • Provide continuous automatic data integration within minutes, with improved accuracy, lower human intervention, and built-in data monitoring and governance to ensure against data stream contamination in the future.

Don't struggle any more for the metrics you need

Whether you are a Chief Revenue Officer, CMO, head of a regional team, leading customer service, or a data analyst/scientist; you no longer need to spend extended time and dollars to gain the insights that should be available at your fingertips. Timely, reliable CAC and CLV metrics that are correct no matter what changes occur in your processes, definitions, calculations, or systems can be yours – when you need them – across departments, product lines, and business units. Dataworkz marks the entry of a new class of data platform that delivers the insights you need – accessible across any data, from any source, combined for any use, when you need it.

← Previous

Implementing an Online Parquet Shredder

In Atlas Data Federation ( ADF ) we’ve found that many customers want to take their data, say from an Atlas cluster, and emit to their own blob storage as a parquet file. This is a popular and–for many customers–important feature we’re proud to provide. Converting data from our native BSON format , complete with flexible-schema and row-based representation, to the fixed-schema, columnar parquet format without compromises is not a computationally easy task. We’ve provided this to customers for years now , but with our February 21st release we’re excited to announce the release of a much-improved BSON-to-Parquet writer. Among behind-the-scenes changes, one exciting improvement that will be immediately visible to customers is that the efficiency of our parquet-writer is substantively improved both for CPU and memory-usage. From our internal benchmarks we see a 2x improvement in throughput, and our production metrics indicate that about bears out for our customer workloads, of course with some variance. We have only ever seen throughput improvements; we have not found a single workload that has worse performance. In this blog post I’ll share a high-level view of the core technical challenge and how we built together at MongoDB to create this solution. The technical challenge If you use ADF and issue an aggregate that ends with an $out stage and the parquet format specified, we’ll take the result of that aggregation and emit it to a parquet file in the named blob storage. We can imagine an aggregate’s result as a stream of BSON documents. The role ADF plays, then, is to take that document-stream, convert it to a parquet file, and place it in the blob storage. Smoothing away the rough details, we’re left with the core technical challenge: We want to convert a stream of row-based, flexible-schema BSON documents into a columnar, fixed-schema parquet file. That’s quite a challenge! Our initial approach Our past approach understandably focused on stability and simplicity over everything else. We built out our initial parquet offering by leveraging existing, production-tested subcomponents. In broad strokes, it did the following: Scan through the entire document stream to build the global schema that describes them all. Open a parquet file with that schema, and then scan through the documents again, “shredding” their contents one-by-one into the parquet file’s columns. This two-pass algorithm worked and served our customers, but has two fundamental drawbacks. The most immediate drawback is exactly that it is two-pass, and the second pass has to entirely wait on the first: we’re unable to create a parquet file until we know the schema, and we can’t be certain of what the schema is until we’ve seen the last document (for example, the last document may introduce a new field). The second drawback is how, when we shred our documents, we’re placing them directly in the file. A consequence of this is that, absent significant re-engineering, it is very difficult to flush the contents to disk until the file is complete. 1 As we saw customers using $out-to-parquet more and more, we knew it would be worth the investment to shift to more performant machinery. Changes for our new approach Our new approach introduces what we call the Online Shredder . This algorithm enables us to effectively shred a stream of documents into a columnar format in a single pass, building the schema in parallel. It doesn’t shred immediately into a parquet file, but instead to an intermediate file that enables us to more easily swap memory to disk. Finally, we’re excited to leverage the apache-go parquet writer . We’ve found it to be an exceptionally performant and feature-rich library. Most of the rest of this blog post is a high-level description of the design and implementation of the online shredder that supports this new approach. The new approach in-depth We have a stream of documents, and we want to shred them into a columnar form. The overall challenge behind this operation is supporting MongoDB’s flexible schema and maintaining the right information for lossless parquet emission. In particular, the “language barrier” between MongoDB’s row-based, flexible schema and Parquet’s column-based, fixed schema introduces 3 hurdles: Missing or New Fields: One document may have a field, and then a later document doesn’t include that field. Or, we may have a new document that for the first-time-ever introduces a new field. Polymorphism: Two documents may have a field with the same namepath, but different types. Structural Metadata: Parquet enables reassembly of its records using metadata that encodes information derived from the global schema. With our online approach, we have to commit to this metadata without yet fully knowing the global schema. We’ll describe the design of the online shredder first by explaining how it builds a schema as it scans over the document. Then we’ll build off of those concepts and extend them to not just build the schema, but shred the documents into an in-memory columnar representation. We will see that the technical challenge in shredding comes from efficiently maintaining the structural metadata. The punchline will be that careful application of invariants from our schema-building gives us an elegant approach to maintaining this metadata. Introducing the schema Our document stream may contain documents that vary wildly in structure. For example, say the collection we want to emit as Parquet has these 4 documents {e: 5} {a: “s”, b: {c: “t”}} {} {a: 8, b: {c: “r”, d: true}} To put these 4 documents in the same file, parquet requires they all be described by the same schema . For our purposes we can understand a schema as a sort of meta-document: a document with key-value pairs, but with special meta-values. Concretely, here are the individual schemas for those four documents: The nested sub-document under b in the second and fourth documents illustrate how a schema for a document is recursively defined. Design: The schema illustrations above are quite close to our in-memory representation. Our schema is a tree: the blue nodes represent the names , mapping closely to the typical b.c way of referring to a field in a (nested) document. The square nodes maintain what types we’ve seen under that key-path. The schema of multiple documents is naturally defined by the union (or merge ) of each document’s individual schema. So the schema for our four-document collection is: A few highlights: Note that we don’t preserve the “frequency” of keys. Our schema doesn’t distinguish whether a field appears in every document, or just some documents. Merging schemas is commutative; we’ll always get the same schema no matter the order we visit the documents. The schema information is enough for us to tell the parquet file what columns it will need, so we can start laying it out. Roughly, each leaf (representing a scalar value in our schema, such a string or integer) will map to a column in the parquet file. Typically we’d name each column based on the names we see on the path from the root to that leaf. For instance, in this tree we’d have b.c and b.d as two column names. The atypical case is in supporting polymorphism. See how in our example the name a refers to two different parquet columns: one of type int32 , and another as type string. In these cases we disambiguate the columns by appending their type-name at the end. So the 5 columns from this schema will be named: a.int32 , a.string , b.c , b.d , e . 2 Code to build the schema We know we need our schema, and we know what it should look like. All that’s left is writing code to actually compute the schema. We can compute it inductively: Start with an empty schema On the next document, walk that document’s tree-structure and the schema-tree in lockstep. If there is ever a node in our document that is missing from our schema-tree, add that node. Repeat step-2 for the next document until done. At the end of each step-2, the schema we have built is the schema for all the documents-seen-so-far. Here is some pseudocode describing step-2 in more depth: The first parameter of each function is the document we’re walking, and the second parameter is the current schema-tree-node we’re in. The visitType function refers to the square nodes in our tree, and the visitName function to the round nodes. Note how on line 6 we are recursing into the individual key-value pairs of our document: that’s our actual tree-walk. To illustrate its implementation, here is how we’d iteratively build the schema over our four documents. Each image is the schema after step-2. We yellow-highlight the nodes to indicate those nodes we allocate, or visit, as we traversed the latest document. Note that this walk is driven by the document-size: even if our schema is ultimately very big, the time it takes to process a document is only dependent on the size of that document . When we shred document 3, we only visit the root, for instance. Hooray! This document-driven approach was our first step towards performance improvements in parquet-writing. Previously we had a more schema-driven approach that used more allocations and node-traversals. We’re ready to extend this foundation and shred our documents in the same pass. Before we jump in, let’s conclude with one crucial invariant that is apparent as a result of how we build our schema: Online schema invariant Once we place a leaf node in our tree, the path to that leaf never changes. In other words, we only ever place new leaves; we never insert a node “above” an existing one. We will rely on this invariant heavily to show that we’re correctly maintaining the structural metadata in the course of shredding. Introducing shredding We can now rest assured that, when the time comes to create a parquet file, we will have the right schema-information to tell it what columns to create. Of course, now we need to get the actual data to fill in those columns. Recall one of the key challenges we anticipated was maintaining the metadata. This metadata is what parquet needs to remember the structure of each document. These are called definition levels and repetition levels . Each column in parquet, in addition to its data-actual column, can have two more columns of this additional metadata. 3 Before exploring how our shredding builds and maintains this information, let’s take a brief detour to build an intuition about how this metadata works. Using definition and repetition levels Say we’ve naively shredded our 4-document stream without regard to any metadata. For visualization, we’ll attach the column-contents to each leaf in our schema. So after shredding our data looks like: This presents a challenge. Say one day we want to re-assemble our documents. What would our documents look like? We might very well guess the correct sequence, but absent additional metadata, we might as well reassemble this data into something like: {a: 5, b: {c: “s”, d: true}, e: 8} {a: r, b: {“t”}} Plainly these are very different from our original documents. That’s not good! So we need a way to disambiguate our shredded values; repetition and definition levels enable this in parquet. The full explanation can be found in the Dremel paper 4 (see also this very clear 2013 Twitter blog post 5 ). For brevity, here we’re just going to be looking at definition levels. Repetition levels are different of course, but similar enough where the mechanisms we’ll ultimately highlight will work the same way. We elide discussing repetition levels for brevity. Definition Levels: The i-th definition level in a column indicates how many names along the path to that column were present for document i 6 . This means as we process a document, we promise to write some definition-level for each leaf 7 . To introduce this idea, consider the leaf b.d. After processing all documents, it will have the definition levels "0,1,0,2". Here is how these b.d definition-levels are derived as we process our four documents. Now we can consider how our whole tree looks with all the definition levels. This picture tries to demonstrate a lot of the impact of definition-levels. Some highlights: Consider column e : It has definition levels 1, 0, 0, and 0. The 1 indicates that the first document ({e:5}) has 1 name defined along the path to e . Said another way, that first 1 indicates that the value in the column-store, 5 , is present in the first document. As e does not appear in any other documents, it has 0 for the rest of its definition-levels. We discussed column b.d , with def-levels 0, 1, 0, 2 above. We can compare it to its sibling, b.c . In particular, this highlights how the number of values a column has is exactly equal to the number of times the max-definition-level appears. The definition-level stream for b.d only has the max-definition-level, 2, once. It only has one value: true . Its sibling, however, has the max-definition-level, 2, twice, and correspondingly has two values. Columns a.string and a.int32 show that even though the name a appears in two document streams, we have to disambiguate the underlying data-type. This is saying that the second document contains a as a string, and the fourth document contains a as an int32. This is a detail that reveals some of the extra work we take to translate polymorphic documents into parquet-amenable formats. Hopefully this suggests a reassembly-into-documents algorithm: for the i-th document, we look at the i-th def-level of each column. When it’s equal to the number-of-names on our path, we pop off the next value in the value-stream associated with that column. There are more complications and subtleties in practice, but this captures the spirit of it. Code to shred the document stream We’ve covered how definition levels are used to reassemble the original documents. Now we can turn to our task of actually computing the definition levels as we shred our documents. Two challenges present themselves: We want to emit the correct definition-level as we process each document, but that definition level implicitly depends on the structure of the schema (it’s “how many names were on that path”). We’re building our schema on-line—how can we be confident that the definition level we write shouldn’t have been different as the schema grows to accommodate more documents? If the 10th document introduces some column named z , then we clearly haven’t been adding any definition levels to that column for the past 9 documents—we didn’t know the column existed yet! How might we “backfill” the required def-levels? It’s not enough to simply fill them with 0s, because (recall our 2nd example document, the b.d path) sometimes we need an intermediate def-value. The first challenge is easily addressed: recall the crucial Online Schema Invariant, that a path to a leaf never changes. A consequence of that invariant is that we can be confident in the definition-levels for any leaf. Its depth (i.e., the number of names to that leaf) is not going to change. Hooray! Hopefully as-presented this punchline is very clear and obvious to you; during the course of development, however, this result took some careful reasoning to establish. It is the second challenge that requires even more thought. Ultimately we found we could leverage the “lazy syncing” of def-levels the Dremel paper introduces. 8 We were already interested in using it to keep our per-document walk proportional to the document size, but it was a happy moment to see it could help us in the new-column case. Lazy syncing is mechanically simple but conceptually subtle, let’s dive in. Lazy syncing To set the stage for lazy syncing, we extend our definition-streams to interior (type-)nodes. In our example, that means our final shredding would look more like: See how the two document nodes now have definition-levels. The definition levels express the same idea: how many names did we see along this path for the i-th document. Let’s walk through our four-document example one more time, now shredding in addition to building the schema. Shredding document 1 This is the simple case. This is our first-ever document, so we know there’s no def-levels to “backfill”, and no nodes we’re trying to skip. We highlight the nodes where we’re adding definition-levels. Shredding document 2 Now things are getting interesting. We don’t highlight the e column because our algorithm doesn’t touch it for document 2. We do, however, inevitably touch one of its ancestors (in this case, the root). Syncing: When we create the a , b , c columns, how did we know to backfill the definition levels for the first document all with 0? This is because as we write the def-level of the current document, we first always check to make sure we’re in sync with our parent for any prior documents, and so on transitively. To narrate an example of syncing: When we created the a column, our code basically said “and your second document def-level is 1”. You might imagine the a column panicking–it never heard what its first document def-level is! What might you do when you panic? You might talk to your parent: in this case a ’s parent is the root, and a asks “what did you see for the first document?” — the root says 0, and a can trust that 0 is the deepest the first document ever got to a . For a more complicated case, see what happens with b and b.c . In that case, they both were written-to for the first time in this document. They were both initially skipped, so they both learn 0 from their parent (transitively), but of course b.c was written-to in this document, so it stops syncing at that latest document and writes a 2 instead of the 1 that b writes. We only sync a column when we write to it. In particular, see how we don’t sync the def-levels for e . This subtlety is key to enabling us to shred documents efficiently: when we shred a document, we only have to traverse the nodes in that document, not the (potentially much larger) number of nodes in the whole schema. This is syncing in-action. Let’s see a few more examples. Shredding document 3 This is a sort-of easy case: we hardly write anything. Instead of (for now) having to walk the whole schema, we’re only touching the root node—after all, we don’t have much more information to write for the empty-document. Implicitly, we’re relying on the syncing machinery to ultimately propagate this information as required. Shredding document 4 We finally ingest our last document. See in particular how the new-to-us column of b.d is able to sync the previous 3 def-levels from its parent, correctly learning the “1” for the second row. Almost without noticing, our syncing machinery gives us the ability to handle entirely-new columns: we simply create the new column, and then sync with its existing parent. The only thing we need to rely on is that we have some global ancestor that sees every document: but of course, that’s exactly the root, as we saw with Document 3. Finally, note also that this doesn’t quite look like the final form we expected: we ultimately do need 4 def-levels on every leaf. One last “global-sync”, where we propagate the defs down to the leaves until everyone’s caught up, finalizes our shredder-state into what we expect: Phew, the pseudocode Congratulations on getting through all that shredding. Hopefully this has been a thorough illustration of how this one key technology, syncing, enables us to extend our shredder to handle new columns as they appear online. While conceptually nontrivial, the (psuedo)-code extends easily from our schema-building example. To drive that home, the code-diffs are highlighted in magenta. It’s not that big of a diff to get full shredding! Concluding the implementation We are now able to take a stream of documents and shred those documents into an in-memory columnar format complete with their structural metadata. We only ever need to look at each document once for this process, leading to a substantive performance improvement. What we have not explored, and won’t in this blog post, are all the extra details we need to get this functional: swapping our in-memory representation to disk as-needed to bound our memory usage, actually emitting our in-memory representation into the physical Parquet file (thanks to the apache-go library), and more. Getting the new approach in production While the shredder algorithm is a fascinating component, it is just one part of the larger process of delivering product improvements. Any large deliverable involves many people, and anything that impacts our users calls for meaningful validation. We’ll conclude our discussion of shredding by placing its work and impact in this larger context. Build together This work benefited greatly from a number of collaborators. We were fortunate to be able to draw on a diverse range of experts and collaborators for this work, and doubly-so because so many of us were able to learn from each other along the way. A distinguished engineer pushed for this project, drawing her analytics expertise to show how we could have a much more column-first perspective on parquet emission, ultimately leading us to adopt the official apache arrow project as our parquet-file-writing library, interfaced with via the above-described shredder. She also outlined a lot of the key requirements for our swap-to-disk procedures, in addition to teaching me and other colleagues an enormous amount about the analytics world. At the cross-team level, we sought out performance-measurement experts to help characterize what performance success should look like at a systems level. At the individual collaboration level, I was glad for the privilege of introducing several junior colleagues to performance work and how to get effective measurements. Their contributions included many other technical deliverables that substantially improved our product’s performance, and insightful ways of measuring performance. Validating the behavior Finally, for all the technical and collaborative work we’ve discussed, the value of the work is predicated on us getting this into production. The popularity and importance of our parquet-writing motivated this work and made us excited to release it as soon as possible. By the same token, we needed to release this without compromising stability or correctness. That’s part of being a developer data platform . Along with table-stakes validation like targeted unit-tests validating the core machinery, integration tests validating how that machinery is used by the larger product, and end-to-end tests validating the, well, end-to-end experience, we sought out synthetic and real-world datasets to make sure we handled input of all shapes and sizes. We also strived to validate our project from a customer’s perspective: people emit parquet files with the intention of using them elsewhere, so we extensively tested our produced parquet files with third-party readers to assure their compatibility. Ultimately, the performance improvement brought on by this work is considerable. For our largest customers, our logging metrics indicate we’ve over doubled our parquet-writing throughput. This is consistent with our customer-oriented internal benchmarks, and the effect is that we are now saving many compute-days per day servicing our customer requests. We believe this work has laid a strong foundation for our parquet-emission support. Thanks for reading! 1 For those familiar with the parquet-specific term, we are able to flush on each rowgroup, not just the whole file–but the issue and challenge remains the same. We write “file” here for simplicity. 2 Note that a typename, such as “.int32” or “.string”, will only appear at the end of a path. 3 Not all columns require these streams, here we’re referring to the general case. For more details, see the Dremel paper, cited below. 4 https://dl.acm.org/doi/abs/10.14778/1920841.1920886 5 https://blog.twitter.com/engineering/en_us/a/2013/dremel-made-simple-with-parquet 6 For those already familiar with parquet: technically the definition level range is only for names-that-could-possibly-be-missing, i.e., excluding fields marked REQUIRED. For our discussion and simplicity, we’re assuming all columns are OPTIONAL. 7 Things get a little more complicated when also considering repetition levels (multiple definition levels might map to one document), but again for brevity we do not discuss repetition levels. 8 Section 4.2, and Appendix A, of the Dremel paper.

April 11, 2023
Next →

Exploring New Security, Billing, and Customization Features in Atlas Charts

MongoDB is excited to announce a few new updates to Atlas Charts that enable you to securely share insights, gain deeper visibility into expenses, and customize your most frequently used data visualizations. Based on specific feedback received from users of our native visualization tool, these significant improvements will make data analysis even more productive. We: Improved security in Atlas Charts for passcode-protected public dashboards Increased visibility into Atlas spending through an updated billing dashboard Introduced new customization for table charts through hyperlinks and hidden columns Secure insights with passcode-protected public dashboards First, there’s the new passcode-protected public dashboards feature that brings an extra layer of security to publicly shared dashboards—we understand that not everyone who benefits from Atlas Charts operates within MongoDB Atlas. Alongside the ability to schedule email reports and support for publicly-shared dashboards , we’re offering a new and secure way to spread insights with the launch of our latest feature. Add an extra layer of security to previously publicly shared dashboards, ensuring that only authorized users with the passcode can access your data. Enabling passcode protection on a dashboard is simple. As a dashboard owner, a new option is available to protect dashboard links with a passcode when sharing it publicly. Check the box to protect your public link with a passcode Once enabled, a passcode is automatically generated and can be copied to the clipboard (and regenerated on demand as needed). Viewers navigating to dashboards via the public link will see a new screen prompting them to enter a passcode. Once authenticated successfully, they can view the dashboard just as before. Easily access your dashboards by inputting your password when prompted Whether you're sharing insights with clients, stakeholders, or team members, rest assured that your data remains easily accessible yet secure. To learn more about the different ways we support dashboard sharing, check out our documentation . What’s new in the Atlas Charts billing dashboard Next, we continue to make enhancements to the MongoDB Atlas Charts billing dashboard , all of which provide insights into Atlas expenses. We are delighted to share that it’s now possible to see resource tags data, as well as billing data from all linked organizations inside the Atlas Charts billing dashboard. Additionally, users can now ingest billing data from another organization, provided they possess the organization’s API keys. These newly introduced features rely on the availability of billing data within the organization. And for those leveraging resource tags, the billing data will seamlessly integrate, empowering users to generate personalized charts or to incorporate tailored dashboard filters within the Atlas Charts billing dashboard. If cross-organization billing is enabled, editing the configuration will ingest the linked organization’s billing data for the last three months, with the option to extend this period to up to a year by creating a new ingestion. Project tags data in the Atlas Charts billing dashboard Resource tags are now seamlessly integrated into billing data and can be included in any of the charts or the dashboard filters inside the Atlas billing dashboard. For example, our MongoDB organization uses the Atlas auto-suggested tags “application” and “environment,” alongside a custom resource tag labeled "team." The following chart uses the tags data and shows the billing cost per team and per environment. A chart which depicts cost per team and environment using tags The subsequent chart presents the billing cost allocated per project and team, providing valuable insights into the primary cost drivers for each team's projects. A chart depicting cost allocated per project and team Users can also add a dashboard filter to the “tags” field, which will allow them to see the whole dashboard based on the selected tag values. In the next example, we have selected a specific “team” : “Charts” from the tags dashboard filter, so we can see all of the billing insights per team thanks to our custom tag. Billing insights filtered by specific "charts" team in an intuitive dashboard Linked organization’s data in the Atlas Charts billing dashboard For complex Atlas projects spanning multiple organizations, the Atlas Charts billing dashboard now seamlessly integrates billing data from all linked organizations. The most productive use case is to add a dashboard filter based on the "organizationId" to enable filtering data according to specific organizations for a more granular analysis of the spending. Dashboard filtered by the organizationID field to show insights for one organization Billing data from another organization Users can now ingest billing data from other organizations that are not directly linked, provided they possess authorization API keys, bringing the data you need to where you are. Provide the API key to ingest billing data from other organizations These new features in the Atlas Charts billing dashboard are designed to provide richer, more detailed insights into organization spend. Check out our documentation and our previous blog post to learn more about it. Hyperlinks and hidden columns for tables in Atlas Charts Of all the data visualization methods available in Atlas Charts, table charts rank as one of the most popular among our users. So it should come as no surprise that one of the most highly requested features from our customers is the ability to format columnar data as hyperlinks. We're excited to announce that this is now possible in Atlas Charts through the new hyperlink customization options available for table charts . With hyperlink customization, you can format columnar data as hyperlinks using any of the following URI protocols: http, https, mailto, or tel, and can be constructed statically or dynamically using encoded fields. Let’s assume we’ve created a table using the sample movies dataset in Atlas, with encodings like title, imdb.id, runtime, genre, poster_display—which is a calculated field —and more. Customization panel in Atlas Charts To turn movie titles into clickable links that direct users to their respective IMDB pages, navigate to the customization panel and click into the hyperlinking feature in the fields tab . We will format the title field as a hyperlink which links to the Internet Movie Database (IMDB) entry for that movie. IMDB URLs are formatted as follows, where id needs to be substituted with the value of the imdb.id field for each document. https://www.imdb.com/title/tt<id>/ Customize the “title” field in the table chart to link to IMDB using the “imdb.id” field in the URI input. Below, a preview displays the fully formatted URI with fields substituted for their values, helping to ensure it’s correct before we save it to be applied to the chart. Preview of URI in the hyperlinking panel Since we only need the imdb.id field to be encoded for the purpose of constructing the URI applied to the title field, we can hide the column from rendering using another new customization option. Select the imdb.id field in the customization panel, and toggle on the “Hide Column” option. Toggle "Hide Column" We also support using URI values directly from fields (provided they use one of the supported protocols). Let’s see this in action by creating a hyperlink to the movie poster. In the URI input, trigger the encoded field menu using the @ keyboard shortcut, and select the poster field. Similar to the previous example, a preview will be displayed. After saving and applying the hyperlink formatting, we can hide the rendering of the poster field as needed to keep the chart clean. Use the @ keyboard shortcut to trigger the encoded field menu All these options are accessible in the customization panel, making it straightforward to enhance table charts with interactive hyperlinks. For more detailed instructions, visit our documentation . As we conclude this roundup, we hope you’re as excited about these updates as we are. The Atlas Charts team is dedicated to continuously improving Atlas Charts to meet your needs and enhance your data visualization experience. Stay tuned for more updates, and happy charting! New to Atlas Charts? Get started today by logging into or signing up for MongoDB Atlas , deploying or selecting a cluster, and activating Charts for free.

September 5, 2024