How to filter out the documents based on dynamic keys before using $objectToArray?

[Tiya_Jose]

I need to group based on dynamic keys. So I’m using $objectToArray to query on dynamic keys.

But I need to filter out the documents based on a condition so as to reduce the input going to $objectToArray. Because I have millions of documents and I just want only a subset of the object fields to be fed to the $objectToArray operator.

My aim is to get better query performance by reducing the amount of data passed to $objectToArray operator

A sample format of my MongoDB schema:

[
  {
    "fields": {
      "field_1": {              /* dynamic key */
        "name": "f1",
        "first": {
          "check": true
        }
      },
      "field_2": {             /* dynamic key */
        "name": "f2",
        "second": {
          "check": true
        }
      },
      "description": "abc",
      "summary": {
        "val": "xyz"
      }
    }
  },
  {
    "fields": {
      "field_1": {           /* dynamic key */
        "name": "f1",
        "second": {
          "check": false
        }
      },
      "field_2": null,      /* dynamic key */
      "field_3": {         /* dynamic key */
        "name": "f3",
        "second": {
          "check": true
        },
        "first": {
          "check": true
        }
      },
      "description": "lmn",
      "summary": {
        "val": "abc"
      }
    }
  }
]

There are also many fields other than dynamic fields.

I need to filter out the documents before being fed to $objectToArray operator based on the following condition:

1. fields.<*dynamicKey*>.first exists
   OR
2. fields.<*dynamicKey*>.second exists

Expected output:

[
  {
    "fields": {
      "field_1": {
        "name": "f1",
        "first": {
          "check": true
        }
      },
      "field_2": {
        "name": "f2",
        "second": {
          "check": true
        }
      }
    }
  },
  {
    "fields": {
      "field_1": {
        "name": "f1",
        "second": {
          "check": false
        }
      },
      "field_3": {
        "name": "f3",
        "second": {
          "check": true
        },
        "first": {
          "check": true
        }
      }
    }
  }
]

How can I achieve this use case without changing my document structure?

db.collection.aggregate([
  {
I need a stage to filter out documents here
  },
  {
    $project: {
      data: {
        $objectToArray: "$fields"
      }
    }
  }

My aggregation query to group based on dynamic keys:

[Takis]

Hello

I think you ask something similar to this

How to group an array using reduce, without unwind+group?

Thank you for trying to help me,i think its not possible to group an array only by using reduce in mongoql. I will make a new topic,asking about the root of the problem. Thank you alot

MongoDB missing basic document(object) operators?

Thank you i was searching for that on JIRA also it would be nice we to have those,i upvoted and i am watching it,but i don’t think i can create JIRA to ask for those. if we have get we can do contains,and put using get and mergeobjects (new value(we can do it even now) or updated). I think remove is also needed,i don’t know a way to do it. $$mydoc = one document $$mykey = one key get("$$mydoc","$$mykey") //we need this to be added contains?("$$mydoc","$$mykey") //if we ha…

If the keys are uknown,the only way that i know is javascript and $function operator.

Upvote that Jira(see second link) if you want also,but i think that jira is only for get($$doc,$$key),i tihnk we need remove also at least

[Lauren_Schaefer]

I played with this for a bit and wasn’t able to figure out a way to do the filtering before doing $objectToArray. In case it’s helpful, here is a pipeline I created that does the filtering after $objectToArray.

[{$project: {
  fields: {$objectToArray: "$fields"}
}}, {$addFields: {
  "shouldInclude": {
    $map: {
      input: "$fields",
      as: "myFields",
      in: { $or: [
        { $ifNull: ["$$myFields.v.first", false] },
        { $ifNull: ["$$myFields.v.second", false] }
        ]
      }
    }
  }
}}, {$addFields: {
  shouldInclude: {$anyElementTrue: ["$shouldInclude"]}
}}, {$match: {
  shouldInclude: true
}}]

[Naomi_Pentrel]

Given that you say specifically you want to do that without changing the document structure I assume you know that this data modeling is probably not ideal. Changing the data modeling will likely improve performance. But without changing the data model, I think the below solution works. Note that it just checks if there is a field called second not that the field value of second.check is “true” - I think that is what you wanted but am not sure… If that was not what you wanted it can be easily changed though.

Note that you need to be using MongoDB 4.4 for this to work because it uses custom aggregation expressions

db.collection.aggregate(
    [{
      $addFields: {
        has_first_or_second: {
          $function: {
            body: function(fields) {
              for (var f in fields) {
                if (Object.prototype.hasOwnProperty.call(fields[f], 'first')) {
                  return true;
                }
                if (Object.prototype.hasOwnProperty.call(fields[f], 'second')) {
                  return true;
                }
              }
              return false;
            },
            args: ["$fields"],
            lang: "js"
          }
        }
      }
    }, 
    {
      $match: {has_first_or_second: true}
    },
    {
      $project: {has_first_or_second: 0}
    }
    ])