"Element 'Id' does not match any field or property of class" when using Project in Facet stage

Mateusz_Wroblewski · November 16, 2022, 5:47pm

Hello,

I have a problem with the Project step in my Facet stage which maps Entity object into DTO:

            var dataFacet = AggregateFacet.Create("data",
                PipelineDefinition<BankTransaction, BankTransactionDto>.Create(new IPipelineStageDefinition[]
                {
                    PipelineStageDefinitionBuilder.Skip<BankTransaction>((pageNumber - 1) * pageSize),
                    PipelineStageDefinitionBuilder.Limit<BankTransaction>(pageSize),
                    PipelineStageDefinitionBuilder.Project<BankTransaction, BankTransactionDto>(x => new BankTransactionDto
                    {
                        Id = x.Id,
                        TransactionDate = x.TransactionDate,
                        Customer = x.Customer,
                        Description = x.Description,
                        Value = x.Value,
                        CategoryId = x.CategoryId,
                        CategoryName = x.Category.Name
                    }),
                }));

When I run it, it throws an error “Element ‘Id’ does not match any field or property of class”. Here entire stack:

System.FormatException: Element 'Id' does not match any field or property of class SaveMeter.Modules.Transactions.Core.DTO.BankTransactionDto.
   at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.DeserializeClass(BsonDeserializationContext context)
   at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
   at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Deserialize[TValue](IBsonSerializer`1 serializer, BsonDeserializationContext context)
   at MongoDB.Bson.Serialization.Serializers.EnumerableSerializerBase`2.Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
   at MongoDB.Bson.Serialization.Serializers.SerializerBase`1.MongoDB.Bson.Serialization.IBsonSerializer.Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
   at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Deserialize(IBsonSerializer serializer, BsonDeserializationContext context)
   at MongoDB.Driver.AggregateFacetResultsSerializer.Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
   at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Deserialize[TValue](IBsonSerializer`1 serializer, BsonDeserializationContext context)
   at MongoDB.Bson.Serialization.Serializers.EnumerableSerializerBase`2.Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
   at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Deserialize[TValue](IBsonSerializer`1 serializer, BsonDeserializationContext context)
   at MongoDB.Driver.Core.Operations.AggregateOperation`1.CursorDeserializer.Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
   at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Deserialize[TValue](IBsonSerializer`1 serializer, BsonDeserializationContext context)
   at MongoDB.Driver.Core.Operations.AggregateOperation`1.AggregateResultDeserializer.Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
   at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Deserialize[TValue](IBsonSerializer`1 serializer, BsonDeserializationContext context)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, ICoreSession session, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadOperationExecutor.ExecuteAsync[TResult](IRetryableReadOperation`1 operation, RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.AggregateOperation`1.ExecuteAsync(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.AggregateOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.AggregateAsync[TResult](IClientSessionHandle session, PipelineDefinition`2 pipeline, AggregateOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at MongoDB.Driver.IAsyncCursorSourceExtensions.FirstAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)
   at SaveMeter.Modules.Transactions.Core.Queries.Handlers.GetBankTransactionsByFilterHandler.HandleAsync(GetBankTransactionsByFilter query, CancellationToken cancellationToken) in /Users/mawb/learning/repos/savemeter-api-monolith/src/Modules/Transactions/SaveMeter.Modules.Transactions.Core/Queries/Handlers/GetBankTransactionsByFilterQueryHandler.cs:line 70

The interesting fact, when I rename “Id” property into “TransactionId” in DTO object, everything works fine. So I assume, that something is wrong with the property name “Id”. Also, there is no error when I remove Project step.

I’m using MongoDB.Driver 2.18.0
Example document:

{
    _id: UUID("8edbc21e5a214672b4f04a91bac5e0c9"),
    _t: [
        'Entity',
        'BankTransaction'
    ],
    CreatedAt: ISODate('2022-11-01T07:55:21.840Z'),
    UpdatedAt: ISODate('2022-11-01T07:55:21.840Z'),
    TransactionDate: ISODate('2022-03-14T00:00:00.000Z'),
    Customer: 'Example customer',
    Description: 'Example description.',
    Value: NumberDecimal('-72.99'),
    CategoryId: UUID("792badf0b21642f1a312c105d1b87051"),
    BankName: 'null',
    UserId: UUID("...")
}

This document is mapped with BsonClassMap:

        BsonClassMap.RegisterClassMap<Entity>(map =>
        {
            map.SetIsRootClass(true);
            map.MapIdMember(x => x.Id);
            map.MapMember(x => x.CreatedAt);
            map.MapMember(x => x.UpdatedAt);
        });
        BsonClassMap.RegisterClassMap<BankTransaction>(map =>
        {
            map.AutoMap();
            map.SetIgnoreExtraElements(true);
            map.MapMember(x => x.Value).SetSerializer(new DecimalSerializer(BsonType.Decimal128));
            map.GetMemberMap(x => x.Category).SetShouldSerializeMethod(_ => false);
        });

And classes used in example:

internal class BankTransaction : Entity
{
    public DateTime TransactionDate { get; set; }
    public string Customer { get; set; }
    public string Description { get; set; }
    public decimal Value { get; set; }
    public Guid? CategoryId { get; set; }
    public Category Category { get; set; }
    public string BankName { get; set; }
    public Guid UserId { get; set; }
}

public record BankTransactionDto
{
    public Guid Id { get; init; }
    public DateTime TransactionDate { get; init; }
    public string Customer { get; init; }
    public string Description { get; init; }
    public decimal Value { get; init; }
    public Guid? CategoryId { get; init; }
    public string CategoryName { get; init; }
}

I forgot to mention that I’m using lookup operation to join Category, but I tested it without lookup and had the same error.

Dmitry_Lukyanov · November 16, 2022, 6:00pm

Hey @Mateusz_Wroblewski , what driver version do you use? Also, please provide an example of document you use

Yilmaz_Durmaz · November 16, 2022, 6:10pm

I had a similar issue and a similar solution. I think the driver tries to map the _id field of the actual document on the database to the Id property of the class. Even with annotation to link it to a different field in the document, this happens.

Try using it as x["_id"] and see if it helps.

My class has this basic structure:

 class Model{
      [BsonId]
      [BsonRepresentation(BsonType.ObjectId)]
      public string? Id { get; set; }   // this maps to `_id` in the document
      [BsonElement("id")]
      public int SId { get; private set; }  // had to give up naming this as "Id"
      [BsonElement("class")]
      public string Class { get; private set; } = null!;
}

Mateusz_Wroblewski · November 16, 2022, 6:53pm

Hi @Dmitry_Lukyanov, I have updated the post.

Mateusz_Wroblewski · November 16, 2022, 7:03pm

Thanks @Yilmaz_Durmaz, but I think it’s not the same case as you mentioned. In my example, BankTransactionDto does not have any MongoDB annotations or BsonClassMap configuration.

Yilmaz_Durmaz · November 16, 2022, 9:26pm

this seems to be the problem. your BankTransaction class does not have an Id property, thus this assignment and/or mapping fails.

I must stress that “_id” field in stored documents is mapped to “Id” of classes. this mapping operation will just take it and assign to another name. you can also do this with annotations [BsonId][BsonRepresentation(BsonType.ObjectId)] on a property. so the name does not have to be Id, but you need to use whatever name you choose.

You don’t have to have Id property (or a property mapped to _id) at all when all you do is read data and this field is not relevant to you. but once you try to use it as the above two lines does, it becomes apparent that you need it defined.

Mateusz_Wroblewski · November 16, 2022, 10:10pm

It is not obvious, but BankTransaction class does have Id property and inherits it from Entity class

    public abstract class Entity
    {
        public Guid Id { get; set; }
        public DateTime CreatedAt { get; set; }
        public DateTime UpdatedAt { get; set; }
    }

For me it’s not a problem with BankTransaction class, but with BankTransactionDto because when I change the Project operation to this, everything works.

PipelineStageDefinitionBuilder.Project<BankTransaction, BankTransactionDto>(x => new BankTransactionDto
  {
      TransactionId = x.Id,
      TransactionDate = x.TransactionDate,
      Customer = x.Customer,
      Description = x.Description,
      Value = x.Value,
      CategoryId = x.CategoryId,
      CategoryName = x.Category.Name
  }),

So it’s strange because when I have “Id” property in BankTransactionDto, projection fails, but when I have “TransactionId” in BankTransactionDto there is no error.

Yilmaz_Durmaz · November 17, 2022, 12:52am

I could finally have a working minimal code to get the issue and the fix you mention.

I have also found a StackOverflow post about the issue dating back 2 years: c# - Mongo .Net Driver PipelineStageDefinitionBuilder.Project automatically ignores all Id values with a facet - Stack Overflow

PipelineStageDefinitionBuilder.Project fails to map Id when used inside Facet. I have checked this forum and the Jira pages of the driver. it seems no one has reported this issue yet. (Note I used the keyword “issue”)

I will add my minimal working code here to work with it later. It has lots of commented-out code but two parts are important: adding a few test data and renaming xId to Id to get the error.

Click here for the code to add and query data

// DTO
{
    MongoClient dbClient = new MongoClient(URI);

    var database = dbClient.GetDatabase("testme");
    var collection = database.GetCollection<BankTransaction>("banktransaction");

    // create a few transactions
    // var transaction1 = new BankTransaction()
    // {
    //     Id = 2,
    //     CreatedAt = "2022-11-01T07:55:21.840Z",
    //     UpdatedAt = "2022-11-01T07:55:21.840Z",
    //     Customer = "Example customer",
    //     BankName = "helloBank",
    //     UserId = 12345
    // };
    // collection.InsertOne(transaction1);
    // var transaction2 = new BankTransaction()
    // {
    //     Id = 3,
    //     CreatedAt = "2022-11-01T07:55:21.840Z",
    //     UpdatedAt = "2022-11-01T07:55:21.840Z",
    //     Customer = "elder customer",
    //     BankName = "Bankhello",
    //     UserId = 54321
    // };
    // collection.InsertOne(transaction2);
    // read transactions
    Console.WriteLine(collection.Find<BankTransaction>(_ => true).ToList().ToJson());

    // set class mapping
    // BsonClassMap.RegisterClassMap<Entity>(map =>
    // {
    //     map.SetIsRootClass(true);
    //     map.MapIdMember(x => x.Id);
    //     map.MapMember(x => x.CreatedAt);
    //     map.MapMember(x => x.UpdatedAt);
    // });
    Console.WriteLine("\nmap?\n");
    // BsonClassMap.RegisterClassMap<BankTransaction>(map =>
    // {
    //     map.AutoMap();
    //     map.SetIgnoreExtraElements(true);
    // });

    var dataFacet = AggregateFacet.Create("bankdata",
    PipelineDefinition<BankTransaction, BankTransactionDto>.Create(new IPipelineStageDefinition[]
    {
        PipelineStageDefinitionBuilder.Skip<BankTransaction>(1),
        PipelineStageDefinitionBuilder.Limit<BankTransaction>(10),
        PipelineStageDefinitionBuilder.Project<BankTransaction, BankTransactionDto>(x => new BankTransactionDto
        {
            xId = x.Id,
            Customer = x.Customer,
            CustomerId=x.UserId
        }),
    }));

    var aggregation = collection.Aggregate().Match(_ => true).Facet(dataFacet);
    var result = aggregation.Single().Facets.ToJson();
    Console.WriteLine(result);

}

Click here for Required Classes


internal class BankTransaction : Entity
{
    public string Customer { get; set; }
    public string BankName { get; set; }
    public int UserId { get; set; }
}

public record BankTransactionDto
{
    public int Id { get; init; }
    public int xId { get; init; }
    public string Customer { get; init; }
    public int CustomerId { get; init; }
}

public abstract class Entity
{
    public int Id { get; set; }
    public string CreatedAt { get; set; }
    public string UpdatedAt { get; set; }
}

Here is the output for this program after the facet stage:

[{ "_t" : "AggregateFacetResult`1", "Name" : "bankdata", 
   "Output" : [
             { "_id" : 0, "xId" : 2, "Customer" : "Example customer", "CustomerId" : 12345 },
             { "_id" : 0, "xId" : 3, "Customer" : "elder customer", "CustomerId" : 54321 }
   ]
}]

I think we should open a new post something like “PipelineStageDefinitionBuilder.Project fails to map Id in Facet” to get attention and/or open an issue on Jira.

PS: I hope I am not exagerating this

PS again: did you notice Id is mapped to _id!? but it is basically empty!

Mateusz_Wroblewski · November 17, 2022, 9:29pm

Thanks @Yilmaz_Durmaz for preparing minimal code. I made some tests on my code and I reproduced situation where Id with Guid type was empty (all zeros). It would be nice to get more attention to this issue .

Yilmaz_Durmaz · November 20, 2022, 6:09pm

Although we have a working buggy code, it is possible we are trying to use the Facet stage wrong. It is easy when working with Javascript and JSON/BSON format, but making classes and moving data around in classes is not.

Let’s forget about class DTO for a while and try going on with easy way. How exactly would you want the result to look like? There is a nice feature in Atlas web interface (I guess also in Compass, haven’t tried yet) that lets you build an aggregation pipeline with json format, and export to C# syntax with BSON methods if you want. Have you ever tried that before?

Connect to Atlas and open your deployment page. in the menu to the left, select Deployment->Database. Open “Browse Collection”, select the collection you work on, and “Aggregation” in the middle panel. Add stages to the pipeline, watch the result and check the data. does the result you get conforms to what you want in your program?

This one is an approximation to your query:

[{
    $facet: {
        data: [{
            $project: {
                _id: 1,
                Customer: 1,
                Description: 1
            }
        }]
    }
}]

Mateusz_Wroblewski · November 23, 2022, 6:22pm

@Yilmaz_Durmaz Yes I tried to create a facet in MongoDB Compass and it worked.

Yilmaz_Durmaz · November 24, 2022, 12:20am

I had some more trial/error work on this. no solution other than using another name instead of “Id”

I have opened a new post with a bit more description of the problem and a simpler/generic code sample here: "Facet" fails to map "_id" field of a projection to "Id" of class (BUG?). You may set a “watching” or “tracking” notification level on it. I have also a link here not to lose the track of it

Mateusz_Wroblewski · November 29, 2022, 5:49pm

Thanks @Yilmaz_Durmaz! I hope that someone will look at it .