[Django] crash when loading document with JSONField and datetime or ObjectId with it

Other MongoDB Topics
,
1

I am using the latest available version of django-mongodb-backend 5.1.0b1, and my data contains quite a few lists with objetcs in there. So it means I cannot use EmbeddedModelField, I need to use ArrayField with base_field=models.JSONField

I get json dump errors id these inner objects have something python json cannot serialize (ObjectIds but also datetimes)

This is in django_mongodb_backend.operations.DatabaseOperations.convert_jsonfield_value

on line 150.

This call uses a plain json.dumps and we cannot give it an encoder.

How come all these bson fields are not handled in there?

My embedded documents load fine if they are in EmbeddedModelField but the document says I cannot use that in ArrayField.

Anyone has faced the same problem? Any advised solution? It’s quite deep in the code, somewhere I cannot simply override.

1 Like
2

Thank you for trying django-mongodb-backend @Stephane! Can you provide any code or data or steps to reproduce the issue?

3

I think it’s a reasonable request, however, it appears non-trivial. I created INTPYTHON-551.

1 Like
4

Good suggestion. I believe this is more a bug fix request than a feature request: the connector crashes when a jsonfield (an embedded document any non-straight matched BSON to JSON datatype, like ObjectID or datetime)

Even nicer would be to have EmbeddedModelFields in ArrayField

5

You can try with this:

from django.test import TestCase
from pymongo import MongoClient
import datetime

from asmorm.models import Observation


class ObservationTestCase(TestCase):
    def test_observation(self):
        Observation.objects.create()
        observation = Observation.objects.all()[0]

        self.assertIsNotNone(observation)
        client = MongoClient("mongodb://localhost/")
        db = client["test_demo"]
        db.asmorm_observation.update_one(
            {"_id": observation.pk},
            {
                "$set": {
                    "history": [
                        {"update": datetime.datetime.now(datetime.UTC), "value": "1"}
                    ]
                }
            },
        )
        observation.refresh_from_db()
        print(observation.history)

in tests.py

And this

import django_mongodb_backend
from django.db import models
from django_mongodb_backend.fields import ArrayField


class Observation(models.Model):
    _if = django_mongodb_backend.fields.ObjectIdAutoField(primary_key=True)
    history = ArrayField(base_field=models.JSONField(), default=list)

in models.py

6

Yes,

I’ll make a small project in a minute but here is an example with my real data:

for o in models.Observation.objects.exclude(history__isnull=True)[:100]:
            print(o)

Yield this error:

  raise TypeError(f'Object of type {o.__class__.__name__} '

                    f'is not JSON serializable')

E TypeError: Object of type datetime is not JSON serializable

Here:

…/…/.venv/lib64/python3.13/site-packages/django_mongodb_backend/operations.py:150: in convert_jsonfield_value
return json.dumps(value)

The history field is defined as:

history = ArrayField(
        base_field=models.JSONField(encoder=DjangoJSONEncoder), null=True, blank=True
    )

And the data is:

{
  "_id": "website_status_code_https_promotions_usa_gov",
  "history": [
    {
      "updated": {
        "$date": "2021-12-20T14:04:17.071Z"
      },
      "value": "200"
    },
    {
      "updated": {
        "$date": "2021-12-20T14:04:17.071Z"
      },
      "value": "403"
    },
    {
      "updated": {
        "$date": "2023-03-26T02:37:23.456Z"
      },
      "value": "200",
      "priority": {
        "status_code": 0
      },
      "calculated_priority": 0,
      "score": 1
    },
    {
      "updated": {
        "$date": "2023-08-04T18:47:40.545Z"
      },
      "value": "403",
      "priority": {
        "status_code": 0.32030000000000003
      },
      "calculated_priority": 0.32030000000000003,
      "score": 0.8
    }
  ]
}
7

Support for an array of embedded documents is forthcoming.