Read after change stream event does not reflect change (Replica Set, RC/WC: majority)

Working with Data
, , ,
1

Hi all!

I am observing a bug in my code which, given my understanding of majority-commits, should not be there. My code reads a document right after receiving an event from a change stream. But sometimes, the field being watched is not (yet) set when reading the field immediately after the change occurred.

This happens under the following conditions:

  • 2-node Replica Set, Version 7.0.12
  • (motor) client initialized with:
    • Read Concern: majority
    • Write Concern: majority
    • Read Preference: nearest
    • Journal: true
  • Code does not use explicit client sessions

I am aware of the following:

  • I will not get Causal Consistency unless using explicit client session
  • Both nodes have to be alive in order to make any successful requests

The code is rather involved, but basically does the following:

event = await motorCollection.watch(
    pipeline=[
        {"$match": {"updateDescription.updatedFields.myField": {"$exists": True}}}
    ],
    full_document=None
).next()
fetched = await motorCollection.aggregate(
    pipeline=[
        {"$match": {"_id": event[documentKey"]["_id"]}}
    ]
).next()
assert fetched.get("myField") # this sometimes fails

(myField will never be changed or deleted after it was written once)

So now I am wondering why myField is sometimes not (yet) set. As I understand it …

  • The watch only emits majority-committed changes
  • The aggregation only reads majority-committed changes
  • The aggregation happens STRICTLY AFTER the event was received

From this, I conclude that the assertion should never raise. Still, it does. Why is this?
I am not concerned with the various hazards emerging from not using sessions. I just want to understand why a majority-committed write is visible to a change stream but not visible to an aggregation happening strictly after the event was received.

To be clear, there might still be an unrelated bug in my code base. Nevertheless, I want to sharpen my understanding of MongoDB commit semantics. That said, my question basically boils down to:

Will a majority-committed write be visible globally and immediately, to all clients reading only majority-committed data?

Kind regards
Johannes