Unexpected Document Replacement on MongoDB Realm Sync

Hi everyone,

I’m facing an issue with my Flutter app using MongoDB Realm. Every time I open the realm, it replaces all documents on the server with the local ones, even if there are no changes to the documents. This behavior is causing all my triggers, which watch for insert/delete/update events on documents, to execute each time the app is opened.

Has anyone encountered this issue or have any insights on why this might be happening? I would appreciate any advice on how to prevent unnecessary document replacements.

Thank you!

Here is the code that i use to open my realm

App app = App.getById(env.realm) ?? App(AppConfiguration(env.realm));
List<SchemaObject> schema = [
        ...AppointmentsService().schema,
        ...CategoriesService().schema,
        ...ChatsService().schema,
        ...CommissionsService().schema,
        ...CompaniesService().schema,
        ...CustomersService().schema,
        ...EmployeesService().schema,
      ];
      FlexibleSyncConfiguration flexConfig = Configuration.flexibleSync(
        app.currentUser!,
        schema,
        encryptionKey: _getEncryptionKey(app.currentUser!.id),
        syncErrorHandler: (SyncError error) {
          print('SyncError');
          print(error);
        },
        clientResetHandler: RecoverOrDiscardUnsyncedChangesHandler(
          // All the following callbacks are optional
          onBeforeReset: (beforeResetRealm) {
            print('beforeResetRealm1');
            print(beforeResetRealm);
            // Executed before the client reset begins.
            // Can be used to notify the user that a reset is going
            // to happen.
          },
          onAfterRecovery: (beforeResetRealm, afterResetRealm) {
            print('beforeResetRealm2');
            print(beforeResetRealm);
            print(afterResetRealm);
            // Executed if and only if the automatic recovery has succeeded.
          },
          onAfterDiscard: (beforeResetRealm, afterResetRealm) {
            print('beforeResetRealm3');
            print(beforeResetRealm);
            print(afterResetRealm);
            // Executed if the automatic recovery has failed
            // but the discard unsynced changes fallback has completed
            // successfully.
          },
          onManualResetFallback: (clientResetError) {
            print('clientResetError');
            print(clientResetError);
            bool success = clientResetError.resetRealm();
            // Automatic reset failed. Handle the reset manually here.
            // Refer to the "Manual Client Reset Fallback" documentation
            // for more information on what you can include here.
          },
        ),
      );
      _realm = Realm(flexConfig);

And the logs:

Realm.Sync.Client.Session - Realm sync client ([realm-core-14.9.0])
Realm.Sync.Client.Session - Platform: iOS Darwin 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:34:31 PDT 2024; root:xnu-10063.122.3~3/RELEASE_ARM64_T8110 iPhone14,5
Realm.Sync.Client.Session - Connection[1] Session[1]: Binding '/var/mobile/Containers/Data/Application/E0326DF5-986A-4171-954C-C01A26156C03/Documents/mongodb-realm/app-ihxmjgm/6661f343689a992a36d1b17c/default.realm' to ''
Realm.Sync.Client.Session - Connection[1] Session[1]: client_reset_config = false, Realm exists = true 
Realm.Sync.Client.Session - Connection[1] Connecting to 'wss://ws.services.cloud.mongodb.com:443/api/client/v2.0/app/app-ihxmjgm/realm-sync'
Realm.Sync.Client.Network - Connected to endpoint '18.231.94.103:443' (from '192.168.1.146:52918')
Realm.Sync.Client.Network - ERROR: Websocket: Expected HTTP response 101 Switching Protocols, but received:
HTTP/1.1 308 Permanent Redirect

cache-control: no-cache, no-store, must-revalidate

connection: close

content-length: 0

date: Thu, 13 Jun 2024 17:43:00 GMT

location: https://us-east-1.aws.services.cloud.mongodb.com/api/client/v2.0/app/app-ihxmjgm/realm-sync

server: mdbws

strict-transport-security: max-age=31536000; includeSubdomains;

vary: Origin

x-appservices-request-id: 666b2fa49556a7695954aef9

x-frame-options: DENY

x-xgen-up-proto: HTTP/1.1




Realm.Sync.Client.Session - Connection[1] Closing the websocket with error code=WebSocket: Moved Permanently, message='Bad WebSocket response 308 permanent redirect', was_clean=true
Realm.Sync.Client.Session - Connection[1] Connecting to 'wss://ws.services.cloud.mongodb.com:443/api/client/v2.0/app/app-ihxmjgm/realm-sync'
Realm.Sync.Client.Network - Connected to endpoint '18.231.94.103:443' (from '192.168.1.146:52926')
Realm.Sync.Client.Network - ERROR: Websocket: Expected HTTP response 101 Switching Protocols, but received:
HTTP/1.1 308 Permanent Redirect

cache-control: no-cache, no-store, must-revalidate

connection: close

content-length: 0

date: Thu, 13 Jun 2024 17:43:01 GMT

location: https://us-east-1.aws.services.cloud.mongodb.com/api/client/v2.0/app/app-ihxmjgm/realm-sync

server: mdbws

strict-transport-security: max-age=31536000; includeSubdomains;

vary: Origin

x-appservices-request-id: 666b2fa59556a7695954b007

x-frame-options: DENY

x-xgen-up-proto: HTTP/1.1




Realm.Sync.Client.Session - Connection[1] Closing the websocket with error code=WebSocket: Moved Permanently, message='Bad WebSocket response 308 permanent redirect', was_clean=true
Realm.Sync.Client.Session - Connection[2] Session[2]: Binding '/var/mobile/Containers/Data/Application/E0326DF5-986A-4171-954C-C01A26156C03/Documents/mongodb-realm/app-ihxmjgm/6661f343689a992a36d1b17c/default.realm' to ''
Realm.Sync.Client.Session - Connection[2] Session[2]: client_reset_config = false, Realm exists = true 
Realm.Sync.Client.Session - Connection[2] Connecting to 'wss://us-east-1.aws.ws.services.cloud.mongodb.com:443/api/client/v2.0/app/app-ihxmjgm/realm-sync'
Realm.Sync.Client.Network - Connected to endpoint '107.20.72.25:443' (from '192.168.1.146:52929')
Realm.Sync.Client.Session - Connection[2] Session[3]: Binding '/var/mobile/Containers/Data/Application/E0326DF5-986A-4171-954C-C01A26156C03/Documents/mongodb-realm/app-ihxmjgm/6661f343689a992a36d1b17c/default.realm' to ''
Realm.Sync.Client.Session - Connection[2] Session[3]: client_reset_config = false, Realm exists = true 
Realm.Sync.Client.Session - Connection[2] Connected to app services with request id: "666b2fa6850c0b31cb2a223b"
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 15. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 15, producing client version 95 in 8 ms. 0 changesets remaining in bootstrap
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 16. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 16, producing client version 99 in 7 ms. 0 changesets remaining in bootstrap
Realm.Sync.Client.Session - Connection[2] Session[3]: Received: ERROR "Session closed (no error)" (error_code=200, is_fatal=false, error_action=Transient)
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 17. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 17, producing client version 103 in 7 ms. 0 changesets remaining in bootstrap
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 18. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 18, producing client version 107 in 7 ms. 0 changesets remaining in bootstrap
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 19. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 19, producing client version 110 in 4 ms. 0 changesets remaining in bootstrap
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 20. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 20, producing client version 113 in 4 ms. 0 changesets remaining in bootstrap
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 21. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 21, producing client version 116 in 4 ms. 0 changesets remaining in bootstrap
Realm.Sync.Client.Session - Connection[2] Session[3]: Received: ERROR "Session closed (no error)" (error_code=200, is_fatal=false, error_action=Transient)
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 22. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 22, producing client version 122 in 6 ms. 0 changesets remaining in bootstrap
Realm.Sync.Client.Session - Connection[2] Session[3]: Begin processing pending FLX bootstrap for query version 23. (changesets: 1, original total changeset size: 0)
Realm.Sync.Client.Session - Connection[2] Session[3]: Integrated 1 changesets from pending bootstrap for query version 23, producing client version 126 in 5 ms. 0 changesets remaining in bootstrap
1 Like

I am facing the same issue! Any help would be appreciated!

I’ve seen this multiple times, both as a TSE for MongoDB, and even recently for clients.

The issue you’re encountering, where your Flutter app replaces all documents on the server with the local ones each time the realm is opened, is likely related to how your realm is being synchronized and how the local and server-side states are being managed.

Possible Causes and Solutions:

  1. Flexible Sync Configuration:

    • Ensure that your Flexible Sync configuration is set up correctly. If your sync configuration is too broad or if there are issues with the query filters, the app might re-sync the entire dataset each time the realm is opened, leading to unnecessary document replacements. Double-check the sync query you are using. If the query is not filtering data correctly, it may cause the app to re-download and overwrite all data every time it connects.
    FlexibleSyncConfiguration flexConfig = Configuration.flexibleSync(
        app.currentUser!,
        schema,
        queryConfig: QueryConfig(
            syncAll: true, // Ensure this matches your use case
        ),
        // Other configurations
    );
    
  2. Local Changes Being Pushed to Server:

    • Even if no visible changes are made, if your local realm is marked as modified (due to schema mismatches, data type mismatches, etc.), it could trigger a full sync of the local data to the server. Be sure to review the schema and the types of data being used in your app to ensure they match the server-side expectations. Even a slight mismatch can cause the local realm to be seen as “dirty” and prompt a full sync.
  3. Handling of Existing Data:

    • If the app is treating every open as a fresh state, it might be discarding server-side data and replacing it with the local data. put in the logic to check whether local data is up-to-date before pushing changes to the server. Use the syncErrorHandler to handle potential issues gracefully and ensure that changes are not being pushed unnecessarily.
  4. Sync Error Handling:

    • The sync error handling configuration you have in place is correct, but you should ensure that errors related to sync are being caught and handled appropriately. Misconfiguration or unhandled errors can lead to unintended behavior like document replacements. Try adding more detailed logging to track when and why documents are being replaced. This can help in identifying whether the issue is due to errors in the sync process or incorrect data handling.
        print('SyncError: ${error.code}');
        print('Error message: ${error.message}');
        // Additional logging
    },
    
  5. App Services Triggers:

    • If your triggers are firing every time the app opens, it indicates that the app is detecting changes in the documents. This could be due to the above reasons where the app treats each open as a new state. You should consider revising the triggers on your MongoDB side to ensure they are only firing on actual data changes and not on re-syncs. This might involve adding additional logic to your triggers to differentiate between meaningful changes and sync-related activities.

Moving forward:

  1. Investigate Schema and Data Types: Ensure that the schema used in your Flutter app matches the schema on the server-side exactly.

  2. Check Sync Queries: Optimize your sync queries to ensure that only necessary data is being synchronized.

  3. Add Detailed Logging: Implement more detailed logging within your error handlers and sync process to capture when and why data is being replaced.

  4. Revise Triggers: If possible, adjust your MongoDB triggers to reduce unnecessary executions on data syncs.