Atlas Device Sync Protocol
On this page
- Overview
- Key Concepts
- Changeset
- Operational Transformation
- Client File Identifier
- Network Security
- Sync Session Process
- Client Connects to the App Server
- Client Initiates a Sync Session
- App Services Allocates a New Client File Identifier
- Client Sends a Client Identifier
- Client Uploads & Downloads Sync Changesets
- Client Terminates the Sync Session
- Request Types
- Client -> Server Messages
- Server -> Client Messages
Overview
Atlas Device Sync uses a protocol to correctly and efficiently sync data changes in real time across multiple clients that each maintain their own local Realm files. The protocol defines a set of pre-defined request types as well as a process by which a client, like a Realm SDK, can connect to an Atlas App Services application server and sync data.
Note
The Realm SDKs internally implement and manage the sync protocol, so for most applications you don't need to understand the sync protocol to use Device Sync. This page covers the protocol at a high level and is not an implementation spec.
Key Concepts
Changeset
A changeset is a list of instructions that describe granular modifications made to a known object state or version by one or more write operations. Changesets are the base unit of the sync protocol. Synced realm clients send changesets to the Device Sync server whenever they perform a write operation. The server sends each connected client the changesets for write operations executed by other clients.
The Device Sync server accepts changesets from any connected sync client (including changes in a synced MongoDB cluster) at any time and uses an operational transformation algorithm to serialize changes into a linear order and resolve conflicting changesets before sending them to connected clients.
Note
Delta Sync
When you make a change to a synced object, App Services does not re-upload the entire object. Instead, App Services sends only the difference ("delta") between before and after. The service compresses the deltas with zlib compression. This reduces network load, which is especially useful in mobile network conditions.
Operational Transformation
An operational transformation is a function that, given two changesets, produces a third changeset that represents logically applying one of the given changesets after the other. Device Sync uses operational transformation to resolve conflicts between changesets from different sync clients that apply to the same base state.
Realm is an offline-first local database even when sync is enabled, which means that any device may perform offline writes and upload the corresponding changesets later when network connectivity is re-established. The operational transformation algorithm is designed to gracefully handle changesets that arrive "out of order" with respect to the logical server clock such that every synced Realm file converges to the same version of each changed object.
Tip
An operational transformation on Realm changesets is analogous to a rebase operation in Git.
Client File Identifier
A client file identifier is a value that uniquely identifies a synced client Realm file and its corresponding server file. The server generates a client file identifier whenever an SDK requests one during its initial sync of a Realm file. Each identifier is a 64-bit, non-zero, positive signed integer strictly less than 2^63.
Note
The server guarantees that all identifiers generated on behalf of a particular server file are unique with respect to each other. The server is free to generate identical identifiers for two client files if they are associated with different server files.
Network Security
The SDK synchronizes with the application server over a WebSocket connection secured by HTTPS using TLS 1.3.
Sync Session Process
To initiate, execute, and terminate a Device Sync session, a Realm SDK and application server send and receive a set of protocol-specific requests.
The SDK negotiates a WebSocket connection over HTTP and then establishes a sync
session by sending BIND and IDENT
requests to the server over the WebSocket connection.
Once the session is established, the SDK and server send synced changesets for a given
Realm file to each other via UPLOAD and
DOWNLOAD messages. To end the session, the SDK sends an
UNBIND request.
Realm SDK App Server | | | <---- 1. HTTP Handshake -----> | | | | --------- 2. BIND -----------> | | | | <-- 3. IDENT (first time) ---- | | | | --------- 4. IDENT ----------> | | | | <---- 5. UPLOAD/DOWNLOAD ----> | | | | --------- 6. UNBIND ---------> |
Client Connects to the App Server
The sync protocol is primarily handled over a WebSocket connection between the SDK and the server. To establish a connection, the SDK sends a handshake HTTP request that includes the following:
a protocol version
a WebSocket key
a valid access token for an authenticated App Services application user
The server sends an HTTP 101 Switching Protocols response that specifies a WebSocket connection for the SDK. The rest of the sync protocol occurs over this connection.
Client Initiates a Sync Session
To begin a sync session, a Realm SDK sends a BIND
request to a Device Sync server. The request identifies a
specific local Realm Database file to sync and includes a WebSocket connection key that
the server will use to open a bidirectional connection to the SDK.
If the SDK is attempting to sync a particular Realm Database file for the first time,
it does not yet possess a server-generated client identifier for the file. In
this case, the BIND request also indicates that the
Device Sync server should allocate one.
App Services Allocates a New Client File Identifier
If a BIND request indicates that the SDK needs a client
file identifier, the Device Sync server generates a unique value for the specified
Realm Database file and sends it to the SDK in an IDENT
response. When the SDK receives the IDENT, it stores
the new client identifier persistently in the local Realm Database file.
An SDK only needs to request a client file identifier the first time it syncs each Realm Database file. For subsequent sync sessions, the SDK can use the persisted identifier.
Client Sends a Client Identifier
Once an SDK has initiated a sync session with a BIND
request, it must identify the local Realm Database file that it wants to sync. To do
this, the SDK sends the application server an IDENT
message that contains the client file identifier. If the SDK has previously
synced the realm with the server, it can specify the most recently synced
server version to optimize the sync process.
When it receives the IDENT message, the server
establishes the session. The SDK and server can can now freely send upload and
download sync changesets at any time.
Client Uploads & Downloads Sync Changesets
Once a sync session is established, the SDK and server can freely send and
receive UPLOAD and DOWNLOAD
messages to sync changes whenever they occur.
The SDK sends an UPLOAD message for every changeset it
applies except for those that it received in a DOWNLOAD
message from the server.
When the server receives an UPLOAD message, it applies
operational transformations to resolve
any conflicts with other changesets and then applies the transformed changeset
to the server version of the realm. This triggers the server to send
DOWNLOAD messages to other connected clients, including
the synced Atlas cluster which mirrors the server realm. A
DOWNLOAD message groups one or more transformed
changesets in chronological order from oldest to most recent according to the
server's history. The SDK applies the changesets in the same order.
Request Types
Client -> Server Messages
The following table describes the request types that a sync client can send to a Device Sync server:
Request | Description |
|---|---|
Starts a new sync session on the server and provides a signed authorization token for the current application user. If the client does not yet possess a client file identifier for the Realm file it wants to sync, this also indicates that the server should generate one and send it back to the client. A client must send a BIND before it can send any other requests. | |
Provides the client file identifier that indicates the following:
This request is related to but distinct from the
| |
Specifies one or more changesets for operations that
occurred on the client. The changesets are listed by client version in
increasing order. | |
Specifies a changeset that describes a serialized
transaction that occurred on the client. The client may not upload any
other changesets until the server confirms or rejects the transaction. | |
Ends a running sync session. A client may not send any other requests for an
| |
Requests that the server notify the client when it has synced the latest
changeset in the server history (at the time of the
request). | |
Re-authorizes a current sync session with a new user token. | |
Requests that the server send one or more STATE
messages, which the client uses to download the current server version of
the Realm file. Clients issue state requests when they
asynchronously open a synced realm. | |
Requests that the server send the client version of the latest changeset
that was sent by the client and processed by the server. This is most
commonly used when an SDK executes a client reset. | |
Indicates that the client is still connected and that the server should
maintain the sync session. A client must send at least one PING to the
server every 10 minutes. The server acknowledges to each PING with a
If the server has not received a PING from a client in more than 10 minutes, it considers the client to be disconnected and may automatically end the session. |
Server -> Client Messages
The following table describes the request types that the Device Sync server can send to a sync client:
Request | Description |
|---|---|
Provides a client file identifier that
the server generated in response to a BIND that
requested the identifier. | |
Specifies one or more changesets (up to 16MB total) for operations that occurred on other clients. The changesets are listed by server version in increasing order. The changesets in a DOWNLOAD may not be the exact changesets uploaded by other clients. Instead, they may be equivalent changesets output by Device Sync's operational transformation algorithm. | |
Specifies that the server ended a sync session in response to an
UNBIND. | |
Indicates whether or not the server successfully processed a changeset
specified in a TRANSACT from the client. | |
Contains one or more segments of encoded data that the client can
concatenate to construct the latest server version of the realm. Sent
in response to a STATE_REQUEST. | |
Specifies the client version of the latest changeset that was sent by the
client and processed by the server. Sent in response to a
CLIENT_VERSION_REQUEST. | |
Indicates that the server encountered an issue that appears to have been
caused by the connected client. For details, see
Sync Client Errors. | |