fsync

Impact on Larger Deployments

When the fsync command runs on mongos, it performs the fsync operation on the entire cluster. By setting the lock field to true, it sets a lock on the cluster, preventing additional writes.

To take a usable self-managed backup, before locking a sharded cluster:

• Ensure that no chunk migration, resharding, or DDL operations are active.

• Stop the balancer to prevent additional chunk migrations from starting.

Alternatives with Journaling

If your mongod has journaling enabled, use a file system or volume/block level snapshot tool to create a backup of the data set and the journal together as a single unit.

Lock Count

The fsync command returns a document includes a lockCount field. When run on mongod, the count indicates the number of fsync locks set on the server.

When run on a sharded cluster, mongos sends the fsync operation to each shard and returns the results, which includes the lockCount for each.

Fsync Locks after Failures

Fsync locks execute on the primary in a replica set or sharded cluster.

If the primary goes down or becomes unreachable due to network issues, the cluster elects a new primary from the available secondaries. If a primary with an fsync lock goes down, the new primary does not retain the fsync lock and can handle write operations. When elections occur during backup operations, the resulting backup may be inconsistent or unusable.

To recover from the primary going down:

1. Run the fsyncUnlock command until the lock count reaches zero to release the lock on all nodes.

2. Issue the fsync command to reestablish the fsync lock on the cluster.

3. Restart the backup.

Additionally, fsync locks are persistent. When the old primary comes online again, you need to use the fsyncUnlock command to release the lock on the node.

Fsync Lock

The fsync command can lock an individual mongod instance or a sharded cluster through mongos. When run with the lock field set to true, the fsync operation flushes all data to the storage layer and blocks all additional write operations until you unlock the instance or cluster.

To lock the database, use the fsync command to set the lock field to true:

db.adminCommand( { fsync: 1, lock: true } )

The operation returns a document that includes the status of the operation and the lockCount:

{
   "info" : "now locked against writes, use db.fsyncUnlock() to unlock",
   "lockCount" : NumberLong(1),
   "seeAlso" : "http://dochub.mongodb.org/core/fsynccommand",
   "ok" : 1
}

When locked, write operations are blocked. Separate connections may continue read operations until the first attempt at a write operation, then they also wait until the sever or cluster is unlocked.

Fsync Unlock

To unlock a server of cluster, use the fsyncUnlock command:

db.adminCommnad( { fsyncUnlock: 1 } )

Repeat this command as many times as needed to reduce the lock count to zero. Once the lock count reaches zero, the server or cluster can resume writes.

Check Lock Status

To check the state of the fsync lock, use db.currentOp(). Use the following JavaScript function in the shell to test if the server or cluster is currently locked:

serverIsLocked = function () {
                     var co = db.currentOp();
                     if (co && co.fsyncLock) {
                         return true;
                     }
                     return false;
                 }

After loading this function into your mongosh session, call it with the following syntax:

serverIsLocked()

This function will return true if the server or cluster is currently locked and false if the server or cluster is not locked.