Restore a Sharded Cluster

Restore the CSRS primary mongod data files.

Select the tab that corresponds to your preferred backup method:

Drop the local database.

Use db.dropDatabase() to drop the local database:

use local
db.dropDatabase()

Insert the filtered file list into the local database.

This step is only required if you are restoring from a namespace-filtered snapshot.

For each shard, locate the filtered file list with the following name format: <shardRsID>-filteredFileList.txt. This file contains a list of JSON objects with the following format:

{
   "filename":"file1",
   "ns":"sampleDb1.sampleCollection1",
   "uuid": "3b241101-e2bb-4255-8caf-4136c566a962"
}

Add each JSON object from each shard file to a new db.systems.collections_to_restore collection in your local database. You can ignore entries with empty ns or uuid fields. When inserting entries, the uuid field must be inserted as type UUID().

Run the filtered file restore command.

This step is only required if you are restoring from a namespace-filtered snapshot.

After inserting all entries, run the following commands:

use admin
db.runCommand({"_configsvrRunRestore":1})

For any planned or completed shard hostname or replica set name changes, update the metadata in config.shards.

You can skip this step if all of the following are true:

• No shard member host machine hostname has or will change during this procedure.

• No shard replica set name has or will change during this procedure.

Issue the following find() method on the shards collection in the Config Database. Replace <shardName> with the name of the shard. By default the shard name is its replica set name. If you added the shard using the addShard command and specified a custom name, you must specify that name to <shardName>.

use config
db.shards.find( { "_id" : "<shardName>" } )

This operation returns a document that resembles the following:

{
   "_id" : "shard1",
   "host" : "myShardName/alpha.example.net:27018,beta.example.net:27018,charlie.example.net:27018",
   "state" : 1
}

Use the updateOne() method to update the hosts string to reflect the planned replica set name and hostname list for the shard. For example, the following operation updates the host connection string for the shard with "_id" : "shard1":

db.shards.updateOne(
  { "_id" : "shard1" },
  { $set : { "host" : "myNewShardName/repl1.example.net:27018,repl2.example.net:27018,repl3.example.net:27018" } }
)

Repeat this process until all shard metadata accurately reflects the planned replica set name and hostname list for each shard in the cluster.

use config
db.shards.find({})

Restart the mongod as a new single-node replica set.

Shut down the mongod. Uncomment or add the following configuration file options:

replication
  replSetName: myNewCSRSName
sharding
  clusterRole: configsvr

If you want to change the replica set name, you must update the replSetName field with the new name before proceeding.

Start the mongod with the updated configuration file:

mongod --config /path/to/mongodb/mongod.conf

If you have mongod configured to run as a system service, start it using the recommended process for your system service manager.

After the mongod starts, connect to it using the mongo shell.

Initiate the new replica set.

Initiate the replica set using rs.initiate() with the default settings.

rs.initiate()

Once the operation completes, use rs.status() to check that the member has become the primary.

Add additional replica set members.

For each replica set member in the CSRS, start the mongod on its host machine. Once you have started up all remaining members of the cluster successfully, connect a mongo shell to the primary replica set member. From the primary, use the rs.add() method to add each member of the replica set. Include the replica set name as the prefix, followed by the hostname and port of the member's mongod process:

rs.add("config2.example.net:27019")
rs.add("config3.example.net:27019")

If you want to add the member with specific replica member configuration settings, you can pass a document to rs.add() that defines the member hostname and any members settings your deployment requires.

rs.add(
 {
   "host" : "config2.example.net:27019",
   priority: <int>,
   votes: <int>,
   tags: <int>
 }
)

Each new member performs an initial sync to catch up to the primary. Depending on factors such as the amount of data to sync, your network topology and health, and the power of each host machine, initial sync may take an extended period of time to complete.

The replica set may elect a new primary while you add additional members. Use rs.status() to identify which member is the current primary. You can only run rs.add() from the primary.

Configure any additional required replication settings.

The rs.reconfig() method updates the replica set configuration based on a configuration document passed in as a parameter. You must run reconfig() against the primary member of the replica set.

Reference the original configuration file output of the replica set as identified in step A. Review Replica Set Configurations and apply settings as needed.

Restore the shard primary mongod data files.

Select the tab that corresponds to your preferred backup method:

Create a temporary user with the __system role.

During this procedure you will modify documents in the admin.system.version collection. For clusters enforcing authentication, only the __system role grants permission to modify this collection. You can skip this step if the cluster does not enforce authentication.

1. Authenticate as a user with the userAdmin role on the admin database or userAdminAnyDatabase role:

   use admin
   db.auth("myUserAdmin","mySecurePassword")

2. Create a user with the __system role:

   db.createUser(
     {
       user: "mySystemUser",
       pwd: "<replaceMeWithAStrongPassword>",
       roles: [ "__system" ]
     }
   )

   Passwords should be random, long, and complex to ensure system security and to prevent or delay malicious access.

3. Authenticate as the privileged user:

   db.auth("mySystemUser","<replaceMeWithAStrongPassword>")

Drop the local database.

Use db.dropDatabase() to drop the local database:

use local
db.dropDatabase()

Remove the minOpTimeRecovery document from the admin.system.versions collection.

To update the sharding internals, issue the following deleteOne() method on the system.version collection in the admin database:

use admin
db.system.version.deleteOne( { _id: "minOpTimeRecovery" } )

Optional: For any CSRS hostname or replica set name changes, update shard metadata in each shard's identity document.

You can skip this step if all of the following are true:

• The hostnames for any CSRS host did not change during this procedure.

• The CSRS replica set name did not change during this procedure.

The system.version collection on the admin database contains metadata related to the shard, including the CSRS connection string. If either the CSRS name or any member hostnames changed while restoring the CSRS, you must update this metadata.

Issue the following find() method on the system.version collection in the admin database:

use admin
db.system.version.find( {"_id" : "shardIdentity" } )

The find() method returns a document that resembles the following:

{
  "_id" : "shardIdentity",
  "clusterId" : ObjectId("2bba123c6eeedcd192b19024"),
  "shardName" : "shard1",
  "configsvrConnectionString" : "myCSRSName/alpha.example.net:27019,beta.example.net:27019,charlie.example.net:27019" }

The following updateOne() method updates the document such that the host string represents the most current CSRS connection string:

db.system.version.updateOne(
  { "_id" : "shardIdentity" },
  { $set :
    { "configsvrConnectionString" : "myNewCSRSName/config1.example.net:27019,config2.example.net:27019,config3.example.net:27019"}
  }
)

Restart the mongod as a new single-node replica set.

Shut down the mongod. Uncomment or add the following configuration file options:

replication
  replSetName: myNewShardName
sharding
  clusterRole: shardsvr

If you want to change the replica set name, you must update the replSetName field with the new name before proceeding.

Start the mongod with the updated configuration file:

mongod --config /path/to/mongodb/mongod.conf

If you have mongod configured to run as a system service, start it using the recommended process for your system service manager.

After the mongod starts, connect to it using the mongo shell.

Initiate the new replica set.

Initiate the replica set using rs.initiate() with the default settings.

rs.initiate()

Once the operation completes, use rs.status() to check that the member has become the primary.

Add additional replica set members.

For each replica set member in the shard replica set, start the mongod on its host machine. Once you have started up all remaining members of the cluster successfully, connect a mongo shell to the primary replica set member. From the primary, use the rs.add() method to add each member of the replica set. Include the replica set name as the prefix, followed by the hostname and port of the member's mongod process:

rs.add("repl2.example.net:27018")
rs.add("repl3.example.net:27018")

If you want to add the member with specific replica member configuration settings, you can pass a document to rs.add() that defines the member hostname and any members settings your deployment requires.

rs.add(
 {
   "host" : "repl2.example.net:27018",
   priority: <int>,
   votes: <int>,
   tags: <int>
 }
)

Each new member performs an initial sync to catch up to the primary. Depending on factors such as the amount of data to sync, your network topology and health, and the power of each host machine, initial sync may take an extended period of time to complete.

The replica set may elect a new primary while you add additional members. Use rs.status() to identify which member is the current primary. You can only run rs.add() from the primary.

Configure any additional required replication settings.

The rs.reconfig() method updates the replica set configuration based on a configuration document passed in as a parameter. You must run reconfig() against the primary member of the replica set.

Reference the original configuration file output of the replica set as identified in step A. Review Replica Set Configurations and apply settings as needed.

Remove the temporary privileged user.

For clusters enforcing authentication, remove the privileged user created earlier in this procedure:

1. Authenticate as a user with the userAdmin role on the admin database or userAdminAnyDatabase role:

   use admin
   db.auth("myUserAdmin","mySecurePassword")

2. Delete the privileged user:

   db.removeUser("mySystemUser")