Upgrade MongoDB to 2.4

Considerations

Beware of the following properties of the cluster upgrade process:

• Before you start the upgrade, ensure that the amount of free space on the filesystem for the config database is at least 4 to 5 times the amount of space currently used by the config database data files.

  Additionally, ensure that all indexes in the config database are {v:1} indexes. If a critical index is a {v:0} index, chunk splits can fail due to known issues with the {v:0} format. {v:0} indexes are present on clusters created with MongoDB 2.0 or earlier.

  The duration of the metadata upgrade depends on the network latency between the node performing the upgrade and the three config servers. Ensure low latency between the upgrade process and the config servers.

• While the upgrade is in progress, you cannot make changes to the collection meta-data. For example, during the upgrade, do not perform:

  • sh.enableSharding(),

  • sh.shardCollection(),

  • sh.addShard(),

  • db.createCollection(),

  • db.collection.drop(),

  • db.dropDatabase(),

  • any operation that creates a database, or

  • any other operation that modifies the cluster meta-data in any way. See Sharding Reference for a complete list of sharding commands. Note, however, that not all commands on the Sharding Reference page modifies the cluster meta-data.

• Once you upgrade to 2.4 and complete the upgrade procedure do not use 2.0 mongod and mongos processes in your cluster. 2.0 process may re-introduce old meta-data formats into cluster meta-data.

The upgraded config database will require more storage space than before, to make backup and working copies of the config.chunks and config.collections collections. As always, if storage requirements increase, the mongod might need to pre-allocate additional data files. See How can I get information on the storage use of a database? for more information.

Meta-data Upgrade Procedure

Changes to the meta-data format for sharded clusters, stored in the config database, require a special meta-data upgrade procedure when moving to 2.4.

Do not perform operations that modify meta-data while performing this procedure. See Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4 for examples of prohibited operations.

1. Before you start the upgrade, ensure that the amount of free space on the filesystem for the config database is at least 4 to 5 times the amount of space currently used by the config database data files.

   Additionally, ensure that all indexes in the config database are {v:1} indexes. If a critical index is a {v:0} index, chunk splits can fail due to known issues with the {v:0} format. {v:0} indexes are present on clusters created with MongoDB 2.0 or earlier.

   The duration of the metadata upgrade depends on the network latency between the node performing the upgrade and the three config servers. Ensure low latency between the upgrade process and the config servers.

   To check the version of your indexes, use db.collection.getIndexes().

   If any index on the config database is {v:0}, you should rebuild those indexes by connecting to the mongos and either: rebuild all indexes using the db.collection.reIndex() method, or drop and rebuild specific indexes using db.collection.dropIndex() and then db.collection.ensureIndex(). If you need to upgrade the _id index to {v:1} use db.collection.reIndex().

   You may have {v:0} indexes on other databases in the cluster.

2. Turn off the balancer in the sharded cluster, as described in Disable the Balancer.

   Note

   Optional

   For additional security during the upgrade, you can make a backup of the config database using mongodump or other backup tools.

3. Ensure there are no version 2.0 mongod or mongos processes still active in the sharded cluster. The automated upgrade process checks for 2.0 processes, but network availability can prevent a definitive check. Wait 5 minutes after stopping or upgrading version 2.0 mongos processes to confirm that none are still active.

4. Start a single 2.4 mongos process with configDB pointing to the sharded cluster's config servers and with the --upgrade option. The upgrade process happens before the process becomes a daemon (i.e. before --fork.)

   You can upgrade an existing mongos instance to 2.4 or you can start a new mongos instance that can reach all config servers if you need to avoid reconfiguring a production mongos.

   Start the mongos with a command that resembles the following:

   mongos --configdb <config servers> --upgrade

   Without the --upgrade option 2.4 mongos processes will fail to start until the upgrade process is complete.

   The upgrade will prevent any chunk moves or splits from occurring during the upgrade process. If there are very many sharded collections or there are stale locks held by other failed processes, acquiring the locks for all collections can take seconds or minutes. See the log for progress updates.

5. When the mongos process starts successfully, the upgrade is complete. If the mongos process fails to start, check the log for more information.

   If the mongos terminates or loses its connection to the config servers during the upgrade, you may always safely retry the upgrade.

   However, if the upgrade failed during the short critical section, the mongos will exit and report that the upgrade will require manual intervention. To continue the upgrade process, you must follow the Resync after an Interruption of the Critical Section procedure.

   Note

   Optional

   If the mongos logs show the upgrade waiting for the upgrade lock, a previous upgrade process may still be active or may have ended abnormally. After 15 minutes of no remote activity mongos will force the upgrade lock. If you can verify that there are no running upgrade processes, you may connect to a 2.2 mongos process and force the lock manually:

   mongo <mongos.example.net>

   db.getMongo().getCollection("config.locks").findOne({ _id : "configUpgrade" })

   If the process specified in the process field of this document is verifiably offline, run the following operation to force the lock.

   db.getMongo().getCollection("config.locks").update({ _id : "configUpgrade" }, { $set : { state : 0 } })

   It is always more safe to wait for the mongos to verify that the lock is inactive, if you have any doubts about the activity of another upgrade operation. In addition to the configUpgrade, the mongos may need to wait for specific collection locks. Do not force the specific collection locks.

6. Upgrade and restart other mongos processes in the sharded cluster, without the --upgrade option.

   See Upgrade Sharded Cluster Components for more information.

7. Re-enable the balancer. You can now perform operations that modify cluster meta-data.

Once you have upgraded, do not introduce version 2.0 MongoDB processes into the sharded cluster. This can reintroduce old meta-data formats into the config servers. The meta-data change made by this upgrade process will help prevent errors caused by cross-version incompatibilities in future versions of MongoDB.

Resync after an Interruption of the Critical Section

During the short critical section of the upgrade that applies changes to the meta-data, it is unlikely but possible that a network interruption can prevent all three config servers from verifying or modifying data. If this occurs, the config servers must be re-synced, and there may be problems starting new mongos processes. The sharded cluster will remain accessible, but avoid all cluster meta-data changes until you resync the config servers. Operations that change meta-data include: adding shards, dropping databases, and dropping collections.

To resync the config servers:

1. Turn off the balancer in the sharded cluster and stop all meta-data operations. If you are in the middle of an upgrade process (Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4), you have already disabled the balancer.

2. Shut down two of the three config servers, preferably the last two listed in the configDB string. For example, if your configDB string is configA:27019,configB:27019,configC:27019, shut down configB and configC. Shutting down the last two config servers ensures that most mongos instances will have uninterrupted access to cluster meta-data.

3. mongodump the data files of the active config server (configA).

4. Move the data files of the deactivated config servers (configB and configC) to a backup location.

5. Create new, empty data directories.

6. Restart the disabled config servers with --dbpath pointing to the now-empty data directory and --port pointing to an alternate port (e.g. 27020).

7. Use mongorestore to repopulate the data files on the disabled documents from the active config server (configA) to the restarted config servers on the new port (configB:27020,configC:27020). These config servers are now re-synced.

8. Restart the restored config servers on the old port, resetting the port back to the old settings (configB:27019 and configC:27019).

9. In some cases connection pooling may cause spurious failures, as the mongos disables old connections only after attempted use. 2.4 fixes this problem, but to avoid this issue in version 2.2, you can restart all mongos instances (one-by-one, to avoid downtime) and use the rs.stepDown() method before restarting each of the shard replica set primaries.

10. The sharded cluster is now fully resynced; however before you attempt the upgrade process again, you must manually reset the upgrade state using a version 2.2 mongos. Begin by connecting to the 2.2 mongos with the mongo shell:

    mongo <mongos.example.net>

    Then, use the following operation to reset the upgrade process:

    db.getMongo().getCollection("config.version").update({ _id : 1 }, { $unset : { upgradeState : 1 } })

11. Finally retry the upgrade process, as in Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4.

Basic Downgrade and Upgrade

Except as described below, moving between 2.2 and 2.4 is a drop-in replacement:

• stop the existing mongod, using the --shutdown option as follows:

  mongod --dbpath /var/mongod/data --shutdown

  Replace /var/mongod/data with your MongoDB dbPath.

• start the new mongod processes with the same dbPath setting, for example:

  mongod --dbpath /var/mongod/data

  Replace /var/mongod/data with your MongoDB dbPath.

Downgrade to 2.2 After Creating a 2dsphere or text Index

If you have created 2dsphere or text indexes while running a 2.4 mongod instance, you can downgrade at any time, by starting the 2.2 mongod with the --upgrade option as follows:

mongod --dbpath /var/mongod/data/ --upgrade

Then, you will need to drop any existing 2dsphere or text indexes using db.collection.dropIndex(), for example:

db.records.dropIndex( { loc: "2dsphere" } )
db.records.dropIndex( "records_text" )

Troubleshooting Upgrade/Downgrade Operations

If you do not use --upgrade, when you attempt to start a 2.2 mongod and you have created a 2dsphere or text index, mongod will return the following message:

'need to upgrade database index_plugin_upgrade with pdfile version 4.6, new version: 4.5 Not upgrading, exiting'

While running 2.4, to check the data file version of a MongoDB database, use the following operation in the shell:

db.getSiblingDB('<databasename>').stats().dataFileVersion

The major data file [1] version for both 2.2 and 2.4 is 4, the minor data file version for 2.2 is 5 and the minor data file version for 2.4 is 6 after you create a 2dsphere or text index.