• Administer Ops Manager >
• Manage Snapshot Storage >
• Manage S3 Oplog Storage

Manage S3 Oplog Storage

Throughout the lifecycle of a backup, MongoDB Agent with Backup enabled tails the oplog of each replica set and sends new oplog entries to Ops Manager. The Agent sends the oplog entries in compressed bundles of approximately 10 MB in size called oplog slices. These oplog slices are stored in one or more MongoDB databases called oplog stores. Every Ops Manager deployment needs at least one oplog store.

When you enable backups, Ops Manager prompts you to create your first oplog store. This can be a local oplog store or an S3 oplog store. Once you create the first oplog store, you manage it separately from your snapshot stores. You can create additional oplog stores.

This tutorial covers creating additional S3 oplog stores to store oplog entries. Like any MongoDB instance, an S3 oplog store can exist on any host running MongoDB and S3 bucket (without subfolders) that the Ops Manager application can access.

Considerations

Requires a Dedicated Bucket

Ops Manager must be the only manager on the S3 bucket that you use for snapshots. You also need to configure the S3 bucket to avoid using features that Ops Manager does not support.

When configuring the S3 bucket:

• Do not create subfolders in the S3 buckets that you use with Ops Manager. Ops Manager only supports using entire S3 buckets.
• Disable AWS S3 bucket versioning. Versioning is not supported in Ops Manager for the S3 buckets used for snapshots.
• Do not create AWS S3 lifecycle rules. Lifecycle rules that expire or transition current versions of Ops Manager snapshot objects to archives results in incomplete snapshots that you can’t use to restore the configuration.

Can’t Move the S3 Oplog Store

After you create an S3 oplog store, you cannot move it to another S3 bucket. If you need to use a different S3 bucket to host your S3 oplog store, you must create a new S3 oplog store in the same S3 bucket. The S3 snapshot bucket must be different than the oplog S3 bucket.

Supports the Storage API

MongoDB supports endpoints for:

• AWS S3 API
• IBM Cloud Object Storage API
• Dell EMC Elastic Cloud Storage API.

IBM and Dell EMC support a subset of the full AWS S3 API.

You can use other S3-compatible endpoints. Ops Manager attempts to validate these endpoints when you save the S3 oplog store setup. If validation passes, Ops Manager saves the configuration. If validation fails, Ops Manager displays an error and doesn’t save the configuration.

Prerequisites

Metadata Storage Prerequisites

• Deploy the dedicated MongoDB instance(s) to serve the S3 oplog store metadata and Oplog Store. Serve these instances on the same hosts as the Ops Manager host, the backing databases, or snapshot stores. Attach one or more storage volumes with enough capacity to store the databases these instances manage.
• Secure the instance that stores your S3 oplog store metadata database using authentication and TLS. S3 oplog store metadata databases support all authentication mechanisms.

AWS S3 Storage Prerequisites

1. Make sure you have an IAM user on AWS.

2. Create your own AWS access keys for your IAM user. This allows you to create S3 buckets and store oplog files in them. MongoDB does not create or issue AWS access keys.

3. Create your own S3 bucket to store your S3 oplog stores.

   Note

   The IAM user for which you created the AWS access keys must own the AWS S3 bucket.

IBM Cloud Object Storage Prerequisites

1. Create an Access Key and Secret Key using IBM credential tools.
2. Create your own S3-compatible bucket.

Dell EMC Elastic Cloud Storage Prerequisites

1. Create an Access Key and Secret Key from your ECS User ID.
2. Create your own S3-compatible bucket.

Other S3-Compatible Storage

Other S3-compatible endpoints can be used. Ops Manager attempts to validate these endpoints when you save the configuration. If validation passes, the configuration, Ops Manager saves it. If validation fails, Ops Manager displays an error and doesn’t save the configuration.

Procedures

The format of the Username and Password depend upon the authentication mechanism. Select one of the following tabs:

• Username and Password
• X.509
• Kerberos
• LDAP

Add One S3 Oplog Store

1

Navigate to the Oplog Storage page.

1. Click the Admin link.
2. Click the Backup tab.
3. (Optional) If you have not previously set the head directory, set it in the Head Directory box.
4. Click the Oplog Storage page.

2

Click Create New S3 Oplog Store.

3

Provide the S3 Oplog Store Details.

Field	Necessity	Contents
Name	Required	Enter the label for the S3 oplog store.
S3 Bucket Name	Required	Enter the name of the S3 bucket where you want to host the the S3 oplog store.
Region Override	Conditional	Type the region where your S3 bucket resides. Use this field only if your S3-compatible store’s S3 Endpoint doesn’t support region scoping. Don’t provide a value for this field with AWS S3 buckets.
S3 Endpoint	Required	Enter the URL for this AWS S3 or S3-compatible bucket. What URL you write depends upon: Your S3 Bucket Name and If you checked Path Style Access. Example You created an S3 bucket called mybucket on AWS in the US East region. Path Style Access Checked Not Checked S3 Endpoint s3.us-east-2.amazonaws.com mybucket.s3.us-east-2.amazonaws.com Requests to Bucket https://s3.us-east-2.amazonaws.com/mybucket https://mybucket.s3.us-east-2.amazonaws.com
S3 Max Connections	Required	Enter a positive integer indicating the maximum number of connections to this AWS S3 or S3-compatible bucket.
Path Style Access	Optional	Select if you want your AWS S3 or S3-compatible bucket to use a path-style URL endpoint (s3.amazonaws.com/<bucket>) instead of a virtual-host-style URL endpoint (<bucket>.s3.amazonaws.com). To review the S3 bucket URL conventions, see the AWS S3 documentation
Server Side Encryption	Optional	Select to enable server-side encryption. Clear to disable server-side encryption.
S3 Authorization Mode	Required	Select the method used to authorize access to the S3 bucket specified in S3 Bucket Name. Keys Ops Manager uses AWS Access Key and AWS Secret Key to authorize access to your S3 bucket. IAM Role Ops Manager uses an AWS IAM role to authorize access to your S3 bucket. AWS Access Key and AWS Secret Key fields are ignored. To learn more, see the AWS documentation
Keys with Custom CA Bundle	Conditional	Click Choose file to add a custom Certificate Authority chain. This chain can validate against a self-signed certificate on the S3 bucket. Ops Manager displays this field when you set S3 Authorization Mode to Keys.
AWS Access Key	Conditional	Enter your AWS Access Key ID. Ops Manager displays this field when you set S3 Authorization Mode to Keys.
AWS Secret Key	Conditional	Enter your AWS Secret Access Key. Ops Manager displays this field when you set S3 Authorization Mode to Keys.
Datastore Type	Required	Select Standalone, Replica Set or Sharded Cluster. This MongoDB database stores the metadata for the blockstore.
MongoDB Host List	Conditional	Enter a comma-separated list of mongod instances (for a Replica Set) or mongos instances (for a Sharded Cluster) in the <hostname:port> format that comprise the blockstore metadata database. Example host1.example.com:27017,host2.example.com:27017,host2.example.com:27018 Ops Manager displays this field when you set Datastore Type to Replica Set or Sharded Cluster. Important Ops Manager uses this metadata store as a sync store. Make sure to provision this store with sufficient storage space.
MongoDB Hostname	Conditional	Enter the hostname of the S3 oplog store metadata database. Ops Manager displays this field when you set Datastore Type to Standalone. Important Ops Manager uses this metadata store as a sync store. Make sure to provision this store with sufficient storage space.
MongoDB Port	Conditional	Enter the port number of the S3 oplog store metadata database. Ops Manager displays this field when you set Datastore Type to Standalone.
Username	Optional	If you set this value: Type the name of the user authorized to access the this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring SCRAM authentication, see SCRAM. Type the RFC 2253-formatted subject from the client certificate of the user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring x.509 authentication, see x.509. Type the UPN of the user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring Kerberos authentication, see Kerberos. Type the name of the LDAP user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring LDAP authentication, see LDAP.
Password	Optional	If you set this value: Warning If you did not use the credentialstool to encrypt this password, it is stored as plaintext in the database. Type the password associated with the username that can access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring SCRAM authentication, see SCRAM. Leave it blank. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring x.509 authentication, see x.509. Kerberos retrieves the password from its keytab file. Don’t type a password into this field. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring Kerberos authentication, see Kerberos. Type the password of the LDAP user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring LDAP authentication, see LDAP.
Connection Options	Optional	Enter any additional configuration file options for the MongoDB instance. This field supports unescaped values only. For proper syntax, see Connection String URI Format in the MongoDB manual.
Encrypted Credentials	Optional	Select if the credentials for the database were encrypted using the credentialstool. The credentials include the Username, Password, AWS Access Key ID and AWS Secret Key.
Use TLS/SSL	Optional	Select if the S3 oplog store metadata database only accepts connection encrypted using TLS. Beyond this checkbox, to connect this S3 oplog store using TLS, you must enable TLS on the S3 oplog store database.
New Assignment Enabled	Optional	Select if you want to enable this S3 oplog store after creating it. This is selected by default so the S3 oplog store can be assigned backup jobs. If you clear this checkbox, the S3 oplog store is created but you cannot assign backups to this S3 oplog store.

4

Click Create.

Edit One Existing S3 Oplog Store

Ops Manager lists oplog stores in a table on the Oplog Storage page. Each row contains the settings for each local and S3 oplog store.

1

Navigate to the Oplog Storage page.

1. Click the Admin link.
2. Click the Backup tab.
3. (Optional) If you have not previously set the head directory, set it in the Head Directory box.
4. Click the Oplog Storage page.

2

Go to the row for the S3 oplog store you want to edit.

3

Update any values that need to be changed.

In the MongoDB Connection column, update any values that need to be changed in the following fields:

Field	Necessity	Contents
S3 Bucket Name	Required	Enter the name of the S3 bucket where you want to host the the S3 oplog store.
Region Override	Conditional	Type the region where your S3 bucket resides. Use this field only if your S3-compatible store’s S3 Endpoint doesn’t support region scoping. Don’t provide a value for this field with AWS S3 buckets.
S3 Endpoint	Required	Enter the URL for this AWS S3 or S3-compatible bucket. What URL you write depends upon: Your S3 Bucket Name and If you checked Path Style Access. Example You created an S3 bucket called mybucket on AWS in the US East region. Path Style Access Checked Not Checked S3 Endpoint s3.us-east-2.amazonaws.com mybucket.s3.us-east-2.amazonaws.com Requests to Bucket https://s3.us-east-2.amazonaws.com/mybucket https://mybucket.s3.us-east-2.amazonaws.com
S3 Max Connections	Required	Enter a positive integer indicating the maximum number of connections to this AWS S3 or S3-compatible bucket.
Path Style Access	Optional	Click if you want your AWS S3 or S3-compatible bucket to use a path-style URL endpoint (s3.amazonaws.com/<bucket>) instead of a virtual-host-style URL endpoint (<bucket>.s3.amazonaws.com). To review the S3 bucket URL conventions, see the AWS S3 documentation
Server Side Encryption	Optional	Click to enable server-side encryption. Clear to disable server-side encryption.
S3 Authorization Mode	Required	Select the method used to authorize access to the S3 bucket specified in S3 Bucket Name. Keys Ops Manager uses AWS Access Key and AWS Secret Key to authorize access to your S3 bucket. IAM Role Ops Manager uses an AWS IAM role to authorize access to your S3 bucket. AWS Access Key and AWS Secret Key fields are ignored. To learn more, see the AWS documentation
Keys with Custom CA Bundle	Conditional	Click Choose file to add a custom Certificate Authority chain. This chain can validate against a self-signed certificate on the S3 bucket. Ops Manager displays this field when you set S3 Authorization Mode to Keys.
AWS Access Key	Conditional	Enter your AWS Access Key ID. Ops Manager displays this field when you set S3 Authorization Mode to Keys.
AWS Secret Key	Conditional	Enter your AWS Secret Access Key. Ops Manager displays this field when you set S3 Authorization Mode to Keys. Note Ops Manager doesn’t display the existing Secret Access Key.
<hostname>:<port>	Required	Enter in one or more hosts that comprise the S3 Snapshot Store metadata database in the <hostname:port> format. S3 Oplog Hosting Concerns Ops Manager uses this metadata store as a sync store. Make sure to provision this store with sufficient storage space. If you change these hosts, the S3 oplog store they host must have the same data as the original S3 oplog store. Changing the host to a new S3 oplog store results in data loss. If the S3 oplog store metadata database is a Replica Set or Sharded Cluster, type a comma-separated list of mongod instances (for a Replica Set) or mongos instances (for a Sharded Cluster). Example host1.example.com:27017,host2.example.com:27017,host2.example.com:27018 If the S3 oplog store metadata database is a standalone MongoDB instance, type the hostname:port of the instance.
MongoDB Auth Username	Optional	If you set this value: Type the name of the user authorized to access the this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring SCRAM authentication, see SCRAM. Type the RFC 2253-formatted subject from the client certificate of the user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring x.509 authentication, see x.509. Type the UPN of the user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring Kerberos authentication, see Kerberos. Type the name of the LDAP user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring LDAP authentication, see LDAP.
MongoDB Auth Password	Optional	If you set this value: Type the password associated with the username that can access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring SCRAM authentication, see SCRAM. Leave it blank. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring x.509 authentication, see x.509. Kerberos retrieves the password from its keytab file. Don’t type a password into this field. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring Kerberos authentication, see Kerberos. Type the password of the LDAP user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring LDAP authentication, see LDAP. Warning If you did not use the credentialstool to encrypt this password, it is stored as plaintext in the database. Note Ops Manager doesn’t display the existing MongoDB Auth Password.
Encrypted Credentials	Optional	Select if the credentials for the database were encrypted using the credentialstool. The credentials include the Username, Password, AWS Access Key ID and AWS Secret Key.
Use TLS/SSL	Optional	Select if the S3 oplog store database only accepts connection encrypted using TLS. Beyond this checkbox, to connect this S3 oplog store using TLS, you must enable TLS on the S3 oplog store database.
Connection Options	Optional	Enter any additional configuration file options for the MongoDB instance. This field supports unescaped values only. To review the proper syntax, see Connection String URI Format in the MongoDB manual.
Assignment Labels	Optional	Enter a comma-separated list of labels to assign the S3 oplog stores to specific projects.
Write Concern	Required	Select your preferred Write Concern: Default Deployment Type Default Write Concern Standalone Journaled Replica sets or sharded clusters W2 Journaled A primary or standalone MongoDB instance acknowledged the write and wrote that write to their on-disk journals. Acknowledged A primary or standalone acknowledged the write. W2 More than one of the cluster members acknowledged the write. Majority A majority of the replica set members acknowledged the write.

4

Optional: Set the S3 Oplog Store to Accept Backup Jobs.

To enable this S3 oplog store, select Assignment Enabled.

This is selected by default so the S3 oplog store can be assigned backup jobs. If you clear this checkbox, Ops Manager creates the S3 oplog store but you cannot assign backups to it.

5

Click Save.

6

Optional: Restart Ops Manager Instances If Needed.

If you change any connection string values or the Write Concern, restart all the Ops Manager instances including those running Backup Daemons.

Warning

Modifying the connection string values or the Write Concern for an existing S3 oplog store requires you to restart all Ops Manager components, including those only running the Backup Daemon to apply those changes. Connection parameters include:

• <hostname>:<port>
• MongoDB Auth Username
• MongoDB Auth Password
• Encrypted Credentials
• Use TLS/SSL
• Connection Options
• Write Concern

If you change to another S3 oplog store host, Ops Manager doesn’t copy the data on the existing S3 oplog store to the other S3 oplog store.

See also

To learn more about the MongoDB connection string URI, see Connection String URI Format in the MongoDB Manual.

Delete One S3 Oplog Store

1

Navigate to the Oplog Storage page.

1. Click the Admin link.
2. Click the Backup tab.
3. (Optional) If you have not previously set the head directory, set it in the Head Directory box.
4. Click the Oplog Storage page.

2

Click the Delete link.

Click Delete <oplogstore> beneath the name of the S3 oplog store you want to delete.

←   Manage Oplog Storage Configure Block Size in a Blockstore  →

© MongoDB, Inc 2008-present. MongoDB, Mongo, and the leaf logo are registered trademarks of MongoDB, Inc.