Community

Fabio Ramohitaj


The challenges that are raised in modern business contexts are increasingly complex. These challenges range from the ability to minimize downtime during migrations to adopting efficient tools for transitioning from relational to non-relational databases, and from implementing resilient architectures that ensure high availability to the ability to scale horizontally, allowing large amounts of data to be efficiently managed and queried.

Two of the main challenges, which will be covered in this article, are:

-   The need to create resilient IT infrastructures that can ensure business continuity or minimal downtime even in critical situations, such as the loss of a data center.

-   Conducting migrations from one infrastructure to another without compromising operations.

It is in this context that MongoDB stands out by offering innovative solutions such as MongoSync and live migrate.

Ensuring business continuity with MongoSync: an approach to disaster recovery
-----------------------------------------------------------------------------

MongoDB Atlas, with its capabilities and remarkable flexibility, offers two distinct approaches to implementing business continuity strategies. These two strategies are:

-   Creating a cluster with a geographic distribution of nodes.

-   The implementation of two clusters in different regions synchronized via MongoSync.

In this section, we will explore the second point (i.e., the implementation of two clusters in different regions synchronized via MongoSync) in more detail.

What exactly is MongoSync? For a correct definition, we can refer to the [official documentation](https://www.mongodb.com/pt-br/docs/cluster-to-cluster-sync/current/reference/mongosync/#description):

"The `mongosync` binary is the primary process used in Cluster-to-Cluster Sync. `mongosync` migrates data from one cluster to another and can keep the clusters in continuous sync."

This tool performs the following operations:

-   It migrates data from one cluster to another.

-   It keeps the clusters in continuous sync.

Let's make this more concrete with an example:

-   Initially, the situation looks like this for the production cluster and the disaster recovery cluster:

![Image of two clusters: the first already populated with data, while the second is empty and waiting to be populated. The two clusters are located in different datacenters][1]

The current state of the main cluster would look like this:

```
Atlas atlas-qsd40w-shard-0 [primary] test> show dbs
admin               140.00 KiB
config              276.00 KiB
local               524.00 KiB
sample_airbnb        52.09 MiB
sample_analytics      9.44 MiB
sample_geospatial     1.02 MiB
sample_guides        40.00 KiB
sample_mflix        109.01 MiB
sample_restaurants    5.73 MiB
sample_supplies     976.00 KiB
sample_training      41.20 MiB
sample_weatherdata    2.39 MiB
```

The back-up used for disaster recovery is still blank:

```
Atlas atlas-lcu71y-shard-0 [primary] test> show dbs
admin   172.00 KiB
config  212.00 KiB
local   584.00 KiB
```

Before proceeding, it is essential to install the `mongosync` binary. If you have not already done so, you can download it from the [downloads page](https://www.mongodb.com/try/download/mongosync). The commands described below have been tested in the CentOS 7 operating system.

Let's proceed with the configuration of `mongosync` by defining a configuration file and a service:

```
vi /etc/mongosync.conf
```

You can copy and paste the current configuration into this file using the appropriate connection strings. You can also test with two Atlas clusters, which must be M10 level or higher. For more details on how to get the connection strings from your Atlas cluster, you can consult the [documentation](https://www.mongodb.com/pt-br/docs/guides/atlas/connection-string/).

```
cluster0:  "mongodb+srv://test_u:test_p@cluster0.*****.mongodb.net/?retryWrites=true&w=majority"
cluster1:  "mongodb+srv://test_u:test_p@cluster1.*****.mongodb.net/?retryWrites=true&w=majority"
logPath:  "/data/log/mongosync"
verbosity:  "INFO"
```

>Generally, this step is performed on a Linux machine by system administrators. Although the step is optional, it is recommended to implement it in a production environment.

Next, you will be able to create a service named mongosync.service.

```
vi /usr/lib/systemd/system/mongosync.service
```

This is what your service file should look like.

```
 [Unit]
Description=Cluster-to-Cluster Sync
Documentation=https://www.mongodb.com/pt-br/docs/cluster-to-cluster-sync/
[Service]
User=root
Group=root
ExecStart=/usr/local/bin/mongosync --config /etc/mongosync.conf
[Install]
WantedBy=multi-user.target
```

Reload all unit files:

```
systemctl daemon-reload
```

Now, we can start the service: 

```
systemctl start mongosync
```

We can also check whether the service has been started correctly:

```
systemctl status mongosync
```

Output:

```
mongosync.service - Cluster-to-Cluster Sync
    Loaded: loaded (/usr/lib/systemd/system/mongosync.service; disabled; vendor preset: disabled)
Active: active (running) since dom 2024-04-14  21:45:45 CEST; 4s ago
 Docs: https://www.mongodb.com/pt-br/docs/cluster-to-cluster-sync/
Main PID: 1573 (mongosync)
    CGroup: /system.slice/mongosync.service
 └─1573 /usr/local/bin/mongosync --config /etc/mongosync.conf

apr 14  21:45:45 mongosync.mongodb.int systemd[1]: Started Cluster-to-Cluster Sync.
```

> If a service is not created and executed, in a more general way, you can start the process in the following way: 
> `mongosync --config mongosync.conf `

After starting the service, verify that it is in the idle state:

```
curl localhost:27182/api/v1/progress -XGET | jq
```

Output:

```
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
100  191 100  191 0  0 14384 0 --:--:-- --:--:-- --:--:--  14692
{
 "progress": {
 "state": "IDLE",
 "canCommit": false,
 "canWrite": false,
 "info": null,
 "lagTimeSeconds": null,
 "collectionCopy": null,
 "directionMapping": null,
 "mongosyncID": "coordinator",
 "coordinatorID": ""
  }
}
```
We can run the synchronization:

```
curl localhost:27182/api/v1/start -XPOST \
--data '
    {
 "source": "cluster0",
 "destination": "cluster1",
 "reversible": true,
 "enableUserWriteBlocking": true
    } '
```

Output:

```
{"success":true}
```

We can also keep track of the synchronization status:

```
curl localhost:27182/api/v1/progress -XGET | jq
```

Output:

```
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
100  502 100  502 0  0 36001 0 --:--:-- --:--:-- --:--:--  38615
{
 "progress": {
 "state": "RUNNING",
 "canCommit": false,
 "canWrite": false,
 "info": "collection copy",
 "lagTimeSeconds": 54,
 "collectionCopy": {
 "estimatedTotalBytes": 390696597,
 "estimatedCopiedBytes": 390696597
    },
 "directionMapping": {
 "Source": "cluster0: cluster0.*****.mongodb.net",
 "Destination": "cluster1: cluster1.*****.mongodb.net"
    },
 "mongosyncID": "coordinator",
 "coordinatorID": "coordinator"
  }
}

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
100  510 100  510 0  0 44270 0 --:--:-- --:--:-- --:--:--  46363
{
 "progress": {
 "state": "RUNNING",
 "canCommit": true,
 "canWrite": false,
 "info": "change event application",
 "lagTimeSeconds": 64,
 "collectionCopy": {
 "estimatedTotalBytes": 390696597,
 "estimatedCopiedBytes": 390696597
    },
 "directionMapping": {
 "Source": "cluster0: cluster0.*****.mongodb.net",
 "Destination": "cluster1: cluster1.*****.mongodb.net"
    },
 "mongosyncID": "coordinator",
 "coordinatorID": "coordinator"
  }
}
```

At this time, the DR environment is aligned with the production environment and will also maintain synchronization for the next operations: 

![Image of two clusters located in different datacenters, aligned and remained synchronized via mongosync. Mongosync runs on an on-premises server.][2]

```
Atlas atlas-qsd40w-shard-0 [primary] test> show dbs
admin               140.00 KiB
config              276.00 KiB
local               524.00 KiB
sample_airbnb        52.09 MiB
sample_analytics      9.44 MiB
sample_geospatial     1.02 MiB
sample_guides        40.00 KiB
sample_mflix        109.01 MiB
sample_restaurants    5.73 MiB
sample_supplies     976.00 KiB
sample_training      41.20 MiB
sample_weatherdata    2.39 MiB
```

And our second cluster is now in sync with the following data.

```
Atlas atlas-lcu71y-shard-0 [primary] test> show dbs
admin                                172.00 KiB
config                               380.00 KiB
local                                427.22 MiB
mongosync_reserved_for_internal_use  420.00 KiB
sample_airbnb                         53.06 MiB
sample_analytics                       9.55 MiB
sample_geospatial                      1.40 MiB
sample_guides                         40.00 KiB
sample_mflix                         128.38 MiB
sample_restaurants                     6.47 MiB
sample_supplies                        1.03 MiB
sample_training                       47.21 MiB
sample_weatherdata                     2.61 MiB
````

Armed with what we've discussed so far, we could ask a last question like:

*Is it possible to take advantage of the disaster recovery environment in some way, or should we just let it synchronize?*

By making the appropriate `mongosync` configurations --- for example, by setting the "buildIndexes" option to false and omitting the "enableUserWriteBlocking" parameter (which is set to false by default) --- we can take advantage of the [limitation](https://www.mongodb.com/pt-br/docs/cluster-to-cluster-sync/current/reference/limitations/#limitations) regarding non-synchronization of users and roles to create read-only users. We do this in such a way that no entries can be entered, thereby ensuring consistency between the origin and destination clusters and allowing us to use the disaster recovery environment to create the appropriate indexes that will go into optimizing slow queries identified in the production environment.

Live migrate to Atlas: minimizing downtime
------------------------------------------

Live migrate is a tool that allows users to perform migrations to [MongoDB Atlas](https://www.mongodb.com/atlas/database) and more specifically, as mentioned by the [official documentation](https://www.mongodb.com/pt-br/docs/atlas/import/c2c-pull-live-migration/#live-migrate--pull--a-mongodb-6.0.8--cluster-into-atlas), is a process that uses `mongosync` as the underlying data migration tool, enabling faster live migrations with less downtime if both the source and destination clusters are running MongoDB 6.0.8 or later.

So, what is the added value of this tool compared to `mongosync`?

It brings two advantages:

-   You can avoid the need to provision and configure a server to host `mongosync`.

-   You have the ability to migrate from previous versions, as indicated in the [migration path](https://www.mongodb.com/pt-br/docs/atlas/import/live-import/#migration-path).

![Image representing the migration of an on-premises production cluster to MongoDB Atlas.][3]

Conclusion
----------

As we have seen, these tools, with proper care, allow us to achieve our goals while also providing us with a certain flexibility. 

Regardless of the solution that will be used for migration and/or synchronization, you will be able to contact MongoDB support, who will help you identify the best strategy to solve that task successfully.

If you have questions or comments, you can find us in the [MongoDB Developer Community](https://www.mongodb.com/community/forums/)!


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf8bd745448a43713/663e5cdea2616e0474ff1789/Screenshot_2024-05-10_at_1.40.54_PM.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd0c6cb1fbf15ed87/663e5d0fa2616e5e82ff178f/Screenshot_2024-05-10_at_1.41.09_PM.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt9ace7dcef1c2e7e9/663e67322ff97d34907049ac/Screenshot_2024-05-10_at_2.24.24_PM.png

Learn about how to enable cluster-to-cluster sync

Efficient Sync Solutions: Cluster-to-Cluster Sync and Live Migration to Atlas

Defining the process of data migration from a relational database to MongoDB has always been a complex task. Some have opted for a custom approach, adopting custom solutions such as scripts, whereas others have preferred to use third-party tools. 

It is in this context that the Relational Migrator enters the picture, melting the complexity of this transition from a relational database to MongoDB as naturally as the sun melts the snow.

## How Relational Migrator comes to our help

In the context of a relational database to MongoDB migration project, several questions arise — for example:

- What tool should you use to best perform this migration?
- How can this migration process be made time-optimal for a medium/large size database?
- How will the data need to be modeled on MongoDB?
- How much time/resources will it take to restructure SQL queries to MQL?

Consider the following architecture, as an example:

![Data Architecture Example][1]

We present a first approach, which will not make use of Relational Migrator but instead will use a third-party tool to orchestrate the migration within this architecture: Logstash. 

We will make observations and define what could be the limitations of this tool.

What exactly is Logstash? For a correct definition, we can refer to the [official documentation](https://www.elastic.co/logstash):

Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite "stash."

This tool effectively achieves the goal by dynamically performing the following operations:

- Ingestion of data from the source — PostgreSQL
- Data transformation — Logstash
- Distribution of the transformed data to the destination — MongoDB

Great! So, it will be possible to migrate data and benefit from high flexibility in its transformation, and we also assume relatively short time frames because we've done some very good tuning, but different pipelines will have to be defined manually. 

Let us concretize with an example what we have been telling ourselves so far by considering the following scheme:

![Database Schema for Student Enrollment][2]

| matriculation | name     | surname | birthdate  | address           | email                  | phone        |
| ------------- | -------- | ------- | ---------- | ----------------- | ---------------------- | ------------ |
| 1             | Mario    | Rossi   | 1990-05-15 | Via Roma, 123     | mario.rossi@email.com  | 123-456-7890 |
| 2             | Laura    | Bianchi | 1992-08-22 | Via Garibaldi, 56 | laura.b@email.com      | 987-654-3210 |
| 3             | Giuseppe | Verdi   | 1991-03-10 | Via Dante, 78     | giuseppe.verdi@email   | 555-123-4567 |
| 4             | Sofia    | Conti   | 1993-11-30 | Via Milano, 45    | sofia.conti@email.com  | 333-555-9999 |
| 5             | Alessia  | Moretti | 1992-06-18 | Via Firenze, 7    | alessia.more@email.com | 111-222-3333 |
| 6             | Luca     | Ferrari | 1994-09-25 | Via Venezia, 21   | luca.f@email.com       | 444-777-8888 |
| 7             | Marta    | Romano  | 1990-02-03 | Via Bologna, 34   | marta.rom@email.com    | 999-888-7777 |
| 8             | Marco    | Galli   | 1993-07-12 | Via Genova, 56    | marco.galli@email.com  | 666-333-1111 |
| 9             | Elena    | Piazza  | 1995-04-09 | Via Torino, 78    | elena.p@email.com      | 222-999-4444 |
| 10            | Andrea   | Mancini | 1991-12-28 | Via Padova, 12    | andrea.m@email.com     | 777-111-5555 |

Enrollment table:

| enrollmentid | studentmatriculation | coursecode | enrollmentdate |
| ------------ | -------------------- | ---------- | -------------- |
| 11           | 1                    | CS101      | 2023-01-15     |
| 12           | 2                    | LI301      | 2023-02-02     |
| 13           | 3                    | MA201      | 2023-03-10     |
| 14           | 1                    | CH401      | 2023-04-05     |
| 15           | 4                    | CS101      | 2023-04-15     |
| 16           | 5                    | PH501      | 2023-05-20     |
| 17           | 6                    | EC601      | 2023-06-08     |
| 18           | 7                    | HI701      | 2023-07-12     |
| 19           | 8                    | EN801      | 2023-08-05     |
| 20           | 9                    | MU901      | 2023-09-01     |

Course table:

| Coursecode | coursename       | credits | department  |
| ---------- | ---------------- | ------- | ----------- |
| CS101      | Informatics      | 4       | Informatics |
| MA201      | Mathematics      | 3       | Mathematics |
| LI301      | Italian Language | 5       | Letters     |
| CH401      | Chemistry        | 4       | Chemistry   |
| PH501      | Physics          | 4       | Physics     |
| EC601      | Economics        | 5       | Economics   |
| HI701      | History          | 3       | History     |
| EN801      | English          | 4       | Languages   |
| MU901      | Music            | 2       | Music       |
| GE1001     | Geography        | 3       | Geography   |

Our goal, based on an analysis done previously, is to model all of these tables in a single document. Therefore, we will proceed by defining, for Logstash, the pipeline which will consist in the definition of:

- Input stage: will report the basic parameters for connecting to PostgreSQL as the driver, the connection string, user, password, and statement
- Filter stage: will define how the data should be modeled
- Output stage: will report the basic parameters for connecting to MongoDB as a connection string, database, and collection

This will allow us to generate documents in accordance with the desired specification:

```
input {
   jdbc {
 	jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.7.0.jar"
 	jdbc_driver_class => "org.postgresql.Driver"
 	jdbc_connection_string => "jdbc:postgresql://postgresql.test.com:5432/formigration?sslmode=disable"
 	jdbc_user => "migratoruser"
 	jdbc_password => "test"
 	statement => "SELECT *
               	FROM enrollment INNER JOIN student ON student.matriculation=enrollment.studentmatriculation
                               	INNER JOIN course ON course.coursecode=enrollment.coursecode
                               	ORDER BY student.matriculation"
   }
 }
 
 filter {
   aggregate {
 	task_id => "%{matriculation}"
 	code => "
         map['matriculation'] = event.get('matriculation')
     	map['address'] = event.get('address')
     	map['birthdate'] = event.get('birthdate')
     	map['email'] = event.get('email')
     	map['name'] = event.get('name')
     	map['phone'] = event.get('phone')
     	map['surname'] = event.get('surname')
     	map['enrollments'] ||= []
     	map['enrollments'] << {'enrollmentid' => event.get('enrollmentid'),  'studentmatriculation' => event.get('studentmatriculation'),  'coursecode' => event.get('coursecode'),  'enrollmentdate' => event.get('enrollmentdate'), 'course' =>  {'coursecode' => event.get('coursecode'), 'coursename' => event.get('coursename'), 'credits'=> event.get('credits'), 'department' => event.get('department')} }
     	event.cancel()
    "
    push_previous_map_as_event => true
  }
 }

 output {
   mongodb {
 	uri => "mongodb://mongodbstandalone.test.com:27017/test"
 	database => "test"
 	collection => "test_migration"
 	isodate => true
   }
 }
```

And the result will be as follows:

![MongoDB Schema Example][3]

This results in the data modeled as documents, but it is also necessary to draw attention to some points:

- How much time/resources will it take to restructure SQL queries into MQL?
- How much effort was required to model the filter part? 
- No data can be written to the source during the migration, otherwise the consistency of the data is lost.
- It will also be useful to define a custom solution for data validation.

Also, it should be noted that Logstash does not allow direct migration of data to the target MongoDB cluster. It is necessary to go through an intermediate machine and then perform a dump/restore from the standalone MongoDB to the target MongoDB Atlas cluster, which will introduce additional effort into the migration.

Referring to the last part of this migration process, it is appropriate to ask:

Would customers be inclined to invest additional economic resources for a bridge server? (Perhaps there might be budget constraints).

Having made several considerations about this architecture, we can move on to discuss the next architecture.

Let's consider this other architecture:

![MongoDB Relational Migrator Architecture][4]

To orchestrate the migration within this architecture, we chose to adopt Relational Migrator, the official product of MongoDB. But what is Relational Migrator?

MongoDB Relational Migrator is a tool to help you migrate relational workloads to MongoDB. Relational Migrator allows you to:

- Design an effective MongoDB schema, derived from an existing relational schema.
- Migrate data from Oracle, SQL Server, MySQL, and PostgreSQL to MongoDB while transforming to the target schema.
- Generate code artifacts to reduce the time required to update application code.

Taking as reference the schema and tables defined previously, once the connection to the relational database is made, we will be able to see, as in the following picture, the visual representation of the relational schema at the top, and below that, the collection modeled by Relational Migrator:

![Schema Modeled by MongoDB Relational Migrator][5]

Once the data model is defined, it will be possible and very easy to define a new sync job. We need to configure the source, destination, and migration option. It will also be possible to set a data validation check and drop the collections that might be the residue of some steps done previously.

![Migration Job Instructions][6]

Once everything is configured, simply start the synchronization job, getting the following result:

![Relational Migrator Results][7]

Last but not least, we have the possibility to use the code app generator and the query converter in private preview!

![Code Generator and Query Converter][8]

![Code Generator and Query Converter][9]

## Conclusion

Armed with what we've discussed so far about the features of the two products, here’s what we can say:

- Relational Migrator
    - Its ease of configuration provides a significant advantage to users, saving them time and effort in implementation. 
    - Its intuitive interface and advanced features make the overall experience more efficient and effective.
    - It represents an ideal solution for those seeking a user-friendly experience.
    - The user can contact support for product-related issues.
- Logstash
    - It allows great flexibility in document modeling.
    - It is possible to define multiple pipelines and run more than one pipeline in the same process. 

Regardless of the solution that will be used to migrate, you will be able to contact MongoDB support to make a successful migration to its infrastructure.

If you have questions or comments, you can find us in the [MongoDB Developer Community](https://www.mongodb.com/community/forums/)!


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb8fd2b6686f0d533/65947c900543c5aba18f25c0/image4.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb675990ee35856de/65947cc42f46f772668248fa/image9.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5a1c48d93e93ae2a/65947d0f13cde9e855200a1f/image1.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta4c41c7a110ed3fa/65947d3ca8ee437ac719978a/image6.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd5909d7af57817b6/65947d7a1f8952fb3f91391a/image5.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5be52451fdb9c083/65947db3254effecce746c7a/image7.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blte510e7b6a9b0a51c/65947dd3a8ee432b5f199799/image3.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt038cad66aa0954dc/65947dfeb0fbcbe233627e12/image8.png
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb204b4ea9c897b43/65947e27a8ee43168219979e/image2.png

Learn how to migrate from a relational database management system (RDBMS) to the Document Model of MongoDB using the MongoDB Relational Migrator utility.

Fabio Ramohitaj

Conteúdo por Fabio Ramohitaj

Soluções de sincronização eficientes: Cluster-to-Cluster Sync e migração em produção para o Atlas

Migração fácil: do banco de dados relacional para o MongoDB com o MongoDB Relational Migrator