What is the best configuration for DR availability

Ops and Admin
1

I need to configure DR for a mongoDB replicaset.
I need high availability both in the main site, and in the DR site.

My solution was to create:
Primary and a secondary in the MAIN site
Two secondaries and an one arbiter in the DR site.
For the primary in the MAIN site I gave the highest priority -2,
For the secondary in the MAIN site I gave priority 1.5
For one of the secondaries in the DR site I gave priority 1
For the other secondary in the DR site I gave priority 0.5

So, the failover scenario logic was this -
in case the primary fails in the MAIN site, the secondary MAIN site will become the new primary.
In case the whole MAIN site goes down - the highest priority secondary on the DR site will become the new Primary.

However, I discovered that in case the DR site goes down, and all its 3 members are unreachable, the two members on the main site becomes secondaries, and the log says it cannot vote for a primary since there is no majority:

2024-01-11T15:36:34.427+0300 I NETWORK [listener] connection accepted from 10.96.1.102:38290 #67011 (44 connections now open)
2024-01-11T15:36:34.460+0300 I NETWORK [conn66784] end connection
2024-01-11T15:36:44.445+0300 I ELECTION [replexec-116658] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)

2024-01-11T15:36:55.863+0300 I ELECTION [replexec-116659] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
2024-01-11T15:36:57.479+0300 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set octrs0

So - what is the best configuration to cover all scenarios for a MongoDB DR?

Thanks

2

This is one of the challenges to MongoDB it requires 3 sites for true HA/DR, with 2 DCs there is no way to have automatic DR, if the majority of nodes goes down then you will be in a Read Only mode until you reconfigure the replica set.

Options:

  1. Put a node in a 3rd site if available with lowest priority that way no matter what site goes down you have a majority of nodes.
  2. Have some RO nodes in the non majority site, they don’t count as part of the voting, so if the majority of nodes goes down, you can run a rs.reconfigure() to change the priority of the RO nodes so you have a majority back.

Summary: With 2 sites you need manual intervention if the majority of nodes goes down
3 sites solves this problem and it is what MongoDB recommends in their white paper on Operational Best Practices

Consider the following factors when developing the
architecture for your replica set:
• Ensure that the members of the replica set will always
be able to elect a primary. A strict majority of voting
cluster members must be available and in contact with
each other to elect a new primary. Therefore you should
run an odd number of members. There should be at
least three replicas with copies of the data in a replica
set.
• Best practice is to have a minimum of 3 data centers so
that a majority is maintained after the loss of any single
site. If only 2 sites are possible then know where the
majority of members will be in the case of any network
partitions and attempt to ensure that the replica set can
elect a primary from the

2 Likes
3

Hi @Eli Brunsvold (@tapiocaPENGUIN),
Thank you very much for the detailed explanation.

Regarding option 2:
If I add another Read-Only member on the MAIN site, and the majority of nodes goes down (on the DR site) so I am left with only 2 read-write members on the MAIN site and one read-only member on the MAIN site -
the idea is to use rs.reconfigure() in order to turn the read-only member to a read-write member? Would that be enough for it to be counted as eligible for voting and the replica set will have a Primary again?

Thanks,
Tamar

4

Hi again @Eli Brunsvold (@tapiocaPENGUIN),

Got it now - the idea is to keep the read-only member as it is, but reconfigure its priority to a higher value, say 1, so it can participate in the election, and a majority of voting cluster members is available.
I gather that at first after the DR site crash the 2 read-write nodes in the MAIN site will turn into secondaries.
But after the manual read-only priority reconfiguration of the read-only - will the member with the highest priority, the previous Primary, will become automatically Primary again?

Thanks,
Tamar