MongoDB Replica Docker: Cannot connect on replica, only individual connection

[jellyx (Nedim)]

Hi everyne,

This is how my docker-compose.yml looks like:

version: '3.7'

services:
  mongo-1:
    image: mongo:4.4.2
    container_name: mongo-1
    ports:
      - 27030:27017
    networks: 
      - mongo
    restart: always
    command: /usr/bin/mongod --bind_ip_all --replSet rs0 --journal --dbpath /data/db --enableMajorityReadConcern false
    volumes:
      - ./mongo-1/db:/data/db

  mongo-2:
    image: mongo:4.4.2
    container_name: mongo-2
    ports:
      - 27031:27017
    networks: 
      - mongo
    restart: always
    command: /usr/bin/mongod --bind_ip_all --replSet rs0 --journal --dbpath /data/db --enableMajorityReadConcern false
    volumes:
      - ./mongo-2/db:/data/db

  mongo-3:
    image: mongo:4.4.2
    container_name: mongo-3
    ports:  
      - 27032:27017
    networks: 
      - mongo
    restart: always
    command: /usr/bin/mongod --bind_ip_all --replSet rs0 --journal --dbpath /data/db --enableMajorityReadConcern false
    volumes:
      - ./mongo-3/db:/data/db

volumes:
  mongo-1:
  mongo-2:
  mongo-3:

networks:
  mongo:

I’ll create volumes in production instead biding, but not important right now.

After I execute docker container ls I can see the following:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                      NAMES
4de461506c94        mongo:4.4.2         "docker-entrypoint.s…"   18 seconds ago      Up 16 seconds       0.0.0.0:27031->27017/tcp   mongo-2
2f6ff3f6cbef        mongo:4.4.2         "docker-entrypoint.s…"   18 seconds ago      Up 16 seconds       0.0.0.0:27032->27017/tcp   mongo-3
8a467ad33809        mongo:4.4.2         "docker-entrypoint.s…"   18 seconds ago      Up 16 seconds       0.0.0.0:27030->27017/tcp   mongo-1

and then I do the next:

docker exec -it mongo-1 mongo

which gets me into the mongo shell.

If I execte rs.stats() it will get me the following:

rs.status();
{
“operationTime” : Timestamp(0, 0),
“ok” : 0,
“errmsg” : “no replset config has been received”,
“code” : 94,
“codeName” : “NotYetInitialized”,
“$clusterTime” : {
“clusterTime” : Timestamp(0, 0),
“signature” : {
“hash” : BinData(0,“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”),
“keyId” : NumberLong(0)
}
}
}

Now I want to add replica members and I’ll execute the next:

   rs.initiate({
        _id : 'rs0',
        members: [
            { _id : 0, host : "mongo-1:27017" },
            { _id : 1, host : "mongo-2:27017" },
            { _id : 2, host : "mongo-3:27017" } 
        ]
    });

And this is where problems begin.

I cannot login to replica set, but can on individual nodes. For example, I create a new document in primary replica and without any problems I can see that docment in secondary nodes (when I log in individually).

I tried this on my Mac OS X, but the same thing is on Linux too.

This is my error message:

Cannot connect to replica set "localhost:replica"[localhost:27030].

Set's primary is unreachable.

Reason:

No member of the set is reachable. Reason: Connect failed

and this is how my connection looks:

Members:
localhost:27030
localhost:27031
localhost:27032

Set Name: rs0

Note that this is is Robo 3T. It works on Atlas, for example.

Does anyone know what should I do?

I spent last 2 days on this and I think I checked every tutorial, every YouTube video, bu still nothing.

Seems like Mongo Dockerfile is hard coded to expose it to port 27017, so it’s not possible to change that.

I also tried with:

extra_hosts:
 - mongo-1:127.0.0.1

in each service but nothing.

Thanks.

[chris (Chris Dellaway)]

See if you can make this work for you.

Updates the hosts file to match your replicas.

Unable to connect using replicas uri (docker setup)

If you are connecting to admin the databases then connecting to a primary only is adequate. When connecting to a replica set the client has to be able to connect to each node as defined in the replica set configuration. When connecting to a replicaSet the hosts listed (explicit or implicit mongodb+srv) are the seed hosts for the client. The client will then discover the hosts in the cluster from that seed node. If you still want to connect ‘properly’ to the replica set, you can update the po…

[jellyx (Nedim)]

Nothing. I already tried this by changing /etc/hosts.

I also created username, password, and database. I can log in directly (without replica).

[chris (Chris Dellaway)]

The ports in your replicaset have to change too, to match the exposed ports. With a replicaset it is the config in the replica set that a client will connect to.

I prefer to use different ip’s and keep 27017 as the port.

My ports in compose look like:

ports extract docker-compose.yaml

services:
  mongo-0-a:
    ports:
    - 127.0.10.1:27017:27017
...
  mongo-0-b:
    ports:
    - 127.0.10.2:27017:27017
...
  mongo-0-c:
    ports:
    - 127.0.10.3:27017:27017
...

Full Compose for reference

version: '3.7'

services:
  mongo-0-a:
    image: mongo:4.4
    ports:
      - 127.0.10.1:27017:27017
    volumes:
      - mongo-0-a:/data/db
    restart: unless-stopped
    command: "--wiredTigerCacheSizeGB 0.25 --replSet rs0"

  mongo-0-b:
    image: mongo:4.4
    ports:
      - 127.0.10.2:27017:27017
    volumes:
      - mongo-0-b:/data/db
    restart: unless-stopped
    command: "--wiredTigerCacheSizeGB 0.25 --replSet rs0"

  mongo-0-c:
    image: mongo:4.4
    ports:
      - 127.0.10.3:27017:27017
    volumes:
      - mongo-0-c:/data/db
    restart: unless-stopped
    command: "--wiredTigerCacheSizeGB 0.25 --replSet rs0"
volumes:
  mongo-0-a:
  mongo-0-b:
  mongo-0-c:

/etc/hosts

...
127.0.10.1 mongo-0-a
127.0.10.2 mongo-0-b
127.0.10.3 mongo-0-c
...

This command on the secondary below returns the hosts of the replicaset a client must connect to. These hostnames must resolve and the ports have to match. This is why exposing on a different port to what is in the rs.conf() does not work.

ismaster() on secondary

mongo --host 127.0.10.2 --eval 'var m = db.isMaster(); print("Is Primary?", m.ismaster); print("hosts:", m.hosts); print("Primary:", m.primary)' --quiet
Is Primary? false
hosts: mongo-0-a:27017,mongo-0-b:27017,mongo-0-c:27017
Primary: mongo-0-c:27017

Connecting to replicaSet:

ReplicaSet Connect

mongo --host rs0/127.0.10.2 --quiet
2020-12-09T15:25:53.458-0500 I  NETWORK  [js] Starting new replica set monitor for rs0/127.0.10.2:27017
2020-12-09T15:25:53.458-0500 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to 127.0.10.2:27017
2020-12-09T15:25:53.460-0500 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to mongo-0-c:27017
2020-12-09T15:25:53.463-0500 I  NETWORK  [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for rs0 is rs0/mongo-0-a:27017,mongo-0-b:27017,mongo-0-c:27017
2020-12-09T15:25:53.463-0500 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to mongo-0-a:27017
2020-12-09T15:25:53.463-0500 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to mongo-0-b:27017

[jellyx (Nedim)]

Many thanks for this!

I also found another working solution here:

Connect to MongoDB Replica Set running inside Docker with Java (Windows)

By user Sydney.

It says:

Adding the hostnames to the hosts file did not work for me. I think if all hostnames refers to the same host IP (e.g. 127.0.0.1), it's not going to work if all docker ports are the same (e.g. 27017). The replica set is composed by  `mongo1:27017, mongo2:27017 and mongo3:27017`  inside docker. Outside docker it corresponds to  `127.0.0.1:27017, 127.0.0.1:27017 and 127.0.0.1:27017` which won't work. To fix the issue I had to set a different port for each node.

```
docker network create mongo-cluster
docker run --name mongo1 -d --net mongo-cluster -p 9042:9042 mongo:3.6 mongod --replSet docker-rs --port 9042
docker run --name mongo2 -d --net mongo-cluster -p 9142:9142 mongo:3.6 mongod --replSet docker-rs --port 9142
docker run --name mongo3 -d --net mongo-cluster -p 9242:9242 mongo:3.6 mongod --replSet docker-rs --port 9242
docker exec -it mongo1 mongo --port 9042
config = {"_id" : "docker-rs", "members" : [{"_id" : 0,"host" : "mongo1:9042"},{"_id" : 1,"host" : "mongo2:9142"},{"_id" : 2,"host" : "mongo3:9242"}]}
rs.initiate(config)
rs.status() 
```

and finally add the hostnames to the hosts file

```
127.0.0.1 mongo1 mongo2 mongo3
```

I tested on 4.4.2 and it works. Anyway, I’ll go with your solution because I like it more.

However, I wonder why solution from this guy works when it uses ports 9042, 9142, 9242, but does’t work when I set 27030, 27031, 27032, for example. Or even 8042, 8142, 8242.

Do yo know why it happens?

As I said, I won’t go with this solution, but just too curios.

Thanks!

[chris (Chris Dellaway)]

Because they init the replicaset with the same ports as the ones exposed.

@jellyx

config = {"_id" : "docker-rs", "members" : [{"_id" : 0,"host" : "mongo1:9042"},{"_id" : 1,"host" : "mongo2:9142"},{"_id" : 2,"host" : "mongo3:9242"}]}

@jellyx

I tested on 4.4.2 and it works. Anyway, I’ll go with your solution because I like it more.

Using ip’s in the 127.0.0.0/8 is great, its a whole /8 network!! And you can use default ports over and over again.

[jellyx (Nedim)]

Yes, but I changed everything. Wherever 9042 is mentioned I changed it. In docker run, in mongo shell when creating an initialization, etc.

My rs.status() returns members that container 8042, 8142, 8242, for example.

EDIT: I missed something. It’s okay now. Thank you again for your help.

[jellyx (Nedim)]

Ah, something weird is happening:

ERROR: for docker-mongodb_mongo-0-a_1  Cannot start service mongo-0-a: Ports are not available: listen tcp 127.0.10.1:27017: bind: can't assign requested address

ERROR: for mongo-0-c  Cannot start service mongo-0-c: Ports are not available: listen tcp 127.0.10.3:27017: bind: can't assign requested address

ERROR: for mongo-0-b  Cannot start service mongo-0-b: Ports are not available: listen tcp 127.0.10.2:27017: bind: can't assign requested address

ERROR: for mongo-0-a  Cannot start service mongo-0-a: Ports are not available: listen tcp 127.0.10.1:27017: bind: can't assign requested address