ServerSelectionError when connecting dockerized replica set

Sewook_Kim · February 15, 2023, 12:59pm

I made three dockerized mongo container, one primary and two secodaries. But, I can’t connect to the DBs. I get Server Selection Error either “server selection timed out” or “getaddrinfo ENOTFOUND”.

I’m pretty much sure that replica set is fine, because when I use the connection string like this,

“mongodb://localhost:27001/MeshedDev?directConnection=true&replicaSet=myReplicaSet”

this works. However, when I restart the server and if the primary replica has been changed, this connection string will fail unless I change the port number manually.

So, ideally, I want to list the members so that the server can find the primary by itself. I tried some different formats of connection string according to connection formatting doc of yours and some other posts I found about similar issues, but none of them worked for me.

I feel, I’m missing something with docker, finding the right host address, but cannot figure it out.

Any help with connection string uri please? Below are some logs

connection strings I’ve tried examples

"mongodb://localhost:27001,localhost:27002,localhost:27003/MeshedDev?replicaSet=myReplicaSet"
"mongodb://127.0.0.1:27001,127.0.0.1:27002,127.0.0.1:27003/MeshedDev?replicaSet=myReplicaSet"
"mongodb://mongo1.127.0.0.1:27001,mongo2.127.0.0.1:27002,mongo3.127.0.0.1:27003/MeshedDev?replicaSet=myReplicaSet"
"mongodb://mongo1:27001,mongo2:27002,mongo3:27003/MeshedDev?replicaSet=myReplicaSet"
"mongodb://mongo1:27017,mongo2:27017,mongo3:27017/MeshedDev?replicaSet=myReplicaSet"

server error logs

MongooseServerSelectionError: Server selection timed out after 30000 ms
    at NativeConnection.Connection.openUri (/Users/me/Meshed/server/node_modules/mongoose/lib/connection.js:825:32)
    at /Users/me/Meshed/server/node_modules/mongoose/lib/index.js:409:10
    at /Users/me/Meshed/server/node_modules/mongoose/lib/helpers/promiseOrCallback.js:41:5
    at new Promise (<anonymous>)
    at promiseOrCallback (/Users/me/Meshed/server/node_modules/mongoose/lib/helpers/promiseOrCallback.js:40:10)
    at Mongoose._promiseOrCallback (/Users/me/Meshed/server/node_modules/mongoose/lib/index.js:1262:10)
    at Mongoose.connect (/Users/me/Meshed/server/node_modules/mongoose/lib/index.js:408:20)
    at mongoConnect (/Users/me/Meshed/server/src/database/mongo/connection.ts:9:12)
    at Object.<anonymous> (/Users/me/Meshed/server/src/app.ts:35:13)
    at Module._compile (node:internal/modules/cjs/loader:1218:14) {
  reason: TopologyDescription {
    type: 'ReplicaSetNoPrimary',
    servers: Map(3) {
      'mongo1:27017' => [ServerDescription],
      'mongo2:27017' => [ServerDescription],
      'mongo3:27017' => [ServerDescription]
    },
    stale: false,
    compatible: true,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    setName: 'myReplicaSet',
    maxElectionId: ObjectId { [Symbol(id)]: [Buffer [Uint8Array]] },
    maxSetVersion: 1,
    commonWireVersion: 0,
    logicalSessionTimeoutMinutes: null
  },
  code: undefined
}

replica status

{
  set: 'myReplicaSet',
  date: ISODate("2023-02-15T12:39:21.928Z"),
  myState: 1,
  term: Long("7"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
    lastCommittedWallTime: ISODate("2023-02-15T12:39:13.254Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
    appliedOpTime: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
    durableOpTime: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
    lastAppliedWallTime: ISODate("2023-02-15T12:39:13.254Z"),
    lastDurableWallTime: ISODate("2023-02-15T12:39:13.254Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1676464723, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: ISODate("2023-02-15T11:54:12.530Z"),
    electionTerm: Long("7"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1676461982, i: 1 }), t: Long("6") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2023-02-15T11:54:12.584Z"),
    wMajorityWriteAvailabilityDate: ISODate("2023-02-15T11:54:12.999Z")
  },
  members: [
    {
      _id: 0,
      name: 'mongo1:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 2732,
      optime: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
      optimeDate: ISODate("2023-02-15T12:39:13.000Z"),
      lastAppliedWallTime: ISODate("2023-02-15T12:39:13.254Z"),
      lastDurableWallTime: ISODate("2023-02-15T12:39:13.254Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1676462052, i: 1 }),
      electionDate: ISODate("2023-02-15T11:54:12.000Z"),
      configVersion: 1,
      configTerm: 7,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'mongo2:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 2719,
      optime: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
      optimeDurable: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
      optimeDate: ISODate("2023-02-15T12:39:13.000Z"),
      optimeDurableDate: ISODate("2023-02-15T12:39:13.000Z"),
      lastAppliedWallTime: ISODate("2023-02-15T12:39:13.254Z"),
      lastDurableWallTime: ISODate("2023-02-15T12:39:13.254Z"),
      lastHeartbeat: ISODate("2023-02-15T12:39:21.153Z"),
      lastHeartbeatRecv: ISODate("2023-02-15T12:39:21.828Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mongo1:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 7
    },
    {
      _id: 2,
      name: 'mongo3:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 2707,
      optime: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
      optimeDurable: { ts: Timestamp({ t: 1676464753, i: 1 }), t: Long("7") },
      optimeDate: ISODate("2023-02-15T12:39:13.000Z"),
      optimeDurableDate: ISODate("2023-02-15T12:39:13.000Z"),
      lastAppliedWallTime: ISODate("2023-02-15T12:39:13.254Z"),
      lastDurableWallTime: ISODate("2023-02-15T12:39:13.254Z"),
      lastHeartbeat: ISODate("2023-02-15T12:39:21.153Z"),
      lastHeartbeatRecv: ISODate("2023-02-15T12:39:21.342Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mongo2:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 7
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1676464753, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1676464753, i: 1 })
}

docker ps

CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS                      NAMES
569f32314512   mongo:latest   "docker-entrypoint.s…"   24 minutes ago   Up 24 minutes   0.0.0.0:27003->27017/tcp   mongo3
1efd0830a5c5   mongo:latest   "docker-entrypoint.s…"   24 minutes ago   Up 24 minutes   0.0.0.0:27002->27017/tcp   mongo2
2f4ef16a31b9   mongo:latest   "docker-entrypoint.s…"   24 minutes ago   Up 24 minutes   0.0.0.0:27001->27017/tcp   mongo1

mongo1 docker terminal log (for a single connection attempt)

2023-02-15 21:00:33 
{
  "t": { 
    "$date" : "2023-02-15T12:00:33.824+00:00"
  },
  "s":"I",  
  "c":"NETWORK",  
  "id":22943,   
  "ctx":"listener",
  "msg":"Connection accepted",
  "attr": {
    "remote":"172.19.0.1:48010",
    "uuid":"c54cf73c-b5c6-480b-b6e0-1cc2caaa1298",
    "connectionId":29,
    "connectionCount":13
  }

}
2023-02-15 21:00:33 
{
  "t": {
    "$date":"2023-02-15T12:00:33.834+00:00"
  },
  "s":"I",  
  "c":"NETWORK",  
  "id":51800,   
  "ctx":"conn29",
  "msg":"client metadata",
  "attr": {
    "remote":"172.19.0.1:48010",
    "client":"conn29",
    "doc":{
      "driver":{
        "name":"nodejs|Mongoose",
        "version":"4.12.1"
      },
      "os": { 
        "type":"Darwin",
        "name":"darwin",
        "architecture":"arm64",
        "version":"22.1.0"
      },
      "platform":"Node.js v18.13.0, LE (unified)",
      "version":"4.12.1|6.8.3"
    }
  }
}

2023-02-15 21:00:33 
{
  "t": {
    "$date":"2023-02-15T12:00:33.842+00:00"
  },
  "s":"I",  
  "c":"NETWORK",  
  "id":22944,   
  "ctx":"conn29",
  "msg":"Connection ended",
  "attr":{
    "remote":"172.19.0.1:48010",
    "uuid":"c54cf73c-b5c6-480b-b6e0-1cc2caaa1298",
    "connectionId":29,
    "connectionCount":12
  }
}

steevej · February 15, 2023, 1:45pm

The only connection string that corresponds to the following replica set configuration:

Sewook_Kim:

 servers: Map(3) {
      'mongo1:27017' => [ServerDescription],
      'mongo2:27017' => [ServerDescription],
      'mongo3:27017' => [ServerDescription]
    },
    stale: false,
    compatible: true,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    setName: 'myReplicaSet',

is

Please share what you get when you use it.

Sewook_Kim · February 15, 2023, 2:11pm

Thanks for your reply.

I still get timed out error.

MongooseServerSelectionError: Server selection timed out after 30000 ms
    at NativeConnection.Connection.openUri (/Users/me/Meshed/server/node_modules/mongoose/lib/connection.js:825:32)
    at /Users/me/Meshed/server/node_modules/mongoose/lib/index.js:409:10
    at /Users/me/Meshed/server/node_modules/mongoose/lib/helpers/promiseOrCallback.js:41:5
    at new Promise (<anonymous>)
    at promiseOrCallback (/Users/me/Meshed/server/node_modules/mongoose/lib/helpers/promiseOrCallback.js:40:10)
    at Mongoose._promiseOrCallback (/Users/me/Meshed/server/node_modules/mongoose/lib/index.js:1262:10)
    at Mongoose.connect (/Users/me/Meshed/server/node_modules/mongoose/lib/index.js:408:20)
    at mongoConnect (/Users/me/Meshed/server/src/database/mongo/connection.ts:8:12)
    at Object.<anonymous> (/Users/me/Meshed/server/src/app.ts:35:13)
    at Module._compile (node:internal/modules/cjs/loader:1218:14) {
  reason: TopologyDescription {
    type: 'ReplicaSetNoPrimary',
    servers: Map(3) {
      'mongo1:27017' => [ServerDescription],
      'mongo2:27017' => [ServerDescription],
      'mongo3:27017' => [ServerDescription]
    },
    stale: false,
    compatible: true,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    setName: 'myReplicaSet',
    maxElectionId: null,
    maxSetVersion: null,
    commonWireVersion: 0,
    logicalSessionTimeoutMinutes: null
  },
  code: undefined
}

steevej · February 15, 2023, 2:48pm

It looks like your replica set is not configured correctly.

Connect with the same connection string but with mongosh or Compass. It is funny that mongoose report ReplicatSetNoPrimary yet you also shared

which indicates that mongo1 is PRIMARY.

Note that running all the nodes of a replica set on the same hardware is not really safe. If this hardware fails you lose your data.

Sewook_Kim · February 15, 2023, 4:06pm

First, thanks for your kind advice. I’m aware of it and will have separate nodes in separate instances for actual deployment.

Your connection string works well with mongosh. This might be an awkward question, but how am I suppose to access the DB from my application then? I still can’t connect the DB from my application with any connection string, nor send a query without server connected to the DB.

By the way, my server is not in the docker and just running in local. I just realized that this might confused you and is the reason why container names weren’t working. Still, it’s weird because I can connect to DBs and send transactions with
"mongodb://localhost:27001/MeshedDev?directConnection=true&replicaSet=myReplicaSet" but not with any other listed form of connection string such as "mongodb://localhost:27001,localhost:27002,localhost:27003/MeshedDev?replicaSet=myReplicaSet" which I think should work.

Sewook_Kim · February 15, 2023, 4:25pm

I just found this post(Mongoose unable to connect with Mongo Standalone replicaSet - #2 by kevinadi) suggesting some options to try out.

let opt = {poolSize:1, useNewUrlParser: true, useUnifiedTopology: true}
let conn = await MongoClient.connect(
    'mongodb://username:password@127.0.0.1:27017/test?authSource=admin&replicaSet=rs0',
    opt)

However, this makes error “saying option poolSize is not supported” and neither can I find other properties in MongoClientOptions in mongodb.d.ts.

I thought this was because the post was using MongoDB 5.0 when I’m using 6.0, but couldn’t find any reference to those options above in 5.0 doc(https://www.mongodb.com/docs/v5.0/reference/connection-string/#connection-options).

Does anyone know what happened since last summer?

Nomad_Family · February 15, 2023, 6:00pm

I’m getting the following error on my website. It’s a new website.

{“status”: false, “message”: "MongoServerSelectionError:

connect EADDRNOTAVAIL 127.0.0.1:27017 - Local (127.0.0.1:0) In at Timeout. onTimeout (/var/www/CertificationPlannerNode/node modules/mongo db/lib/sdam/topology.is:292:38)In at listOnTimeout (internal/timers.js:554:17) In

at processTimers

(internal/timers.js:497:7)", “data”:

steevej · February 15, 2023, 6:04pm

Please do not pollute an existing thread with a question you already posted elsewhere.

Thanks

steevej · February 15, 2023, 8:04pm

This confirms that your replica set is alive and configured correctly.

Is your application running on the same machine as the one you used to connect with mongosh?

Since the error is

I suspect that the mongoose version you are using is incompatible with the node driver. Can you try a different version, try with a newer otherwise try an older one.

It is not really weird because you are specifying directConnection=true and you are doing ports forwarding

With directConnection=true, the driver does not use the replica set configuration, it connects directly.

steevej · February 15, 2023, 11:39pm

The thread Access mongodb via External IP - #3 by Stennie should be of interest to you.