Python connection to replicaset

Ops and Admin Replication
, , , ,
1

Hi, im trying to dev my local mongodb. For that im using docker inside ubuntu and python. I first started to initiate one simple database using docker-compose.

image
image801×572 15.1 KB

And made a verry simple interface in python.
It works perfectly, so i tried to make a replicaset for some reason im not able to use docker-compose for replicaset, i saw some documentation online and also used that documention and always happened the same! The containers are created but unable to run it but dont shows any error. So i ran it step by step and understood that if i run the mongo containers with enviroments would happen the same so i run the simple following commands:

sudo docker run -p27017:27017 -d --name m1 --net mongonet mongo --replSet myset

sudo docker run -p27027:27017 -d --name m2 --net mongonet mongo --replSet myset

sudo docker run -p27037:27017 -d --name m3 --net mongonet mongo --replSet myset
sudo docker exec -it m1 mongo
config = { "_id" : "myset", "members" : [ { "_id" : 0, "host" : "m1:27017" }, { "_id" : 1, "host" : "m2:27017" }, { "_id" : 2, "host" : "m3:27017" }] }

rs.initiate(config)

Tested if it was working and all good for now.
So i tryed again the connection with the python app and it showed this:

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\tkinter\__init__.py", line 1921, in __call__
    return self.func(*args)
  File "C:\Users\Asus\PycharmProjects\pythonProject1\interface.py", line 14, in transfer
    db_collection.insert_one(data)
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\collection.py", line 524, in insert_one
    self._insert_one(
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\collection.py", line 474, in _insert_one
    self.__database.client._retryable_write(
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\mongo_client.py", line 1339, in _retryable_write
    with self._tmp_session(session) as s:
  File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\mongo_client.py", line 1616, in _tmp_session
    s = self._ensure_session(session)
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\mongo_client.py", line 1603, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\mongo_client.py", line 1553, in __start_session
    server_session = self._get_server_session()
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\mongo_client.py", line 1589, in _get_server_session
    return self._topology.get_server_session()
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\topology.py", line 530, in get_server_session
    session_timeout = self._check_session_support()
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\topology.py", line 514, in _check_session_support
    self._select_servers_loop(
  File "C:\Users\Asus\PycharmProjects\pythonProject1\venv\lib\site-packages\pymongo\topology.py", line 216, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: m2:27017: [Errno 11001] getaddrinfo failed,m1:27017: [Errno 11001] getaddrinfo failed,m3:27017: [Errno 11001] getaddrinfo failed, Timeout: 30s, Topology Description: <TopologyDescription id: 622c1970456f0a8a08b35096, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('m1', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('m1:27017: [Errno 11001] getaddrinfo failed')>, <ServerDescription ('m2', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('m2:27017: [Errno 11001] getaddrinfo failed')>, <ServerDescription ('m3', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('m3:27017: [Errno 11001] getaddrinfo failed')>]>

Searched a solution but nothing.
I dont know what more i can do and im stuck.Can you guys help me?

Here is the code used in the python app:

#bibliotecas
import tkinter as tk
import pymongo
#variaveis referentes ao mongo
url="mongodb://admin:password@localhost:27017"
client = pymongo.MongoClient(url)
data_base = client.utad
coll=data_base.utad
db_collection = data_base.alunos_utad
db_col2=data_base.novo_doc
#função para tranferir os dados para a db quando o button save é premido
def transfer():
    data={"name":input0.get(),"sobrenome":input.get(),"email":input2.get(),"Al":input1.get(),"curso":input3.get(),}
    db_collection.insert_one(data)
def search():
    y=input0.get()
    t=input.get()
    q=input1.get()
    w=input2.get()
    r=input3.get()
    if t==""and q=="" and w=="" and r==""and y!="":
        data = {"name": input0.get()}
        x=db_collection.find_one(data)
        print(x)
        label4["text"]=x
    if y==""and q=="" and w=="" and r=="" and t!="":
        data = {"sobrenome": input.get()}
        x=db_collection.find_one(data)
        print(x)
        label4["text"]=x
    if t==""and y=="" and w=="" and r==""and q!="":
        data = {"Al":input1.get()}
        x=db_collection.find_one(data)
        print(x)
        label4["text"] = x
    if t==""and q=="" and y=="" and r=="" and w!="":
        data = {"email": input2.get()}
        x=db_collection.find_one(data)
        print(x)
        label4["text"] = x
    if t==""and q=="" and w=="" and y=="" and r!="":
        data = {"curso": input3.get()}
        x=db_collection.find_one(data)
        print(x)
        label4["text"] = x
    if y==""and t=="" and q=="" and r=="" and w=="":
        print("Nenhum valor inserido imprimir tudo na collection")
        for x in db_collection.find():
            print(x)
            label4["text"] = x
    else:
        data = {"name": input0.get(), "sobrenome": input.get(), "email": input2.get(), "Al": input1.get(),
                "curso": input3.get()}
        x=db_collection.find_one(data)
        print(x)
        label4["text"] = x


#criar uma janela grafica
root= tk.Tk()
#definir o tamanho da janela
canvas= tk.Canvas(root,height=500,width=400)
canvas.pack()
#definir a framework
frame=tk.Frame(root)
frame.place(relwidth=1,relheight=1)
#colocar um button
button= tk.Button(root, text="Save",command=transfer)
button.pack()


#labels e caixas de texto relacionadas a entroduzir dados
label=tk.Label(frame, text="Interface para introduzir dados")
label.pack()
lab=tk.Label(frame, text="Nome")
lab.pack()
input0=tk.Entry(frame, bg="red")
input0.pack()
lab=tk.Label(frame, text="Sobrenome")
lab.pack()
input=tk.Entry(frame, bg="red")
input.pack()
lab1=tk.Label(frame, text="Al")
lab1.pack()
input1=tk.Entry(frame, bg="red")
input1.pack()
lab2=tk.Label(frame, text="email")
lab2.pack()
input2=tk.Entry(frame, bg="red")
input2.pack()
lab3=tk.Label(frame, text="curso")
lab3.pack()
input3=tk.Entry(frame, bg="red")
input3.pack()

#procurar um elemento qualquer
label4=tk.Label(frame,bg="blue")
label4.pack()
button1= tk.Button(root, text="Find",command=search)
button1.pack()

#iniciar a aplicação
root.mainloop()
2

Hi @Alexandre_Sousa and welcome in the MongoDB Community :muscle: !

First of all, it doesn’t make sense to try to create a 3 node replica set on a single machine. For development purposes, you can use a single node replica set on your local machine and it’s more than enough to make everything works, just like any other MongoDB replica set out there.

Your network issue comes from the fact that you are running your 3 node replica set in a sub network “mongonet” where m1, m2 and m3 make sense. But they don’t make any sense from your host perspective which is where you are executing your python code.
The first things the pymongo driver doesn’t when it tries to connect is detect the cluster topology. From the rs.conf() commands, it gets the IP address as they are defined in the cluster topology: m1, m2 and m3. That’s why you are seing them in the logs.
But because these can’t be resolve in your host, it doesn’t work.

So one way to “solve” the problem would be to make the host able to resolve these… By editing your /etc/hosts file for example:

[...]
127.0.0.1	m1
127.0.0.1	m2
127.0.0.1	m3
[...]

And if I execute this python code, I can actually connect:

from pymongo import MongoClient
client = MongoClient('mongodb://localhost:27017,localhost:27027,localhost:27037/test?replicaSet=myset&w=majority')
coll = client.get_database('test').get_collection('coll')
coll.insert_one({'name':'Max'})
print(coll.find_one())

Note that your URL was also not correct in your python code. Mine is correct here, given your 3 node cluster setup with the network config.

Also note that removing the 3 lines from the /etc/hosts files breaks this python code and I can’t resolve m1, m2 and m3 without it, even if I’m actually NOT using them in the connection string. Because pymongo finds them in the cluster config.

Now. How I would actually fix this, because it’s a lot more complex than it should be in my opinion.

1/ Only one single replica set node is enough.
2/ No need for a docker sub network if you reuse the same hostname.

Let’s first remove your docker containers:

docker stop m1 m2 m3
docker rm -v m1 m2 m3
docker network rm mongonet

Here is the aliases I use to start my MongoDB Docker container. Add them in your ~/.bashrc or equivalent:

alias mdb='docker run --rm -d -p 27017:27017 -h $(hostname) --name mongo mongo:5.0.6 --replSet=test && sleep 4 && docker exec mongo mongo --eval "rs.initiate();"'
alias msh='docker exec -it mongo mongosh --quiet'

Then you can just execute the command mdb to create a single node RS which will initiate after 4 seconds (if your PC is slow, add some seconds so mongod has time to start).
You can connect using mongosh with msh if you want to check the rs.status(), etc.

Now we can connect to this development cluster without even specifying a connection string because it’s running on localhost:27017 and because of the -h $(hostname) option, my config looks like this:

test [direct: primary] test> rs.conf()
{
  _id: 'test',
  version: 1,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'hafx:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
      votes: 1
    }
  ],
  protocolVersion: Long("1"),
  writeConcernMajorityJournalDefault: true,
  settings: {
    chainingAllowed: true,
    heartbeatIntervalMillis: 2000,
    heartbeatTimeoutSecs: 10,
    electionTimeoutMillis: 10000,
    catchUpTimeoutMillis: -1,
    catchUpTakeoverDelayMillis: 30000,
    getLastErrorModes: {},
    getLastErrorDefaults: { w: 1, wtimeout: 0 },
    replicaSetId: ObjectId("622c4663da741b92fe426db8")
  }
}

You can see that “hafx” (which is my hostname) was reused and my host can resolve its own hostname back to itself: localhost (well else you need to fix /etc/hosts again because it should be in there already!).

The same python script now works without a connection string:

from pymongo import MongoClient
client = MongoClient()
coll = client.get_database('test').get_collection('coll')
coll.insert_one({'name':'Max'})
print(coll.find_one())

I hope I made sense :slight_smile:.

Cheers,
Maxime.

3

I know it doesn t make sense make it local but, this is my project for college and in the first stage i had to try it locally to test reading speed and things. Want to test change streams and for that need a replicaset active. But thanks for the help i will try it.

4

A single node replica set is enough to support Change Streams, ACID transactions etc. You don’t need 3 nodes to get that support.

The only “valid” reason I see to have 3 nodes of a RS deployed on a single machine is for educational purposes :-).
Just to learn the logic and understand how the system works.

Here is how I would deploy a 3 node RS on a single host for learning purposes ONLY :sweat_smile: !

This is my deploy.sh script:

#!/usr/bin/env bash

# cleaning
echo "Cleaning old containers and networks if any."
docker container stop $(docker container ls -q --filter name=mongo*)
docker network rm mongonet

# create RS
echo "Create network 'mongonet'."
docker network create mongonet
echo "Creating the 3 MongoDB containers..."
docker run --rm -d -p 27017:27017 -h $(hostname) --name mongo1 --net mongonet mongo:5.0.6 --replSet=myset
docker run --rm -d -p 27018:27017 -h $(hostname) --name mongo2 --net mongonet mongo:5.0.6 --replSet=myset
docker run --rm -d -p 27019:27017 -h $(hostname) --name mongo3 --net mongonet mongo:5.0.6 --replSet=myset

# sleep a bit so the container have enough time to initialize
echo "Waiting for containers to start..."
sleep 10

# init the RS config using the first container
echo "Sending the rs.init(...) command."
echo 'rs.initiate({
      _id: "myset",
      members: [
         { _id: 0, host: "mongo1:27017" },
         { _id: 1, host: "mongo2:27017" },
         { _id: 2, host: "mongo3:27017" }]});' | docker exec -i mongo1 mongosh --quiet

# giving some time to elect the Primary
echo "Waiting for Primary election..."
sleep 10

echo 'rs.status();' | docker exec -i mongo1 mongosh --quiet

This is my python script python.py:

from pymongo import MongoClient
client = MongoClient('mongodb://mongo1:27017,mongo2:27017,mongo3:27017/test?replicaSet=myset&w=majority')
coll = client.get_database('test').get_collection('coll')
coll.insert_one({'name':'Max'})
print(coll.find_one())

And this is my requirements.txt file:

pymongo[srv]

I need to create an image with pymongo[srv] and the python script. This container will run in the same network than the cluster. So here is the Dockerfile:

FROM python:3.10.2-alpine
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY python.py ./
CMD [ "python", "./python.py" ]

Command lines to create the python image & run the container:

docker build -t my-mdb-python .
docker run -it --rm --name my-mdb-python --net mongonet my-mdb-python

If everything works, it should just say something like this (with a different _id):

{'_id': ObjectId('622f84bb36648ea4546dd03a'), 'name': 'Max'}

I hope this works for you too :smiley: !

Cheers,
Maxime.

1 Like