If you are connecting to admin the databases then connecting to a primary only is adequate.
When connecting to a replica set the client has to be able to connect to each node as defined in the replica set configuration.
When connecting to a replicaSet the hosts listed (explicit or implicit mongodb+srv) are the seed hosts for the client. The client will then discover the hosts in the cluster from that seed node.
If you still want to connect ‘properly’ to the replica set, you can update the ports of the mongod to match the exposed ports and update your system’s hosts file.
127.0.0.1 mongo-primary
127.0.0.1 mongo-secondary
127.0.0.1 mongo-arbiter