Connection to an exposed replicaset is not working

Description:

I exposed my mongodb cluster using istio service mesh and a virtualservice (see below). I can open a connection to every single node. When I try to connect to the replicaset, I get an error message ==>>
MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-1.my-cluster-name-rs0.percona.svc.cluster.local

Seems this is the same problem anounced here: Connection Issues When Accessing MongoDB Replica Set from Outside the Kubernetes Cluster

Please note, that we are not able to use loadbalancer ip’s as we do not have a lot of free and useable ip’s. That’s the reason for using a virtualservice and mapping different ports to different services. We cannot use the “clusterServiceDNSMode: External” workaround therefore.

The Blog also mentioned split DNS. The documentation is very short and unclear about this. Could you explain in detail what is going on behind the scences when I configure horizons like this?

replsets:
  - name: rs0
    expose:
      enabled: true
      exposeType: ClusterIP
    horizons:
      my-cluster-name-rs0-0:
        external: my-cluster-name-rs0-0.eng.cmp.szh.loc
      my-cluster-name-rs0-1:
        external: my-cluster-name-rs0-1.eng.cmp.szh.loc
      my-cluster-name-rs0-2:
        external: my-cluster-name-rs0-2.eng.cmp.szh.loc

I also read following limitation “connecting with horizon domains is only supported if client connects using TLS certificates, and these TLS certificates need to be generated manually”. Why is the usage limited to connections with manually generated TLS certificates?

================================

A) replicaset exposure using ClusterIP


replsets:

  • name: rs0
    expose:
    enabled: true
    exposeType: ClusterIP

B) Virtualservice Definition

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-cluster-name-virtual-service
namespace: percona
spec:
gateways:
- istio-system/cmp-szh-loc-capabilities-gateway
hosts:
- pg.eng.cmp.szh.loc
tcp:
- match:
- port: 5011
route:
- destination:
host: my-cluster-name-rs0-0
port:
number: 27017
- match:
- port: 5012
route:
- destination:
host: my-cluster-name-rs0-1
port:
number: 27017
- match:
- port: 5013
route:
- destination:
host: my-cluster-name-rs0-2
port:
number: 27017

C) endpoints, services and virtualservice

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get endpoints
NAME ENDPOINTS AGE
my-cluster-name-rs0 10.200.11.178:27017,10.200.12.224:27017,10.200.15.132:27017 28h
my-cluster-name-rs0-0 10.200.15.132:27017 25h
my-cluster-name-rs0-1 10.200.11.178:27017 25h
my-cluster-name-rs0-2 10.200.12.224:27017 25h

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-rs0 ClusterIP None 27017/TCP 28h
my-cluster-name-rs0-0 ClusterIP 10.201.24.174 27017/TCP 25h
my-cluster-name-rs0-1 ClusterIP 10.201.31.38 27017/TCP 25h
my-cluster-name-rs0-2 ClusterIP 10.201.59.152 27017/TCP 25h

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get virtualservice
NAME GATEWAYS HOSTS AGE
my-cluster-name-virtual-service [“istio-system/cmp-szh-loc-capabilities-gateway”] [“pg.eng.cmp.szh.loc”] 25h

Could you please shed some light on this? Especially how to get such an exposed replicaset connection working?

Hello @Markus , there are a lot of questions :slight_smile:

I have one too: could you please show the connection string that you use to connect to MongoDB (the one that fails)?

As for your question about TLS:

I also read following limitation “connecting with horizon domains is only supported if client connects using TLS certificates, and these TLS certificates need to be generated manually”. Why is the usage limited to connections with manually generated TLS certificates?

We implemented split horizon, so that now MongoDB replica set is configured in a way to accept connections for the domains listed there. But TLS certificates are not automatically generated for these domains, so you should generate them manually.

@Sergey_Pronin

D:\mongosh-2.0.2-win32-x64\bin>mongosh “mongodb://clusterAdmin:clusterAdmin123456@pg.eng.cmp.szh.loc:5011,pg.eng.cmp.szh.loc:5012,pg.eng.cmp.szh.loc:5013/admin?replicaSet=rs0&ssl=false”
Current Mongosh Log ID: 65cb42d3fcfb9f0cd6a12ff8
Connecting to: mongodb://@pg.eng.cmp.szh.loc:5011,pg.eng.cmp.szh.loc:5012,pg.eng.cmp.szh.loc:5013/admin?replicaSet=rs0&ssl=false&appName=mongosh+2.0.2
MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local

@Markus thanks for sharing. I will spend some time this week to reproduce this and get back to you.