Connection to an exposed replicaset is not working

MongoDB Percona Operator for MongoDB
1

Description:

I exposed my mongodb cluster using istio service mesh and a virtualservice (see below). I can open a connection to every single node. When I try to connect to the replicaset, I get an error message ==>>
MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-1.my-cluster-name-rs0.percona.svc.cluster.local

Seems this is the same problem anounced here: Connection Issues When Accessing MongoDB Replica Set from Outside the Kubernetes Cluster

Please note, that we are not able to use loadbalancer ip’s as we do not have a lot of free and useable ip’s. That’s the reason for using a virtualservice and mapping different ports to different services. We cannot use the “clusterServiceDNSMode: External” workaround therefore.

The Blog also mentioned split DNS. The documentation is very short and unclear about this. Could you explain in detail what is going on behind the scences when I configure horizons like this?

replsets:
  - name: rs0
    expose:
      enabled: true
      exposeType: ClusterIP
    horizons:
      my-cluster-name-rs0-0:
        external: my-cluster-name-rs0-0.eng.cmp.szh.loc
      my-cluster-name-rs0-1:
        external: my-cluster-name-rs0-1.eng.cmp.szh.loc
      my-cluster-name-rs0-2:
        external: my-cluster-name-rs0-2.eng.cmp.szh.loc

I also read following limitation “connecting with horizon domains is only supported if client connects using TLS certificates, and these TLS certificates need to be generated manually”. Why is the usage limited to connections with manually generated TLS certificates?

================================

A) replicaset exposure using ClusterIP

…
replsets:

  • name: rs0
    expose:
    enabled: true
    exposeType: ClusterIP
    …

B) Virtualservice Definition

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-cluster-name-virtual-service
namespace: percona
spec:
gateways:
- istio-system/cmp-szh-loc-capabilities-gateway
hosts:
- pg.eng.cmp.szh.loc
tcp:
- match:
- port: 5011
route:
- destination:
host: my-cluster-name-rs0-0
port:
number: 27017
- match:
- port: 5012
route:
- destination:
host: my-cluster-name-rs0-1
port:
number: 27017
- match:
- port: 5013
route:
- destination:
host: my-cluster-name-rs0-2
port:
number: 27017

C) endpoints, services and virtualservice

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get endpoints
NAME ENDPOINTS AGE
my-cluster-name-rs0 10.200.11.178:27017,10.200.12.224:27017,10.200.15.132:27017 28h
my-cluster-name-rs0-0 10.200.15.132:27017 25h
my-cluster-name-rs0-1 10.200.11.178:27017 25h
my-cluster-name-rs0-2 10.200.12.224:27017 25h

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-rs0 ClusterIP None 27017/TCP 28h
my-cluster-name-rs0-0 ClusterIP 10.201.24.174 27017/TCP 25h
my-cluster-name-rs0-1 ClusterIP 10.201.31.38 27017/TCP 25h
my-cluster-name-rs0-2 ClusterIP 10.201.59.152 27017/TCP 25h

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get virtualservice
NAME GATEWAYS HOSTS AGE
my-cluster-name-virtual-service [“istio-system/cmp-szh-loc-capabilities-gateway”] [“pg.eng.cmp.szh.loc”] 25h

2

Could you please shed some light on this? Especially how to get such an exposed replicaset connection working?

3

Hello @Markus , there are a lot of questions :slight_smile:

I have one too: could you please show the connection string that you use to connect to MongoDB (the one that fails)?

As for your question about TLS:

I also read following limitation “connecting with horizon domains is only supported if client connects using TLS certificates, and these TLS certificates need to be generated manually”. Why is the usage limited to connections with manually generated TLS certificates?

We implemented split horizon, so that now MongoDB replica set is configured in a way to accept connections for the domains listed there. But TLS certificates are not automatically generated for these domains, so you should generate them manually.

4

@Sergey_Pronin

D:\mongosh-2.0.2-win32-x64\bin>mongosh “mongodb://clusterAdmin:clusterAdmin123456@pg.eng.cmp.szh.loc:5011,pg.eng.cmp.szh.loc:5012,pg.eng.cmp.szh.loc:5013/admin?replicaSet=rs0&ssl=false”
Current Mongosh Log ID: 65cb42d3fcfb9f0cd6a12ff8
Connecting to: mongodb://@pg.eng.cmp.szh.loc:5011,pg.eng.cmp.szh.loc:5012,pg.eng.cmp.szh.loc:5013/admin?replicaSet=rs0&ssl=false&appName=mongosh+2.0.2
MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local

5

@Markus thanks for sharing. I will spend some time this week to reproduce this and get back to you.

6

@Sergey_Pronin Any news on this one?

7

Hello @Markus ,

when you configure splitHorizons in the custom resource, you get the horizons in your replica set. You will see it in your rs.conf():

rs0 [primary] admin> rs.conf()
{
  _id: 'rs0',
  version: 9,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'rs-psmdb-db-rs0-0.rs-psmdb-db-rs0.default.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 2,
      tags: { podName: 'rs-psmdb-db-rs0-0', serviceName: 'rs-psmdb-db' },
      horizons: { external: 'somdomain.com:27017' },
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
...

I tried exposing mongo with istio, it worked for me with a sharded cluster, where you have mongos.
I falied to expose a single replica set for now, still investigating. What do you think about converting your cluster to sharded one, and expose it through mongos (as a workaround)?

8

Hi,

Does this mean, an external mongodb client would read now “rs-psmdb-db-rs0-0.rs-psmdb-db-rs0.somdomain.com:27017,rs-psmdb-db-rs0-1.rs-psmdb-db-rs0.somdomain.com:27017,rs-psmdb-db-rs0-3.rs-psmdb-db-rs0.somdomain.com:27017” as replicaset config and use that for reconnecting?

Using sharding as a workaround is not an option for us.

Thanks, Markus

9

@Markus that is correct.

10

@Sergey_Pronin

any idea how to get a replicaset working with istio?

I tried to to define split horizons and virtualservices as show below. Still not working.

…
splitHorizons:
my-cluster-name-rs0-0:
external: psmdb1.eng.cmp.szh.loc
my-cluster-name-rs0-1:
external: psmdb2.eng.cmp.szh.loc
my-cluster-name-rs0-2:
external: psmdb3.eng.cmp.szh.loc
…

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: psmdb1-vs
namespace: percona
spec:
gateways:
- istio-system/cmp-szh-loc-capabilities-gateway
hosts:
- my-cluster-name-rs0-0.psmdb1.eng.cmp.szh.loc
tcp:
- match:
- port: 5011
route:
- destination:
host: my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local
port:
number: 27017

11

There were two blog posts published recently:

The first one goes into depth of split horizons and secone one provides an example for Istio.
I suggest to read the first one too.

12

Hey Sergey, I went through the documents you listed but I still encounter the same issue:

rs.conf() does list the correctly configured horizons, but I still see the same error. Perhaps interesting to note that we use tailscale as a load balancer, added on seprately (not via the helm chart). Has there been some confirmation that the replicaset + split horizons works for load balancers configured separately?

MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local

13

Has there been some confirmation that the replicaset + split horizons works for load balancers configured separately?

It works for istio and it works for LBs that k8s provisions, which in a nutshell are the same as “external” LBs.

  1. have you generated the certificates?
  2. please show rs.conf here
  3. how does your connection string look like?
14

Bullet 1. have you generated the certificates?

This I haven’t done. My mistake, that is probably the cause in that case.
Certificate management is not something I want to get into at this point, so I think I’ll go down the route of having a sharded cluster with shard size 1 as recommended above

15

@Akshay_Viswanathan makes sense to go with sharding. Split horizons will not work without proper TLS.