Connection to an exposed replicaset is not working

Markus · December 19, 2023, 4:39pm

Description:

I exposed my mongodb cluster using istio service mesh and a virtualservice (see below). I can open a connection to every single node. When I try to connect to the replicaset, I get an error message ==>>
MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-1.my-cluster-name-rs0.percona.svc.cluster.local

Seems this is the same problem anounced here: Connection Issues When Accessing MongoDB Replica Set from Outside the Kubernetes Cluster

Please note, that we are not able to use loadbalancer ip’s as we do not have a lot of free and useable ip’s. That’s the reason for using a virtualservice and mapping different ports to different services. We cannot use the “clusterServiceDNSMode: External” workaround therefore.

The Blog also mentioned split DNS. The documentation is very short and unclear about this. Could you explain in detail what is going on behind the scences when I configure horizons like this?

replsets:
  - name: rs0
    expose:
      enabled: true
      exposeType: ClusterIP
    horizons:
      my-cluster-name-rs0-0:
        external: my-cluster-name-rs0-0.eng.cmp.szh.loc
      my-cluster-name-rs0-1:
        external: my-cluster-name-rs0-1.eng.cmp.szh.loc
      my-cluster-name-rs0-2:
        external: my-cluster-name-rs0-2.eng.cmp.szh.loc

I also read following limitation “connecting with horizon domains is only supported if client connects using TLS certificates, and these TLS certificates need to be generated manually”. Why is the usage limited to connections with manually generated TLS certificates?

================================

A) replicaset exposure using ClusterIP

…
replsets:

name: rs0
expose:
enabled: true
exposeType: ClusterIP
…

B) Virtualservice Definition

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-cluster-name-virtual-service
namespace: percona
spec:
gateways:
- istio-system/cmp-szh-loc-capabilities-gateway
hosts:
- pg.eng.cmp.szh.loc
tcp:
- match:
- port: 5011
route:
- destination:
host: my-cluster-name-rs0-0
port:
number: 27017
- match:
- port: 5012
route:
- destination:
host: my-cluster-name-rs0-1
port:
number: 27017
- match:
- port: 5013
route:
- destination:
host: my-cluster-name-rs0-2
port:
number: 27017

C) endpoints, services and virtualservice

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get endpoints
NAME ENDPOINTS AGE
my-cluster-name-rs0 10.200.11.178:27017,10.200.12.224:27017,10.200.15.132:27017 28h
my-cluster-name-rs0-0 10.200.15.132:27017 25h
my-cluster-name-rs0-1 10.200.11.178:27017 25h
my-cluster-name-rs0-2 10.200.12.224:27017 25h

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-rs0 ClusterIP None 27017/TCP 28h
my-cluster-name-rs0-0 ClusterIP 10.201.24.174 27017/TCP 25h
my-cluster-name-rs0-1 ClusterIP 10.201.31.38 27017/TCP 25h
my-cluster-name-rs0-2 ClusterIP 10.201.59.152 27017/TCP 25h

oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get virtualservice
NAME GATEWAYS HOSTS AGE
my-cluster-name-virtual-service [“istio-system/cmp-szh-loc-capabilities-gateway”] [“pg.eng.cmp.szh.loc”] 25h

Markus · February 7, 2024, 12:57pm

Could you please shed some light on this? Especially how to get such an exposed replicaset connection working?

Sergey_Pronin · February 12, 2024, 9:08am

Hello @Markus , there are a lot of questions

I have one too: could you please show the connection string that you use to connect to MongoDB (the one that fails)?

As for your question about TLS:

I also read following limitation “connecting with horizon domains is only supported if client connects using TLS certificates, and these TLS certificates need to be generated manually”. Why is the usage limited to connections with manually generated TLS certificates?

We implemented split horizon, so that now MongoDB replica set is configured in a way to accept connections for the domains listed there. But TLS certificates are not automatically generated for these domains, so you should generate them manually.

Markus · February 13, 2024, 10:29am

@Sergey_Pronin

D:\mongosh-2.0.2-win32-x64\bin>mongosh “mongodb://clusterAdmin:clusterAdmin123456@pg.eng.cmp.szh.loc:5011,pg.eng.cmp.szh.loc:5012,pg.eng.cmp.szh.loc:5013/admin?replicaSet=rs0&ssl=false”
Current Mongosh Log ID: 65cb42d3fcfb9f0cd6a12ff8
Connecting to: mongodb://@pg.eng.cmp.szh.loc:5011,pg.eng.cmp.szh.loc:5012,pg.eng.cmp.szh.loc:5013/admin?replicaSet=rs0&ssl=false&appName=mongosh+2.0.2
MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local

Sergey_Pronin · February 19, 2024, 10:01am

@Markus thanks for sharing. I will spend some time this week to reproduce this and get back to you.

Markus · April 10, 2024, 11:33am

@Sergey_Pronin Any news on this one?

Sergey_Pronin · April 12, 2024, 1:52pm

Hello @Markus ,

when you configure splitHorizons in the custom resource, you get the horizons in your replica set. You will see it in your rs.conf():

rs0 [primary] admin> rs.conf()
{
  _id: 'rs0',
  version: 9,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'rs-psmdb-db-rs0-0.rs-psmdb-db-rs0.default.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 2,
      tags: { podName: 'rs-psmdb-db-rs0-0', serviceName: 'rs-psmdb-db' },
      horizons: { external: 'somdomain.com:27017' },
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
...

I tried exposing mongo with istio, it worked for me with a sharded cluster, where you have mongos.
I falied to expose a single replica set for now, still investigating. What do you think about converting your cluster to sharded one, and expose it through mongos (as a workaround)?

Markus · April 16, 2024, 7:49am

Hi,

Does this mean, an external mongodb client would read now “rs-psmdb-db-rs0-0.rs-psmdb-db-rs0.somdomain.com:27017,rs-psmdb-db-rs0-1.rs-psmdb-db-rs0.somdomain.com:27017,rs-psmdb-db-rs0-3.rs-psmdb-db-rs0.somdomain.com:27017” as replicaset config and use that for reconnecting?

Using sharding as a workaround is not an option for us.

Thanks, Markus

Sergey_Pronin · April 17, 2024, 7:27am

@Markus that is correct.

Markus · April 26, 2024, 12:57pm

@Sergey_Pronin

any idea how to get a replicaset working with istio?

I tried to to define split horizons and virtualservices as show below. Still not working.

…
splitHorizons:
my-cluster-name-rs0-0:
external: psmdb1.eng.cmp.szh.loc
my-cluster-name-rs0-1:
external: psmdb2.eng.cmp.szh.loc
my-cluster-name-rs0-2:
external: psmdb3.eng.cmp.szh.loc
…

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: psmdb1-vs
namespace: percona
spec:
gateways:
- istio-system/cmp-szh-loc-capabilities-gateway
hosts:
- my-cluster-name-rs0-0.psmdb1.eng.cmp.szh.loc
tcp:
- match:
- port: 5011
route:
- destination:
host: my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local
port:
number: 27017

Topic		Replies	Views
Unable to expose mongodb replicaset to nodeport Kubernetes Percona Operator for MongoDB	1	54	April 15, 2024
Connection Issues When Accessing MongoDB Replica Set from Outside the Kubernetes Cluster Percona Operator for MongoDB mongodb , psmdb-operator	2	972	August 3, 2023
Kubernetes, replicaset, no sharding - where is the service to access the replicaset Percona Operator for MongoDB	13	2315	September 7, 2023
Is exposing mongo replicaset with NodePort working? Percona Operator for MongoDB percona , mongodb	28	2331	September 27, 2021
Exposed Replicaset doesn't accept TLS cert's generated by Cert Manager (MongoServerSelectionError: self-signed certificate) Percona Operator for MongoDB closed-no-reply	0	828	July 25, 2023