# Connection to an exposed replicaset is not working **URL:** \ **Category:** Percona Operator for MongoDB\ **Created:** [December 19, 2023, 4:39pm UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250 "2023-12-19T16:39:36Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Markus](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/markus/32/13571_2.png) [@Markus](https://forums.percona.com/u/Markus)\ **Post date:** [December 19, 2023, 4:39pm UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/1 "2023-12-19T16:39:36Z")
## Description: I exposed my mongodb cluster using istio service mesh and a virtualservice (see below). I can open a connection to every single node. When I try to connect to the replicaset, I get an error message ==\>\> **MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-1.my-cluster-name-rs0.percona.svc.cluster.local** Seems this is the same problem anounced here: [Connection Issues When Accessing MongoDB Replica Set from Outside the Kubernetes Cluster](https://forums.percona.com/t/connection-issues-when-accessing-mongodb-replica-set-from-outside-the-kubernetes-cluster/24164) Please note, that we are not able to use loadbalancer ip’s as we do not have a lot of free and useable ip’s. That’s the reason for using a virtualservice and mapping different ports to different services. We cannot use the “clusterServiceDNSMode: External” workaround therefore. The Blog also mentioned split DNS. The documentation is very short and unclear about this. **Could you explain in detail what is going on behind the scences when I configure horizons like this?** ``` replsets: - name: rs0 expose: enabled: true exposeType: ClusterIP horizons: my-cluster-name-rs0-0: external: my-cluster-name-rs0-0.eng.cmp.szh.loc my-cluster-name-rs0-1: external: my-cluster-name-rs0-1.eng.cmp.szh.loc my-cluster-name-rs0-2: external: my-cluster-name-rs0-2.eng.cmp.szh.loc ``` I also read following limitation “connecting with horizon domains is only supported if client connects using TLS certificates, and these TLS certificates need to be generated manually”. **Why is the usage limited to connections with manually generated TLS certificates?** ================================ ### A) replicaset exposure using ClusterIP … replsets: - name: rs0 expose: enabled: true exposeType: ClusterIP … ### B) Virtualservice Definition apiVersion: [networking.istio.io/v1alpha3](http://networking.istio.io/v1alpha3) kind: VirtualService metadata: name: my-cluster-name-virtual-service namespace: percona spec: gateways: - istio-system/cmp-szh-loc-capabilities-gateway hosts: - pg.eng.cmp.szh.loc tcp: - match: - port: 5011 route: - destination: host: my-cluster-name-rs0-0 port: number: 27017 - match: - port: 5012 route: - destination: host: my-cluster-name-rs0-1 port: number: 27017 - match: - port: 5013 route: - destination: host: my-cluster-name-rs0-2 port: number: 27017 ### C) endpoints, services and virtualservice oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get endpoints NAME ENDPOINTS AGE my-cluster-name-rs0 10.200.11.178:27017,10.200.12.224:27017,10.200.15.132:27017 28h my-cluster-name-rs0-0 10.200.15.132:27017 25h my-cluster-name-rs0-1 10.200.11.178:27017 25h my-cluster-name-rs0-2 10.200.12.224:27017 25h oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-cluster-name-rs0 ClusterIP None 27017/TCP 28h my-cluster-name-rs0-0 ClusterIP 10.201.24.174 27017/TCP 25h my-cluster-name-rs0-1 ClusterIP 10.201.31.38 27017/TCP 25h my-cluster-name-rs0-2 ClusterIP 10.201.59.152 27017/TCP 25h oizzwma@szhm90313:~/git/percona/percona-server-mongodb-operator/deploy$ k8s-szh-engineering -n percona get virtualservice NAME GATEWAYS HOSTS AGE my-cluster-name-virtual-service [“istio-system/cmp-szh-loc-capabilities-gateway”] [“pg.eng.cmp.szh.loc”] 25h ---
**Author:** ![Markus](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/markus/32/13571_2.png) [@Markus](https://forums.percona.com/u/Markus)\ **Post date:** [February 7, 2024, 12:57pm UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/2 "2024-02-07T12:57:22Z")
> [@Markus](#): > > Could you explain in detail what is going on behind the scences when I configure horizons like this? Could you please shed some light on this? Especially how to get such an exposed replicaset connection working? ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [February 12, 2024, 9:08am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/3 "2024-02-12T09:08:00Z")
Hello @Markus , there are a lot of questions 🙂 I have one too: could you please show the connection string that you use to connect to MongoDB (the one that fails)? As for your question about TLS: > I also read following limitation “connecting with horizon domains is only supported if client connects using TLS certificates, and these TLS certificates need to be generated manually”. **Why is the usage limited to connections with manually generated TLS certificates?** We implemented split horizon, so that now MongoDB replica set is configured in a way to accept connections for the domains listed there. But TLS certificates are not automatically generated for these domains, so you should generate them manually. ---
**Author:** ![Markus](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/markus/32/13571_2.png) [@Markus](https://forums.percona.com/u/Markus)\ **Post date:** [February 13, 2024, 10:29am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/4 "2024-02-13T10:29:16Z")
@Sergey_Pronin D:\mongosh-2.0.2-win32-x64\bin\> **mongosh “mongodb://clusterAdmin:clusterAdmin123456@pg.eng.cmp.szh.loc:5011,pg.eng.cmp.szh.loc:5012,pg.eng.cmp.szh.loc:5013/admin?replicaSet=rs0&ssl=false”** Current Mongosh Log ID: 65cb42d3fcfb9f0cd6a12ff8 Connecting to: mongodb://@pg.eng.cmp.szh.loc:5011,pg.eng.cmp.szh.loc:5012,pg.eng.cmp.szh.loc:5013/admin?replicaSet=rs0&ssl=false&appName=mongosh+2.0.2 **MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local** ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [February 19, 2024, 10:01am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/5 "2024-02-19T10:01:53Z")
@Markus thanks for sharing. I will spend some time this week to reproduce this and get back to you. ---
**Author:** ![Markus](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/markus/32/13571_2.png) [@Markus](https://forums.percona.com/u/Markus)\ **Post date:** [April 10, 2024, 11:33am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/6 "2024-04-10T11:33:59Z")
> [@Markus](#): > > Could you explain in detail what is going on behind the scences when I configure horizons like this? @Sergey_Pronin Any news on this one? ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [April 12, 2024, 1:52pm UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/7 "2024-04-12T13:52:14Z")
Hello @Markus , when you configure `splitHorizons` in the custom resource, you get the `horizons` in your replica set. You will see it in your `rs.conf()`: ```auto rs0 [primary] admin> rs.conf() { _id: 'rs0', version: 9, term: 1, members: [ { _id: 0, host: 'rs-psmdb-db-rs0-0.rs-psmdb-db-rs0.default.svc.cluster.local:27017', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2, tags: { podName: 'rs-psmdb-db-rs0-0', serviceName: 'rs-psmdb-db' }, horizons: { external: 'somdomain.com:27017' }, secondaryDelaySecs: Long("0"), votes: 1 }, ... ``` I tried exposing mongo with istio, it worked for me with a sharded cluster, where you have mongos. I falied to expose a single replica set for now, still investigating. What do you think about converting your cluster to sharded one, and expose it through mongos (as a workaround)? ---
**Author:** ![Markus](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/markus/32/13571_2.png) [@Markus](https://forums.percona.com/u/Markus)\ **Post date:** [April 16, 2024, 7:49am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/8 "2024-04-16T07:49:14Z")
Hi, Does this mean, an external mongodb client would read now “[rs-psmdb-db-rs0-0.rs-psmdb-db-rs0.somdomain.com:27017](http://rs-psmdb-db-rs0-0.rs-psmdb-db-rs0.somdomain.com:27017),[rs-psmdb-db-rs0-1.rs-psmdb-db-rs0.somdomain.com:27017](http://rs-psmdb-db-rs0-1.rs-psmdb-db-rs0.somdomain.com:27017),[rs-psmdb-db-rs0-3.rs-psmdb-db-rs0.somdomain.com:27017](http://rs-psmdb-db-rs0-3.rs-psmdb-db-rs0.somdomain.com:27017)” as replicaset config and use that for reconnecting? Using sharding as a workaround is not an option for us. Thanks, Markus ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [April 17, 2024, 7:27am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/9 "2024-04-17T07:27:04Z")
@Markus that is correct. ---
**Author:** ![Markus](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/markus/32/13571_2.png) [@Markus](https://forums.percona.com/u/Markus)\ **Post date:** [April 26, 2024, 12:57pm UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/10 "2024-04-26T12:57:52Z")
@Sergey_Pronin any idea how to get a replicaset working with istio? I tried to to define split horizons and virtualservices as show below. Still not working. * * * … splitHorizons: my-cluster-name-rs0-0: external: psmdb1.eng.cmp.szh.loc my-cluster-name-rs0-1: external: psmdb2.eng.cmp.szh.loc my-cluster-name-rs0-2: external: psmdb3.eng.cmp.szh.loc … * * * apiVersion: [networking.istio.io/v1alpha3](http://networking.istio.io/v1alpha3) kind: VirtualService metadata: name: psmdb1-vs namespace: percona spec: gateways: - istio-system/cmp-szh-loc-capabilities-gateway hosts: - my-cluster-name-rs0-0.psmdb1.eng.cmp.szh.loc tcp: - match: - port: 5011 route: - destination: host: my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local port: number: 27017 ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [May 31, 2024, 10:14am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/11 "2024-05-31T10:14:56Z")
There were two blog posts published recently: > **[Beyond the Horizon: Mastering Percona Server for MongoDB Exposure in...](https://www.percona.com/blog/beyond-the-horizon-mastering-percona-server-for-mongodb-exposure-in-kubernetes-part-one/)** > > In this series of blog posts, we will explore it in depth and review various strategies for connecting to your MongoDB cluster from anywhere. In part 1, we cover single replica set exposure and explain split horizons concept. > **[Beyond The Horizon: Mastering Percona Server for MongoDB Exposure in...](https://www.percona.com/blog/beyond-the-horizon-mastering-percona-server-for-mongodb-exposure-in-kubernetes-part-two-istio/)** > > Part two of a series unmasking the complexity of MongoDB cluster exposure in Kubernetes with Percona Operator for MongoDB on exposing a sharded cluster and a single replica set with Istio. The first one goes into depth of split horizons and secone one provides an example for Istio. I suggest to read the first one too. ---
**Author:** ![Akshay\_Viswanathan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/akshay_viswanathan/32/17857_2.png) [@Akshay\_Viswanathan](https://forums.percona.com/u/Akshay_Viswanathan)\ **Post date:** [August 30, 2024, 9:00am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/12 "2024-08-30T09:00:39Z")
Hey Sergey, I went through the documents you listed but I still encounter the same issue: rs.conf() does list the correctly configured horizons, but I still see the same error. Perhaps interesting to note that we use tailscale as a load balancer, added on seprately (not via the helm chart). Has there been some confirmation that the replicaset + split horizons works for load balancers configured separately? MongoNetworkError: getaddrinfo ENOTFOUND my-cluster-name-rs0-0.my-cluster-name-rs0.percona.svc.cluster.local ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [August 31, 2024, 5:37am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/13 "2024-08-31T05:37:55Z")
> Has there been some confirmation that the replicaset + split horizons works for load balancers configured separately? It works for istio and it works for LBs that k8s provisions, which in a nutshell are the same as “external” LBs. 1. have you generated the certificates? 2. please show rs.conf here 3. how does your connection string look like? ---
**Author:** ![Akshay\_Viswanathan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/akshay_viswanathan/32/17857_2.png) [@Akshay\_Viswanathan](https://forums.percona.com/u/Akshay_Viswanathan)\ **Post date:** [September 1, 2024, 5:57pm UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/14 "2024-09-01T17:57:16Z")
Bullet 1. have you generated the certificates? This I haven’t done. My mistake, that is probably the cause in that case. Certificate management is not something I want to get into at this point, so I think I’ll go down the route of having a sharded cluster with shard size 1 as recommended above ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [September 2, 2024, 11:55am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/15 "2024-09-02T11:55:18Z")
@Akshay_Viswanathan makes sense to go with sharding. Split horizons will not work without proper TLS. ---
**Author:** ![surajm](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@surajm](https://forums.percona.com/u/surajm)\ **Post date:** [March 25, 2025, 10:52am UTC](https://forums.percona.com/t/connection-to-an-exposed-replicaset-is-not-working/27250/16 "2025-03-25T10:52:25Z")
Hi @Sergey_Pronin , I read the articles mentioned in this thread and trying the use case of multi cluster deployment with sharding enabled on both clusters deployments. Also, having TLS certs. Same cert is used on both clusters. When connected from Cluter-1, it was showing the rs state as unhealthy for cluster-2 replica sets. Even though I can reach the cluster-2 replicaset from cluster-1 replicaset when checked from Pod. I see below cert error in replicaset cluster-2 pods: ```auto {"t":{"$date":"2025-03-25T09:58:26.471+00:00"},"s":"I", "c":"ACCESS", "id":5286307, "ctx":"conn59825","msg":"Failed to authenticate","attr":{"client":"127.0.0.6:50501","isSpeculative":false,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cluster-1,O=PSMDB","db":"$external","error":"AuthenticationFailed: The provided certificate can only be used for cluster authentication, not client authentication. The current configuration does not allow x.509 cluster authentication, check the --clusterAuthMode flag","result":18,"metrics":{"conversation_duration":{"micros":85,"summary":{}}},"extraInfo":{}}} {"t":{"$date":"2025-03-25T09:58:26.555+00:00"},"s":"I", "c":"ACCESS", "id":5286307, "ctx":"conn59826","msg":"Failed to authenticate","attr":{"client":"127.0.0.6:35627","isSpeculative":false,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cluster-1,O=PSMDB","db":"$external","error":"AuthenticationFailed: The provided certificate can only be used for cluster authentication, not client authentication. The current configuration does not allow x.509 cluster authentication, check the --clusterAuthMode flag","result":18,"metrics":{"conversation_duration":{"micros":74,"summary":{}}},"extraInfo":{}}} {"t":{"$date":"2025-03-25T09:58:26.471+00:00"},"s":"I", "c":"ACCESS", "id":5286307, "ctx":"conn59825","msg":"Failed to authenticate","attr":{"client":"127.0.0.6:50501","isSpeculative":false,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cluster-1,O=PSMDB","db":"$external","error":"AuthenticationFailed: The provided certificate can only be used for cluster authentication, not client authentication. The current configuration does not allow x.509 cluster authentication, check the --clusterAuthMode flag","result":18,"metrics":{"conversation_duration":{"micros":85,"summary":{}}},"extraInfo":{}}} {"t":{"$date":"2025-03-25T09:58:26.555+00:00"},"s":"I", "c":"ACCESS", "id":5286307, "ctx":"conn59826","msg":"Failed to authenticate","attr":{"client":"127.0.0.6:35627","isSpeculative":false,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cluster-1,O=PSMDB","db":"$external","error":"AuthenticationFailed: The provided certificate can only be used for cluster authentication, not client authentication. The current configuration does not allow x.509 cluster authentication, check the --clusterAuthMode flag","result":18,"metrics":{"conversation_duration":{"micros":74,"summary":{}}},"extraInfo":{}}} ``` Also, MongoS on Cluster-2 was not coming up. Seeing below error: ```auto {"t":{"$date":"2025-03-25T09:53:11.693+00:00"},"s":"I", "c":"NETWORK", "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}} {"t":{"$date":"2025-03-25T09:53:11.693+00:00"},"s":"I", "c":"SHARDING", "id":22727, "ctx":"ShardRegistryUpdater","msg":"Error running periodic reload of shard registry","attr":{"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg","shardRegistryReloadIntervalSeconds":30}} {"t":{"$date":"2025-03-25T09:53:11.791+00:00"},"s":"I", "c":"NETWORK", "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}} {"t":{"$date":"2025-03-25T09:53:11.791+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error loading global settings from config server. Sleeping for 2 seconds and retrying","attr":{"error":{"code":133,"codeName":"FailedToSatisfyReadPreference","errmsg":"Error loading clusterID :: caused by :: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}}} {"t":{"$date":"2025-03-25T09:53:11.792+00:00"},"s":"I", "c":"NETWORK", "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set cfg"}} {"t":{"$date":"2025-03-25T09:53:11.792+00:00"},"s":"I", "c":"SHARDING", "id":6973904, "ctx":"QueryAnalysisConfigurationsRefresher","msg":"Failed to refresh query analysis configurations, will try again at the next refresh interval","attr":{"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set cfg"}} ``` any help or pointer here would be appreciated.