Ssue Connecting to MongoDB via External Addresses (splitHorizons)

MongoDB Percona Operator for MongoDB
,
1

Description

We are trying to use Percona Server for MongoDB Operator in Kubernetes with configured splitHorizons to allow external clients to connect via OCI Load Balancer.

Our splitHorizons configuration:

    expose:
      enabled: true
      type: LoadBalancer
      serviceAnnotations:
        oci.oraclecloud.com/load-balancer-type: lb
        service.beta.kubernetes.io/oci-load-balancer-internal: 'true'
        service.beta.kubernetes.io/oci-load-balancer-shape: flexible
        service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: '1000'
        service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: '100'
        service.beta.kubernetes.io/oci-load-balancer-subnet1: ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaap6bdzaxslyz6m54ow2kq62mno3r44a3zvq6scxs2ozm3zbtafw6a
    splitHorizons:
      hybrid-cluster-rs0-0:
        external: dev-oci-lb09.pbu.icb
      hybrid-cluster-rs0-1:
        external: dev-oci-lb10.pbu.icb
      hybrid-cluster-rs0-2:
        external: dev-oci-lb11.pbu.icb

In rs.conf(), we can see that horizons are correctly set:

{
  "_id": "rs0",
  "members": [
    {
      "_id": 0,
      "host": "hybrid-cluster-rs0-0.hybrid-cluster-rs0.mongodb-percona.svc.cluster.local:27017",
      "horizons": { "external": "dev-oci-lb09.pbu.icb:27017" }
    },
    {
      "_id": 1,
      "host": "hybrid-cluster-rs0-1.hybrid-cluster-rs0.mongodb-percona.svc.cluster.local:27017",
      "horizons": { "external": "dev-oci-lb10.pbu.icb:27017" }
    },
    {
      "_id": 2,
      "host": "hybrid-cluster-rs0-2.hybrid-cluster-rs0.mongodb-percona.svc.cluster.local:27017",
      "horizons": { "external": "dev-oci-lb11.pbu.icb:27017" }
    }
  ]
}

ERROR

However, when connecting via mongosh using the connection string:

mongosh "mongodb://dev-oci-lb09.pbu.icb,dev-oci-lb10.pbu.icb,dev-oci-lb11.pbu.icb/?replicaSet=rs0"
Current Mongosh Log ID: 67ebe09d391aa73ff86b140a
Connecting to:          mongodb://dev-oci-lb09.pbu.icb,dev-oci-lb10.pbu.icb,dev-oci-lb11.pbu.icb/?replicaSet=rs0&appName=mongosh+2.4.2
MongoNetworkError: getaddrinfo ENOTFOUND hybrid-cluster-rs0-0.hybrid-cluster-rs0.mongodb-percona.svc.cluster.local

This indicates that mongosh ignores horizons and tries to connect using internal Kubernetes DNS names.

Thank you for your help!

2

Hi folks, I have a pretty similar issue to @yabilyk. I am using PSMDB operator version 1.19.1 and AWS cloud

My deploy.yaml

...
replsets:
  - name: rs0
    size: 3
    splitHorizons:
      replicaset-test-cluster-rs0-0:
        external: ac875915c907149fd85d898fedaaf592-369198c0c1b238f8.elb.us-east-1.amazonaws.com
      replicaset-test-cluster-rs0-1:
        external: ae17a37c956ca4067a793cdaf4f88f86-41858c98f2ba4d67.elb.us-east-1.amazonaws.com
      replicaset-test-cluster-rs0-2:
        external: ae17a37c956ca4067a793cdaf4f88f86-41858c98f2ba4d67.elb.us-east-1.amazonaws.com
    expose:
      enabled: true
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
...

On replicaSet side, it seems applied correctly based on the rs.conf and the Operator’s log:

rs0 [direct: secondary] admin> rs.conf().members.map(m => m.horizons)
[
  {
    external: 'ac875915c907149fd85d898fedaaf592-369198c0c1b238f8.elb.us-east-1.amazonaws.com:27017'
  },
  {
    external: 'ae17a37c956ca4067a793cdaf4f88f86-41858c98f2ba4d67.elb.us-east-1.amazonaws.com:27017'
  },
  {
    external: 'a6b06414eaed64350bbdd54d2a62838e-84cc47d94c1c66a4.elb.us-east-1.amazonaws.com:27017'
  }
]

Operator’s log:

2025-04-22T20:35:44.678Z	INFO	Horizons changed	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "aa0d02e5-b94a-4580-a076-33179f68f5f2", "host": "replicaset-test-cluster-rs0-0.replicaset-test-cluster-rs0.percona-mongodb.svc.cluster.local:27017", "old": null, "new": {"external":"ac875915c907149fd85d898fedaaf592-369198c0c1b238f8.elb.us-east-1.amazonaws.com:27017"}}
2025-04-22T20:35:44.678Z	INFO	Tags changed	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "aa0d02e5-b94a-4580-a076-33179f68f5f2", "host": "replicaset-test-cluster-rs0-1.replicaset-test-cluster-rs0.percona-mongodb.svc.cluster.local:27017", "old": {"nodeName":"ip-172-25-119-245.ec2.internal","podName":"replicaset-test-cluster-rs0-1","serviceName":"replicaset-test-cluster"}, "new": {"nodeName":"ip-172-25-116-102.ec2.internal","podName":"replicaset-test-cluster-rs0-1","serviceName":"replicaset-test-cluster"}}
2025-04-22T20:35:44.678Z	INFO	Horizons changed	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "aa0d02e5-b94a-4580-a076-33179f68f5f2", "host": "replicaset-test-cluster-rs0-1.replicaset-test-cluster-rs0.percona-mongodb.svc.cluster.local:27017", "old": null, "new": {"external":"ae17a37c956ca4067a793cdaf4f88f86-41858c98f2ba4d67.elb.us-east-1.amazonaws.com:27017"}}
2025-04-22T20:35:44.678Z	INFO	Tags changed	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "aa0d02e5-b94a-4580-a076-33179f68f5f2", "host": "replicaset-test-cluster-rs0-2.replicaset-test-cluster-rs0.percona-mongodb.svc.cluster.local:27017", "old": {"nodeName":"ip-172-25-121-167.ec2.internal","podName":"replicaset-test-cluster-rs0-2","serviceName":"replicaset-test-cluster"}, "new": {"nodeName":"ip-172-25-122-116.ec2.internal","podName":"replicaset-test-cluster-rs0-2","serviceName":"replicaset-test-cluster"}}
2025-04-22T20:35:44.678Z	INFO	Horizons changed	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "aa0d02e5-b94a-4580-a076-33179f68f5f2", "host": "replicaset-test-cluster-rs0-2.replicaset-test-cluster-rs0.percona-mongodb.svc.cluster.local:27017", "old": null, "new": {"external":"a6b06414eaed64350bbdd54d2a62838e-84cc47d94c1c66a4.elb.us-east-1.amazonaws.com:27017"}}
2025-04-22T20:35:44.678Z	INFO	Fixing member configurations	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "aa0d02e5-b94a-4580-a076-33179f68f5f2", "replset": "rs0"}
2025-04-22T20:35:48.018Z	INFO	Cluster state changed	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "aa0d02e5-b94a-4580-a076-33179f68f5f2", "previous": "error", "current": "ready"}
2025-04-22T20:36:02.689Z	INFO	StatefulSet is not up to date	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "sts": "replicaset-test-cluster-rs0"}
2025-04-22T20:36:02.898Z	INFO	SmartUpdate	StatefulSet is changed, starting smart update	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "statefulset": "replicaset-test-cluster-rs0", "replset": "rs0"}
2025-04-22T20:36:03.091Z	INFO	SmartUpdate	apply changes to secondary pod	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "statefulset": "replicaset-test-cluster-rs0", "replset": "rs0", "pod": "replicaset-test-cluster-rs0-2"}
2025-04-22T20:36:07.175Z	INFO	Cluster state changed	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "previous": "ready", "current": "initializing"}
2025-04-22T20:36:44.428Z	INFO	Pod started	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "pod": "replicaset-test-cluster-rs0-2"}
2025-04-22T20:36:44.469Z	INFO	SmartUpdate	primary pod detected	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "statefulset": "replicaset-test-cluster-rs0", "replset": "rs0", "pod": "replicaset-test-cluster-rs0-1"}
2025-04-22T20:36:44.510Z	INFO	SmartUpdate	apply changes to secondary pod	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "statefulset": "replicaset-test-cluster-rs0", "replset": "rs0", "pod": "replicaset-test-cluster-rs0-0"}
2025-04-22T20:37:22.376Z	INFO	Pod started	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "pod": "replicaset-test-cluster-rs0-0"}
2025-04-22T20:37:22.376Z	INFO	SmartUpdate	doing step down...	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "statefulset": "replicaset-test-cluster-rs0", "replset": "rs0", "force": false}
2025-04-22T20:37:22.444Z	INFO	SmartUpdate	apply changes to primary pod	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "statefulset": "replicaset-test-cluster-rs0", "replset": "rs0", "pod": "replicaset-test-cluster-rs0-1"}
2025-04-22T20:38:28.806Z	INFO	Pod started	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "pod": "replicaset-test-cluster-rs0-1"}
2025-04-22T20:38:28.806Z	INFO	SmartUpdate	smart update finished for statefulset	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "statefulset": "replicaset-test-cluster-rs0", "replset": "rs0"}
2025-04-22T20:38:32.209Z	INFO	Cluster state changed	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"replicaset-test-cluster","namespace":"percona-mongodb"}, "namespace": "percona-mongodb", "name": "replicaset-test-cluster", "reconcileID": "8b6e2ac0-c3de-4e62-af52-3d9903efe9f1", "previous": "initializing", "current": "ready"}

However, any attempts to connect to the cluster outside of k8s failed:

Connecting to:		mongodb://<credentials>@ac875915c907149fd85d898fedaaf592-369198c0c1b238f8.elb.us-east-1.amazonaws.com:27017,ae17a37c956ca4067a793cdaf4f88f86-41858c98f2ba4d67.elb.us-east-1.amazonaws.com:27017,a6b06414eaed64350bbdd54d2a62838e-84cc47d94c1c66a4.elb.us-east-1.amazonaws.com:27017/admin?authSource=admin&replicaSet=rs0&appName=mongosh+2.3.4
MongoNetworkError: getaddrinfo ENOTFOUND replicaset-test-cluster-rs0-1.replicaset-test-cluster-rs0.percona-mongodb.svc.cluster.local

The cluster is reachable inside k8s.