Exposing the cluster externally and integration issue with kafka

MongoDB Percona Operator for MongoDB
1

Hi friends,

I have an existing percona mongodb deployment. My question is around expsing the mongodb externally and integration issues with other services. Exposing the cluster - Percona Operator for MongoDB

We are having trouble with Kafka Debezium MongoDB connector making a connection with our mongodb cluster with our current setup. Our problem comes from the fact that our replicaset is using the host internal domain names. clusterServiceDNSMode: "Internal"

In the Kafka Debezium MongoDB Connector.

The client will use that just as a seed and retrieve the full list of hostnames from the MongoDb server itself.

Which results in Kafka Connect trying to connect to psmdb-db-rs0-0.psmdb-db-rs0.mongodb.svc.cluster.local:27017 which will obviously fail as that is not accessible outside of our Kubernetes cluster.

Current Setup (psmdb-values.yaml)

clusterServiceDNSMode: "Internal"
...
replsets:
  - name: rs0
    size: 1
    expose:
      enabled: true
      exposeType: LoadBalancer
      serviceAnnotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

rs.status()

  members: [
    {
      _id: 0,
      name: 'psmdb-db-rs0-0.psmdb-db-rs0.mongodb.svc.cluster.local:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 6214376,
      optime: { ts: Timestamp({ t: 1690971550, i: 3 }), t: Long("1") },
      optimeDate: ISODate("2023-08-02T10:19:10.000Z"),
      lastAppliedWallTime: ISODate("2023-08-02T10:19:10.760Z"),
      lastDurableWallTime: ISODate("2023-08-02T10:19:10.760Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1684757200, i: 2 }),
      electionDate: ISODate("2023-05-22T12:06:40.000Z"),
      configVersion: 3,
      configTerm: 1,
      self: true,
      lastHeartbeatMessage: ''
    }
  ],

This issue for us is resolved when we use clusterServiceDNSMode: "External". In this setup the member address is using the aws load balancer address which is exposd externally and Kafka whcih sits outside of our K8s instance can communicate with this address.

  members: [
    {
      _id: 0,
      name: 'k8s-mongodb-psmdbdbr-2ea9294026-<redacted>.elb.eu-west-2.amazonaws.com:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 89500,
      optime: { ts: Timestamp({ t: 1690970595, i: 3 }), t: Long("1") },
      optimeDate: ISODate("2023-08-02T10:03:15.000Z"),
      lastAppliedWallTime: ISODate("2023-08-02T10:03:15.143Z"),
      lastDurableWallTime: ISODate("2023-08-02T10:03:15.143Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1690881119, i: 2 }),
      electionDate: ISODate("2023-08-01T09:11:59.000Z"),
      configVersion: 6,
      configTerm: 1,
      self: true,
      lastHeartbeatMessage: ''
    }
  ],

But i have a problem and it is the disclaimer written on the documentation linked above.

You should be careful with the clusterServiceDNSMode=External variant. Using IP addresses instead of DNS hostnames is discouraged in MongoDB. IP addresses make configuration changes and recovery more complicated. Also, they are particularly problematic in scenarios where IP addresses change (i.e., deleting and recreating the cluster).

Is this disclaimer still relevant, given that when using custerServiceDNSMode=External it is actually using the dns address of the aws load balancer and not ip address directly?

Any other thoughts around how this should be dealt with are welcome.