I am trying to get access to the mongo cluster outside of kubernetes.
I have tried different variations of setting spec.replsets.expose to enabled: true and exposeType: NodePort as well as spec.sharding.mongos.expose.exposeType: NodePort
If I set both replsets and mongos to NodePort, the NodePort service is created for the rs0 nodes but the and for mongos creates and then is destroyed within about 25 seconds, and just repeats, each time with a new port. If I do this I am able to connect to each replica, but I was under the impression i should be connecting to the mongos service.
If I set only mongos to NodePort (and keep replsets to false/LoadBalancer) the rs0 cluster is Headless and the same behavior of mongos service creating with a port then destroying in 20ish seconds.
What is the correct way to expose the cluster via NodePort?
As you see it keeps rs0 service as ClusterIP and creates 3 new services (one per node) for nodeport.
Ports are not changed, services/pods not killed every 20 seconds.
BTW, as a best practice, I would not recommend NodePort and would use LoadBalancer.
If you still have problems - please share the cr.yaml and other important details about your cluster.
So I took a bit more time today to see if I could get the mongo cluster to get exposed outside the k8s cluster. Thanks for confirming that I should be only exposing mongos, that at least points me in a correct direction.
I am still having the same issue, with NodePort and setup MetalLB so I could use LoadBalancer as the service type, however the same behavior happens, the service is created (including pulling a public IP address) and then a few seconds later it is torn down and replaced.
my cr.yaml is exaclty the same as default (as is all teh other yaml) except I set line 253 from exposeType: ClusterIP to exposeType: LoadBalancer
my cluster is a “bare metal” kuberntes cluster provisioned/managed by Rancher 2.5.8, running kubernetes 1.20.6
I would be happy to share any logs, I tried to add the operator log, but it wouldnt allow a post that long.
output from kubectl for the less than 2 minutes. it took to spin up the cluster and for the service to be destroyed and recreated. If I left it running the service creation and destruction would happen indefinitely.
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl apply -f deploy/cr.yaml
perconaservermongodb.psmdb.percona.com/my-cluster-name created
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 3s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 1s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 6s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 2s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 4s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 9s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 5s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 7s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 13s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 9s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 11s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 16s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 12s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 14s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 20s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 16s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 18s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 23s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 19s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 21s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 27s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 23s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 25s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 30s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 26s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 28s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 35s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 31s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 33s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 38s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 34s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 36s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 42s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 38s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 40s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 45s
my-cluster-name-mongos LoadBalancer 10.43.222.87 192.168.1.170 27017:31718/TCP 41s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 43s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 49s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 47s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 56s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 54s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 59s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 57s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 62s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 60s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 64s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 62s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 68s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 66s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 72s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 70s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 76s
my-cluster-name-mongos LoadBalancer 10.43.126.235 192.168.1.170 27017:30708/TCP 0s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 74s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-name-cfg ClusterIP None <none> 27017/TCP 79s
my-cluster-name-mongos LoadBalancer 10.43.126.235 192.168.1.170 27017:30708/TCP 3s
my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 77s
xxx@xxxxxxx:~/percona-server-mongodb-operator$ kubectl delete -f deploy/cr.yaml
perconaservermongodb.psmdb.percona.com "my-cluster-name" deleted
The messages in the log you provided are only indicating that LB was gone.
Operator detects that LB is not there and recreates it according to Custom Resource.
The reason of why LB is deleted is still not clear. Do you have anything in MetalLB logs?
I don’t have a lot of experience with it either, I only added it as a test to see if it would solve this issue. As noted originally I was trying it with NodePort, which I have used before for some testing of TCP services and it worked fine.
I just added the kube-proxy log from one of the hosts, in it it shows the port going from 1 (27017) to 0
As another test I uploaded the operator logs, and logs from one cfg and pne rs pod as well as all three mongos pods and the kube-proxy.