Because of IP limitation, I cannot expose replicaset with LoadBalancer, so I decided to go with NodePort. I’m running on GKE 1.18 with Percona operator 1.9.0.
I got the CR here:
And expose replicaset only as follow (sharding disabled):
replsets: - name: rs0 expose: enabled: true exposeType: NodePort sharding: enabled: false
- 3 pods are created but restarts constantly with
liveness failed: unreachable servererror.
- Operator logs show
"msg":"failed to reconcile cluster","Request.Namespace":"percona-mongodb","Request.Name":"my-cluster-name","replset":"rs0","error":"dial:: failed to ping mongo: context deadline exceeded".
- The operator seems to use
<node_internal_ip>:<exposed_node_port_ip>as host name to connect to mongo instances. Which is somewhat expected, although I’d expect it to use the node’s external IP.
I tried to connect to one of the mongo pod with:
kubectl exec --stdin --tty my-cluster-name-rs0-0 -- /bin/bash
Then tried to connect to the mongod running on the same pod with:
mongo --host localhost --port 27017 -u <username> -p <password>→ This worked.
mongo --host <service_internal_ip> --port 27017 -u <username> -p <password>→ This also worked.
mongo --host <cluster_internal_ip> --port <exposed_node_port> -u <username> -p <password>→ This DOESN’T worked. I got a connection refused error.
The 2 first connection worked as expected but the last one didn’t. To be clear, the
<cluster_internal_ip> is from the pod’s host IP (or from
kubectl get nodes -owide) and the
<exposed_node_port> is from the nodePort field in the pod’s service.
By curiosity, I tried to run a single percona server:
kubectl run --stdin --tty test-mongo --image percona/percona-server-mongodb:4.4.6-8 --port 27017 -- /bin/bash # Inside test-mongo pod mongod --bind_ip_all --port 27017 kubectl expose test-mongo --type="NodePort" --port 27017
Then I tried all the connection above again. Now all of them worked as expected, including using node’s IP and node port.
So I don’t know why mongo instances in replicaset refused the nodeport connection but a standalone mongo instance did not. Anyway, because of the connectivity issue, the cluster was failed to be reconciled.
Have you ever experienced anything similar? What can be the cause of this?