If I want to expose Mongo connection to the outside of Kubernetes cluster, I have to use either NodePort or LoadBalancer, which both work fine.
Now the problem is with NodePort, the node’s IP may change due to kube cluster reconciling. LoadBalancer works fine, but it consumes a lot of resources. Take AWS as provider, each replicaset member will take one LB, and each LB will consume 3 IP addresses, so a mongo replicaset of 3 consumes 3 LBs and 9 IPs. If my kube cluster hosts a lot of mongo, then LB quota and IP limit will be exceeded rapidly.
I thought about using a single shard cluster and expose mongos with LB, so there will be only one LB no matter what. But I don’t know if end-user experience will be any difference from replicaset topology?
Another option may be deploying mongos directly in front of the replicaset without sharding? However mongos doesn’t seem to be designed work that way so I’m not sure if it works.
Do you have any suggestion? To be clear, the need is:
- Expose mongo outside of kube network.
- Minimise the number of LBs and IPs used.
Hello @vhphan ,
this is good question indeed.
I see multiple ways here:
- Run a cluster with one shard. User experience would be the same, but you would need more resources on kubernetes for config server replica set and mongos. This was our thought process when we introduced mongos first - single shard to save resources on load balancers. In 1.6.0 Release notes:
K8SPSMDB-273: Add support for
mongos service to expose a single shard of a MongoDB cluster through one entry point instead of provisioning a load-balancer per replica set node. In the following release, we will add support for multiple shards.
- You can put some TCP ingress service as a front and proxy traffic to MongoDB replica set nodes. You will need to configure ingress to point to the nodes yourself.
So you will have a mapping of ports. Port 33333 on NLB, will translate to Replica Set #1 Node #1 for example. I did something similar back then, but not for mongo.
- I also used this tool in the past: GitHub - DevFactory/smartnat: Kubernetes controller to expose Services with TCP/UDP
It is no longer developed, but it was created specifically to lower the cost. It requires you to run a dedicated EC2 instance with multiple network interfaces (ENIs) and IP-addresses. Then it hooks up with Kubernetes and serves as a TCP proxy for traffic. So similar to (2), but a bit more complex
Let me know if it helps.
Thanks a lot for the answer. Indeed the 2nd option is what we are thinking about. It makes a hint that we might be on the right track. We will try to see if that works.
We will leave the 1st and 3rd options as fail-over, because they looks like workarounds that don’t seem to fit well when scaling to multiple mongo deployments.