We have reserved 3 nodes to run the replicaset. It turns out that each pod is placed to a different node - which makes perfectly sense. The problem is though that 2 nodes are in most cases idle which is a waste of resources. Obviously that is the way it is with a single replicaset. But how about if I have multiple replicasets (basically multiple operators on different namespaces) ?
Question: How can I “stack” multiple replicasets on a 3-node node-pool (GKE) ?
Operator configures affinity rules for mongodb pods that suggests to kubernetes not to schedule the pods to the same node. You can disable these rules through cr.yaml if you want.
To have multiple replica sets:
As you mentioned you can deploy multiple operators on different namespaces or you can have a single operator deployment and deploy multiple custom resources (PerconaServerMongoDB) into the same namespace.
Operator also can deploy multiple replica sets for a single custom resource but that feature is tightly coupled with sharding.