PSMDB pod placement while doing scheduled node maintenance

MongoDB Percona Operator for MongoDB
, ,
1

We are running mongodb clusters with local persistent volumes using PSMDB. Because of these local volumes DBAs often view specific pods as being tied to specific physical nodes. We can manage this using the standard kubernetes statefuset affinity, nodeSelector and antiaffinity rules. However problems arise when doing maintenance tasks like moving or removing specific members.

Scenario 1 (moving a single member):

  • We have a 3-member replicaset running on Node-A (pod-0), Node-B (pod-1), and Node-C (pod-2).
  • We need to decommission Node-A.
  • We want to move pod-0 to a new Node-D, triggering a replication sync on the new node, without outages or modifying the placement of pod-1, pod-2.

Scenario 2 (node decommissioning):

  • We have 5 member replicaset across 5 Nodes
  • We need to remove specific nodes from the cluster (the node holding pod-1 and the node holding pod-3).
  • Since StatefulSets scale down strictly by ordinal index (removing highest numbers first), we cannot simply “scale down” to remove these specific intermediate nodes.

Since the CR applies affinity rules to the entire StatefulSet, is there a recommended workflow to handle these situations?
As the operator uses standard kubernetes statefulsets affinity rules apply to all replicas uniformly.

Currently, our intended solution for the first one is to cordon the target nodes (Node-A), delete the specific pod (pod-0) and its local PVC. Is this the safest approach we could take, or does the operator expose a mechanism to pin specific members to specific nodes?
For scenario 2 we don’t see a way to do it without unnecessary shuffling of pods and doing replication multiple times.

2

I’d like to add some technical context to this discussion.

Currently, the Percona Operator groups members by role: all standard members are put into a single StatefulSet, non-voting members into another, and arbiters into a third.

While Kubernetes provides affinity and anti-affinity rules at the StatefulSet level, our specific requirement is to pin a specific member (pod) to a specific Kubernetes node. This would be straightforward if each member were its own StatefulSet, but the current architecture groups them together, which blocks this level of granularity.

Is there a supported way or workaround to achieve node-specific member placement and independent member lifecycle management?

Thanks,

Meftun.