Single StatefulSet for all replica

MySQL & MariaDB Percona Operator for MySQL
1

MySQL Replicas are managed with Single Statefulset. Each replica leverage the EBS volume for storage. EBS is AZ specific. Is it expected that, All replicas must run in same AZ. So that, on restart, pod will never end up with inaccessible EBS vol. But if we use AZ anti-affinity for HA, on restart, pods can come up in any other AZ and might end up in pending state.

In testing also, I faced this issue. It is the expected result for StatefulSet. But with Percona, Is there any protection available to avoid this? Any pointers?

2

Hello @midson ,

thank you for the question.

This is not about Percona, but how Kubernetes works in general. But good news are, that there is a solution called topology aware volumes.

Please have a look at the following articles:

  1. Troubleshooting topology aware volumes in EKS | AWS re:Post
  2. Scaling Kubernetes with Karpenter: Advanced Scheduling with Pod Affinity and Volume Topology Awareness | Containers (Karpenter might help with properly scaling the cluster)

Please let me know if it helps. But it is a good item that we should mention in our docs for clouds. Will make sure we capture it.

1 Like
3

@Sergey_Pronin Thanks for the reply.
Still not very clear about how the topology aware volume will help to solve the above situation.
Let me try to test it.

4

@Sergey_Pronin This will not help. Bcoz Topology-aware volume’s primary function lies in influencing the initial placement of PVs during the provisioning process and don’t have the ability to relocate existing PVs after they’ve been provisioned. Other technics like selector and affinity will only help and does not guarantee. Plz correct me, if my understanding is not correct.