XtraDB Cluster Nodes Vertical Scaling

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

I have a 3-node Rancher Kubernetes cluster running Percona XtraDB Cluster (PXC). Each Kubernetes worker node currently has 4 vCPU and 16 GB RAM, and I would like to vertically scale each node to 8 vCPU and 32 GB RAM.

The environment is currently configured as follows:

  • Kubernetes: Rancher-managed 3-node cluster
  • PXC: Percona XtraDB Cluster 8.4.7-7.1
  • Percona Operator for MySQL: 1.19.0
  • PXC members: 3
  • Current node resources: 4 vCPU / 16 GB RAM per node
  • Target node resources: 8 vCPU / 32 GB RAM per node
  • Storage: TopoLVM is being used as the CSI/storage solution for PV management
  • Each PXC member is scheduled on a separate Kubernetes node.

I would like to understand the recommended procedure for performing this infrastructure-level vertical scaling while keeping the PXC cluster available and avoiding database downtime.

Specifically:

  1. What is the recommended approach for vertically scaling the Kubernetes nodes hosting PXC?
  2. Should the nodes be scaled one at a time, for example by draining one Kubernetes node, resizing it, bringing it back, and then proceeding to the next node?
  3. Since TopoLVM is being used for PV management, are there any additional considerations when draining and rescheduling a PXC pod whose data is stored on a TopoLVM-managed PV?
  4. What precautions should be taken to ensure that the PXC cluster remains healthy and that all members are fully synchronized before moving to the next node?
  5. Can this process safely be performed while the database is receiving active read/write traffic?
  6. Are there any PXC/Galera-specific considerations regarding ongoing writes, node draining, SST/IST, quorum, or flow control during this process?
  7. Is there a recommended way to verify that the cluster has completely recovered and is synchronized before proceeding with the next node?

The primary requirement is zero database downtime, while allowing normal read/write traffic throughout the scaling operation.

Any recommended procedure or best practices for performing this safely would be appreciated.

2

Hello @Sekhri, Welcome to the forum.

You also might have haproxy pod. You might also be interested if those pods are on different nodes too.

You can check antiAffinity settings in your deployment to confirm if the antiAffinity it being set to place the each pods on different node. It is by default on the different host.

Unless there are two pods active, the cluster should be up.You can always check the cluster status using below command on the active pods,

kubectl exec -it <pxc-pod> -- mysql -uroot -p -e "
SHOW GLOBAL STATUS WHERE Variable_name IN
(
'wsrep_cluster_size',
'wsrep_cluster_status',
'wsrep_connected',
'wsrep_ready',
'wsrep_local_state_comment',
'wsrep_local_recv_queue',
'wsrep_local_send_queue',
'wsrep_flow_control_paused'
);
"

If the cluster keeps primary, you are good.

Another way you can get more cleaner approach, is that you add one worker node in your kubernetes cluster and shift the pods to that new node. In that way all the PXC pods will be UP and running when you work on your nodes to vertically scale it.

In any case you might want to test this first on your test cluster similar setup. That would drive you more confidence.

You can review your gcache size to make sure that when the pod is down, it can survive with IST it is takes longer.

Before moving to next node, you always need to make sure that the node which came back has joined the cluster and is synced. Then only proceed with the next node.

Hope that helps.

3

Yes

Don’t take down more than 1 PXC node at a time

Yes

If that’s your goal, then you should 100% absolutely have more than just 3-node kubernetes cluster. You should have at minimum a 5-node K8S cluster, running your 3-node PXC.