Hello, I have a 3 node kubernetes cluster running my application, a rook-ceph storage cluster, and a pxc operator and cluster.
During HA tests I found that the cluster stops completely when disconnecting the network from one of the nodes, which runs a single pxc pod. The two remaining pods stop the mysqld process and stop accepting any traffic.
It is not the expected behavior, and it causes my application to stop functioning.
After reconnecting the node to the network, the cluster comes up normally without any intervention.
Any input will be appreciated. Thank you.
- verify that cluster is running
- disconnect one of the kubernetes nodes from the network(in this case it was only running pxc-1 pod)
- wait for some time, cluster is still down
- reconnect the node to the network
- the cluster is restored
Operator version: 1.12.0
pxc-operator Helm version: 1.12.1
pxc-db Helm version: 1.12.0
percona-xtradb-cluster image: percona/percona-xtradb-cluster:5.7.43
kubernetes version: v1.25.12
The two remaining pods continue the traffic as usual.
Both remaining pods stop the mysqld process and stop excepting any traffic