# PXC cluster fails after single pod failure **URL:** \ **Category:** Percona Operator for MySQL\ **Created:** [January 10, 2024, 6:34pm UTC](https://forums.percona.com/t/pxc-cluster-fails-after-single-pod-failure/27692 "2024-01-10T18:34:17Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mkl262](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mkl262/32/1267_2.png) [@mkl262](https://forums.percona.com/u/mkl262)\ **Post date:** [January 10, 2024, 6:34pm UTC](https://forums.percona.com/t/pxc-cluster-fails-after-single-pod-failure/27692/1 "2024-01-10T18:34:17Z")
## Description: Hello, I have a 3 node kubernetes cluster running my application, a rook-ceph storage cluster, and a pxc operator and cluster. During HA tests I found that the cluster stops completely when disconnecting the network from one of the nodes, which runs a single pxc pod. The two remaining pods stop the mysqld process and stop accepting any traffic. It is not the expected behavior, and it causes my application to stop functioning. After reconnecting the node to the network, the cluster comes up normally without any intervention. Any input will be appreciated. Thank you. ## Steps to Reproduce: 1. verify that cluster is running 2. disconnect one of the kubernetes nodes from the network(in this case it was only running pxc-1 pod) 3. wait for some time, cluster is still down 4. reconnect the node to the network 5. the cluster is restored ## Version: Operator version: 1.12.0 pxc-operator Helm version: 1.12.1 pxc-db Helm version: 1.12.0 percona-xtradb-cluster image: percona/percona-xtradb-cluster:5.7.43 kubernetes version: v1.25.12 ### Helm Values [pxc-operator-values.yaml.txt](https://forums.percona.com/uploads/short-url/tLEeb20pmwQaICJNh4BaJpIzk5R.txt) (268 Bytes) [pxc-values.yaml.txt](https://forums.percona.com/uploads/short-url/217AATu2WFKXcxnWzLknn6pJnS3.txt) (2.9 KB) ## Logs: See attached Log files [pxc-0.log](https://forums.percona.com/uploads/short-url/8WhRNFKjg6YQctF7pCNhMUBcLOg.log) (53.3 KB) [pxc-1.log](https://forums.percona.com/uploads/short-url/kxV2MbLG9cLR3TxAJIOWYmygbkI.log) (25.3 KB) [pxc-2.log](https://forums.percona.com/uploads/short-url/vG0aknXwAmEoWegNk2lMUwqPomt.log) (49.5 KB) [pxc-operator-pod.log](https://forums.percona.com/uploads/short-url/5yUiOuEDuIAUGKfnteZTsl75Vaz.log) (631.7 KB) ## Expected Result: The two remaining pods continue the traffic as usual. ## Actual Result: Both remaining pods stop the mysqld process and stop excepting any traffic ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [January 15, 2024, 10:10am UTC](https://forums.percona.com/t/pxc-cluster-fails-after-single-pod-failure/27692/2 "2024-01-15T10:10:45Z")
Hello @mkl262 , how do you disconnect the network from one of the nodes? Also which CNI driver do you use? Seems that you might be breaking the connectivity as a whole. I tried to reproduce it just now on GKE with chaos mesh, where I introduced network partinitioning between PXC nodes. And it all recovered just fine. ---
**Author:** ![mkl262](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mkl262/32/1267_2.png) [@mkl262](https://forums.percona.com/u/mkl262)\ **Post date:** [February 13, 2024, 10:30am UTC](https://forums.percona.com/t/pxc-cluster-fails-after-single-pod-failure/27692/3 "2024-02-13T10:30:44Z")
Hi, I disconnect the nodes by disabling the network interface, or by physically disconnecting the connection. I use the flannel CNI. I also wasnt able to reproduce this issue in a cloud environment, I tried in AWS EKS, and it worked as expected ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [February 19, 2024, 10:00am UTC](https://forums.percona.com/t/pxc-cluster-fails-after-single-pod-failure/27692/4 "2024-02-19T10:00:13Z")
@mkl262 how did you reproduce it in EKS? I may try to do it as well and see. ---
**Author:** ![mkl262](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mkl262/32/1267_2.png) [@mkl262](https://forums.percona.com/u/mkl262)\ **Post date:** [March 11, 2024, 9:25am UTC](https://forums.percona.com/t/pxc-cluster-fails-after-single-pod-failure/27692/5 "2024-03-11T09:25:54Z")
Hi, I wasn’t able to reproduce it in EKS, the cluster was able to recover after losing one of its nodes. I also upgraded the operator to 1.14, The mysql to 8.0.28, and the rke kubernetes to 1.27.11, and the issue is still reproducible.