Hello ,
I have percona cluster setup with 3 nodes. When i check the node wsrep_cluster_node_status=primary. The cluster is hosted in K8s.
Now wsrep_node_name=pxc-0 only .
So when i try to mimic the situation where this pod 0 goes down and pod 1 and pod 2 running the traffic should be handled by either of them post promoting either of them as primary . However the connections get lost and wait for pod 0 to be up and till then the connections are not getting connected to DB. I have ha proxy which also gets 3 replica count , when pod 0 goes down even proxy 0 goes down with error that the pod 0 is not available only 2 nodes available .
can someone help how to make sure this traffic gets routed across other nodes when one goes down
Hello @skybot_win,
Are you using our operator? The Percona operator will automatically handle failover using HAProxy or ProxySQL by correctly configuring the needed K8S services and endpoints. I recommend you try our operator for easier management of your PXC.
Without knowing more of your K8S config/setup, it’s hard to understand what might be going wrong in your environment. It sounds like you’ve got some dependency on your HAProxy pod for node0 pod and you need to remove that.
Hello @matthewb ,
Thanks for the response . Yes we are already using the pxc operator.
so i have taken down the pxc pod 0 and the expectation is ha proxy of 1 and 2 still be working and it should handle traffic. However the ha proxy 1 and 2 also goes down along with ha proxy 0 when pxc pod 0 is killed.
and same time in pxc operator we can see
2024-09-10T08:30:23.879Z INFO Unable to find primary pod for replication. No pod with name or ip like this {"controller": "pxc-controller", "namespace": "review-general", "name": "percona-review-pxc-db", "reconcileID": "a7836e57-612b-4961-a9ad-fde2f28c3e38", "primary name": "percona-review-pxc-db-pxc-0"}
2024-09-10T08:30:24.049Z INFO Unable to find primary pod for replication. No pod with name or ip like this {"controller": "pxc-controller", "namespace": "review-general", "name": "percona-review-pxc-db", "reconcileID": "6f0e2afd-b87d-468f-b2df-d1facf889b43", "primary name": "percona-review-pxc-db-pxc-0"}
2024-09-10T08:30:29.009Z INFO Unable to find primary pod for replication. No pod with name or ip like this {"controller": "pxc-controller", "namespace": "review", "name": "percona-review-pxc-db", "reconcileID": "098a203f-659a-45c6-beaf-50cca7cde9d8", "primary name": "percona-review-pxc-db-pxc-0"}
[mysql] 2024/09/10 08:30:35 packets.go:37: unexpected EOF
[mysql] 2024/09/10 08:30:46 packets.go:37: unexpected EOF
in kubectl events i can see
0s Normal Killing Pod/percona-review-pxc-db-pxc-0 Stopping container logs
0s Normal Killing Pod/percona-review-pxc-db-pxc-0 Stopping container mysqld-exporter
0s Normal Killing Pod/percona-review-pxc-db-pxc-0 Stopping container pxc
0s Normal Killing Pod/percona-review-pxc-db-pxc-0 Stopping container logrotate
0s Normal SuccessfulCreate StatefulSet/percona-review-pxc-db-pxc create Pod percona-review-pxc-db-pxc-0 in StatefulSet percona-review-pxc-db-pxc successful
0s Normal Scheduled Pod/percona-review-pxc-db-pxc-0 Successfully assigned review-general/percona-review-pxc-db-pxc-0 to gke-k8stest-percona-review-v1-88a59f4f-b9sa
0s Warning FailedCreatePodSandBox Pod/percona-review-pxc-db-pxc-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7196d4b946e5fdca2e53bd2696fa6581926df792971c82d9ddcfcf1d70f43cca": plugin type="gke" failed (add): container veth name provided (eth0) already exists
0s Warning Unhealthy Pod/percona-review-pxc-db-haproxy-0 Liveness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2
0s Warning Unhealthy Pod/percona-review-pxc-db-haproxy-1 Readiness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2
0s Warning Unhealthy Pod/percona-review-pxc-db-haproxy-0 Readiness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2
0s Normal Pulling Pod/percona-review-pxc-db-pxc-0 Pulling image "percona/percona-xtradb-cluster-operator:1.12.0"
0s (x4 over 3d13h) Warning Unhealthy Pod/percona-review-pxc-db-haproxy-2 Readiness probe errored: command "/usr/local/bin/readiness-check.sh" timed out
0s Normal Pulled Pod/percona-review-pxc-db-pxc-0 Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0" in 137ms (137ms including waiting)
0s Normal Created Pod/percona-review-pxc-db-pxc-0 Created container pxc-init
0s Normal Started Pod/percona-review-pxc-db-pxc-0 Started container pxc-init
0s Normal Pulling Pod/percona-review-pxc-db-pxc-0 Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-logcollector"
0s Normal Pulled Pod/percona-review-pxc-db-pxc-0 Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-logcollector" in 121ms (121ms including waiting)
0s Normal Created Pod/percona-review-pxc-db-pxc-0 Created container logs
0s Normal Started Pod/percona-review-pxc-db-pxc-0 Started container logs
0s Normal Pulling Pod/percona-review-pxc-db-pxc-0 Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-logcollector"
0s Normal Pulled Pod/percona-review-pxc-db-pxc-0 Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-logcollector" in 103ms (103ms including waiting)
0s Normal Created Pod/percona-review-pxc-db-pxc-0 Created container logrotate
0s Normal Started Pod/percona-review-pxc-db-pxc-0 Started container logrotate
0s Normal Pulling Pod/percona-review-pxc-db-pxc-0 Pulling image "percona/percona-xtradb-cluster:8.0.32-24.2"
0s Normal Pulled Pod/percona-review-pxc-db-pxc-0 Successfully pulled image "percona/percona-xtradb-cluster:8.0.32-24.2" in 185ms (185ms including waiting)
0s Normal Created Pod/percona-review-pxc-db-pxc-0 Created container pxc
0s (x67 over 3d16h) Warning Unhealthy Pod/percona-review-pxc-db-haproxy-1 Readiness probe errored: command "/usr/local/bin/readiness-check.sh" timed out
1s Normal Started Pod/percona-review-pxc-db-pxc-0 Started container pxc
1s Normal Pulled Pod/percona-review-pxc-db-pxc-0 Container image "prom/mysqld-exporter" already present on machine
0s Normal Created Pod/percona-review-pxc-db-pxc-0 Created container mysqld-exporter
0s Normal Started Pod/percona-review-pxc-db-pxc-0 Started container mysqld-exporter
0s (x3 over 3d16h) Normal Pulling Pod/percona-review-pxc-db-haproxy-2 Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy"
0s Normal Pulled Pod/percona-review-pxc-db-haproxy-2 Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy" in 172ms (177ms including waiting)
0s (x3 over 3d16h) Normal Created Pod/percona-review-pxc-db-haproxy-2 Created container pxc-monit
0s (x3 over 3d16h) Normal Started Pod/percona-review-pxc-db-haproxy-2 Started container pxc-monit
0s (x5 over 3d13h) Warning Unhealthy Pod/percona-review-pxc-db-haproxy-2 Readiness probe errored: command "/usr/local/bin/readiness-check.sh" timed out
0s Warning BackOff Pod/percona-review-pxc-db-haproxy-2 Back-off restarting failed container pxc-monit in pod percona-review-pxc-db-haproxy-2_review-general(0d5ec1a9-4733-4f62-8747-5a69212cc92d)
0s (x8 over 3d16h) Normal Pulling Pod/percona-review-pxc-db-haproxy-0 Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy"
0s (x4 over 3d16h) Normal Pulling Pod/percona-review-pxc-db-haproxy-1 Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy"
0s Normal Pulled Pod/percona-review-pxc-db-haproxy-0 Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy" in 164ms (164ms including waiting)
0s (x8 over 3d16h) Normal Created Pod/percona-review-pxc-db-haproxy-0 Created container pxc-monit
0s Normal Pulled Pod/percona-review-pxc-db-haproxy-1 Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy" in 278ms (278ms including waiting)
0s (x4 over 3d16h) Normal Created Pod/percona-review-pxc-db-haproxy-1 Created container pxc-monit
0s (x8 over 3d16h) Normal Started Pod/percona-review-pxc-db-haproxy-0 Started container pxc-monit
0s (x4 over 3d16h) Normal Started Pod/percona-review-pxc-db-haproxy-1 Started container pxc-monit
0s (x68 over 3d16h) Warning Unhealthy Pod/percona-review-pxc-db-haproxy-1 Readiness probe errored: command "/usr/local/bin/readiness-check.sh" timed out
1s (x2 over 3d16h) Warning BackOff Pod/percona-review-pxc-db-haproxy-1 Back-off restarting failed container pxc-monit in pod percona-review-pxc-db-haproxy-1_review-general(df086730-7110-4d03-8c2a-3ee0a3a160a0)
0s (x69 over 3d16h) Warning Unhealthy Pod/percona-review-pxc-db-haproxy-1 Readiness probe errored: command "/usr/local/bin/readiness-check.sh" timed out
0s (x4 over 3d16h) Normal Pulling Pod/percona-review-pxc-db-haproxy-2 Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy"
0s Normal Pulled Pod/percona-review-pxc-db-haproxy-2 Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy" in 98ms (98ms including waiting)
0s (x4 over 3d16h) Normal Created Pod/percona-review-pxc-db-haproxy-2 Created container pxc-monit
0s (x4 over 3d16h) Normal Started Pod/percona-review-pxc-db-haproxy-2 Started container pxc-monit
0s (x5 over 3d16h) Normal Pulling Pod/percona-review-pxc-db-haproxy-1 Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy"
0s Normal Pulled Pod/percona-review-pxc-db-haproxy-1 Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy" in 146ms (146ms including waiting)
0s (x5 over 3d16h) Normal Created Pod/percona-review-pxc-db-haproxy-1 Created container pxc-monit
1s (x5 over 3d16h) Normal Started Pod/percona-review-pxc-db-haproxy-1 Started container pxc-monit
0s (x29 over 3d16h) Warning BackOff Pod/percona-review-pxc-db-haproxy-0 Back-off restarting failed container pxc-monit in pod percona-review-pxc-db-haproxy-0_review-general(7e6f58ed-a165-4c32-90a7-4f44ea756afe)