I am evaluating DataBase solution for my kubernetes cluster, So i installed percona-xtradb-cluster with three replica. To analyze failure situation i shutdown one node and meanwhile my test pods is reading data after every 5 seconds from DB Cluster but i observed out of 10 request 1 or 2 are getting timeout by saying “ERROR 2003 (HY000) Cant connect to MySQL Server on ‘pxc’ (110)” , if i further down another node which mean only one node is up error percentage is about 60% which out of 10 only 4 are successful
- I tried by adding PodDisruptionBudget with minAvailable: 1
- I tried by creating NodePort and infact this is also showing intermittent behavior mean if i directly try on specific node it some time time out.
One more Observation, if i shutdown primary node then it doesn’t work whole cluster is unavailable, infact still two nodes are up
2020-06-17T13:18:49.027004Z 0 [Note] WSREP: (0c831c98, ‘tcp://0.0.0.0:4567’) reconnecting to d4b530ac (tcp://10.244.2.1:4567), attempt 24002020-06-17T13:18:52.027693Z 0 [Note] WSREP: (0c831c98, ‘tcp://0.0.0.0:4567’) connection to peer 00000000 with addr tcp://10.244.2.1:4567 timed out, no messages seen in PT3S2020-06-17T13:18:56.028533Z 0 [Note] WSREP: (0c831c98, ‘tcp://0.0.0.0:4567’) connection to peer 00000000 with addr tcp://10.244.2.1:4567 timed out, no messages seen in PT3S