I am evaluating DataBase solution for my kubernetes cluster, So i installed percona-xtradb-cluster:5.7.19 with three StatefulSet replicas on Bare Metal Server(CentOS) with Kubernetes 1.17.3
To analyze failure situation i shutdown one node and meanwhile my test pods(MYSQL Client) is reading data after every 5 seconds from DB Cluster I observed couple of client requests got timeout but mostly it works fine(99% works).
if i down further 1 node then out of 10 requests 5 or 6 are getting timeout by saying “ERROR 2003 (HY000) Cant connect to MySQL Server on ‘pxc’ (110)”
I tried by adding PodDisruptionBudget with
- minAvailable: 1
- I tried by creating NodePort and infact this is also showing intermittent behavior, mean if i directly try on specific node it sometime time-out.
It look like its due to StatefulSet as service is unable to identified failed replica, i tried with service, headless service and NodePort.
Please suggest what kind of service can help in this regard? How community is using StatefulSet in production?
I searched on net but couldn’t find satisfactory reply.
PS: its very rare situation but this can happen that out of three 2 nodes are down.
Know someone who can answer?