Stateful service time out if Nodes are down in Kubernetes cluster

MySQL & MariaDB Percona Operator for MySQL
,
1

I am evaluating DataBase solution for my kubernetes cluster, So i installed percona-xtradb-cluster:5.7.19 with three StatefulSet replicas on Bare Metal Server(CentOS) with Kubernetes 1.17.3

To analyze failure situation i shutdown one node and meanwhile my test pods(MYSQL Client) is reading data after every 5 seconds from DB Cluster I observed couple of client requests got timeout but mostly it works fine(99% works).

if i down further 1 node then out of 10 requests 5 or 6 are getting timeout by saying “ERROR 2003 (HY000) Cant connect to MySQL Server on ‘pxc’ (110)”

I tried by adding PodDisruptionBudget with

  1. minAvailable: 1
  2. I tried by creating NodePort and infact this is also showing intermittent behavior, mean if i directly try on specific node it sometime time-out.

It look like its due to StatefulSet as service is unable to identified failed replica, i tried with service, headless service and NodePort.

Please suggest what kind of service can help in this regard? How community is using StatefulSet in production?

I searched on net but couldn’t find satisfactory reply.

PS: its very rare situation but this can happen that out of three 2 nodes are down.

Know someone who can answer?

2

Hello @nethra

  1. ,

    thank you for submitting this. To help you here I would need a bit more details:

    Your cr.yaml

  2. Output of the kubectl get sts YOUR_STS -o yaml when you see timeouts
  3. Output of the kubectl get nodes and kubectl get pods -n PXC_NAMESPACE when you see timeouts

  4. There are many variables here that can cause this behavior., but there is nothing specific about StatefulSets - they use regular primitives: ReplicaSets, Services, Endpoints, etc.

    The query is routed through external loadbalancer (ex. ELB)

  5. Gets into k8s through a Service object on any node (kube-proxy does the magic)
  6. Service resource routes the traffic to the pods of the proxy (HAProxy/ProxySQL)
  7. Proxy routes the traffic to pods

We need to understand which step is causing timeouts. I would suspect either external load balancer or kube-proxy routing the traffic to dead nodes.

1 Like