# Kubernetes: percona-xtradb-cluster-operator fails to initialize - readiness probe failed **URL:** \ **Category:** Percona Operator for MySQL\ **Created:** [January 19, 2023, 1:20am UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680 "2023-01-19T01:20:05Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [January 19, 2023, 1:20am UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/1 "2023-01-19T01:20:05Z")
Hi, I have followed the instruction to install the operator in Kubernetes @ [Generic Kubernetes installation - Percona Operator for MySQL based on Percona XtraDB Cluster](https://docs.percona.com/percona-operator-for-mysql/pxc/kubernetes.html) I am able to run the operator pod, but the cluster pod fails to come up. Please let me know how to resolve it. **\>kubectl get pods** NAME READY STATUS RESTARTS AGE percona-xtradb-cluster-operator-5c95894bbf-hwv5h 1/1 Running 0 122m _test-cluster-pxc-0 2/3 Running 0 13m_ **\>kubectl describe pod/test-cluster-pxc-0** Events: Type Reason Age From Message * * * Warning FailedScheduling 7m28s (x3 over 7m39s) default-scheduler 0/20 nodes are available: 20 pod has unbound immediate PersistentVolumeClaims. Normal Scheduled 7m24s default-scheduler Successfully assigned pxc/test-cluster-pxc-0 to 10.47.244.15 Normal SuccessfulAttachVolume 7m24s attachdetach-controller AttachVolume.Attach succeeded for volume “pvc-ab3faef3-3a15-4592-8d83-774212a9c926” Normal Pulling 7m6s kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.12.0” Normal Pulled 7m6s kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.12.0” in 693.444389ms Normal Created 7m6s kubelet Created container pxc-init Normal Pulling 7m5s kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.12.0-logcollector” Normal Started 7m5s kubelet Started container pxc-init Normal Created 7m4s kubelet Created container logs Normal Pulled 7m4s kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.12.0-logcollector” in 663.401732ms Normal Created 7m3s kubelet Created container logrotate Normal Started 7m3s kubelet Started container logs Normal Pulling 7m3s kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.12.0-logcollector” Normal Pulled 7m3s kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.12.0-logcollector” in 645.501357ms Normal Started 7m2s kubelet Started container logrotate Normal Pulling 7m2s kubelet Pulling image “percona/percona-xtradb-cluster:8.0.29-21.1” Normal Pulled 7m2s kubelet Successfully pulled image “percona/percona-xtradb-cluster:8.0.29-21.1” in 646.002133ms Normal Created 7m2s kubelet Created container pxc Normal Started 7m1s kubelet Started container pxc Warning DNSConfigForming 5m52s (x6 over 7m7s) kubelet Search Line limits were exceeded, some search paths have been omitted, the applied search line is: pxc.svc.cluster.local svc.cluster.local cluster.local Warning Unhealthy 5m17s (x3 over 6m17s) kubelet Readiness probe failed: ERROR 2003 (HY000): Can’t connect to MySQL server on ‘10.20.80.134:33062’ (111) - [[‘’ == \P\r\i\m\a\r\y]] - exit 1 Warning Unhealthy 112s kubelet Liveness probe failed: ERROR 2003 (HY000): Can’t connect to MySQL server on ‘10.20.80.134:33062’ (111) - [[-n ‘’]] - exit 1 **\>kubect logs pod/test-cluster-pxc-0** Percona XtraDB Cluster: Finding peers 2023/01/19 01:06:22 Peer finder enter 2023/01/19 01:06:22 Determined Domain to be pxc.svc.cluster.local **2023/01/19 01:06:22 lookup test-cluster-pxc-unready on 10.21.0.10:53: no such host** **2023/01/19 01:06:23 lookup test-cluster-pxc-unready on 10.21.0.10:53: no such host** ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [January 20, 2023, 11:27am UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/2 "2023-01-20T11:27:41Z")
Hi @Ravi_Kumar_Pokala , As I can see you have some issues connected with DNS. We need to know more about your k8s setup. E.g. which k8s version do you use? So, please tell us more about your k8s deployment. ---
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [February 2, 2023, 6:49pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/3 "2023-02-02T18:49:14Z")
@Slava_Sarzhan : We are using the vendor-managed(Platform 9) Kubernetes deployment. please find the version information below. Server Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.5”, GitCommit:“6b1d87acf3c8253c123756b9e61dac642678305f”, GitTreeState:“clean”, BuildDate:“2021-03-18T01:02:01Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”} ---
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [February 4, 2023, 1:53am UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/4 "2023-02-04T01:53:10Z")
Cluster creations fails with below error: **\>kubect get pods** NAME READY STATUS RESTARTS AGE percona-xtradb-cluster-operator-5c95894bbf-7g52j 1/1 Running 0 5m24s test-cluster-haproxy-0 0/2 Pending 0 2m26s **Operator logs** > k logs pod/percona-xtradb-cluster-operator-5c95894bbf-7g52j 2023-02-04T01:48:40.334Z ERROR Reconciler error {“controller”: “perconaxtradbcluster-controller”, “object”: {“name”:“test-cluster”,“namespace”:“pxc”}, “namespace”: “pxc”, “name”: “test-cluster”, “reconcileID”: “75d74f89-417d-4015-b720-74714e3725ae”, “error”: “PodDisruptionBudget for test-cluster-haproxy: reconcile pdb: get object: no matches for kind "PodDisruptionBudget" in version "policy/v1"”, “errorVerbose”: “no matches for kind "PodDisruptionBudget" in version "policy/v1"\nget object\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).createOrUpdate\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1299\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).reconcilePDB\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:945\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).deploy\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:685\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:312\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm\_amd64.s:1594\nreconcile pdb\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).reconcilePDB\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:945\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).deploy\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:685\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:312\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm\_amd64.s:1594\nPodDisruptionBudget for test-cluster-haproxy\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).deploy\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:687\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(\*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:312\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm\_amd64.s:1594”} [sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326 [sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273 [sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234 ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [February 6, 2023, 10:28am UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/5 "2023-02-06T10:28:04Z")
@Ravi_Kumar_Pokala PXCO v1.12.0 does not support K8S 1.20. Have a look at our official doc: > **[System Requirements - Percona Operator for MySQL based on Percona XtraDB Cluster](https://docs.percona.com/percona-operator-for-mysql/pxc/System-Requirements.html)** > > Documentation It is connected with `PodDisruptionBudget` API version. We started to use PDB API v1 to support k8s \>= 1.21 in PXCO v1.12.0 release. ---
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [February 7, 2023, 1:06am UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/6 "2023-02-07T01:06:49Z")
@Slava_Sarzhan : I have tried with with PXCO v1.11.0 now I am running into another issue. Haproxy stays in Pending state **\>kubectl get pods** NAME READY STATUS RESTARTS AGE mytest-cluster-haproxy-0 0/3 Pending 0 14m mytest-cluster-pxc-0 3/3 Running 0 14m mytest-cluster-pxc-1 3/3 Running 0 11m mytest-cluster-pxc-2 3/3 Running 0 8m44s percona-xtradb-cluster-operator-655769795d-h8q67 1/1 Running 0 112m **\>kubectl get sts** NAME READY AGE mytest-cluster-haproxy 0/3 14m mytest-cluster-pxc 3/3 14m **\>kubectl describe pod/mytest-cluster-haproxy-0** Name: mytest-cluster-haproxy-0 Namespace: pxc Priority: 0 Service Account: default Node: Labels: [app.kubernetes.io/component=haproxy](http://app.kubernetes.io/component=haproxy) [app.kubernetes.io/instance=mytest-cluster](http://app.kubernetes.io/instance=mytest-cluster) [app.kubernetes.io/managed-by=percona-xtradb-cluster-operator](http://app.kubernetes.io/managed-by=percona-xtradb-cluster-operator) [app.kubernetes.io/name=percona-xtradb-cluster](http://app.kubernetes.io/name=percona-xtradb-cluster) [app.kubernetes.io/part-of=percona-xtradb-cluster](http://app.kubernetes.io/part-of=percona-xtradb-cluster) controller-revision-hash=mytest-cluster-haproxy-5df959fb5b rack=rack-22 [statefulset.kubernetes.io/pod-name=mytest-cluster-haproxy-0](http://statefulset.kubernetes.io/pod-name=mytest-cluster-haproxy-0) Annotations: [iam.amazonaws.com/role:](http://iam.amazonaws.com/role:) role-arn [percona.com/configuration-hash:](http://percona.com/configuration-hash:) f0cf38deede8503f8be79b49927dba41 Status: Pending IP: IPs: Controlled By: StatefulSet/mytest-cluster-haproxy Containers: haproxy: Image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy Ports: 3306/TCP, 3307/TCP, 3309/TCP, 33062/TCP, 33060/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP Limits: cpu: 700m memory: 1G Requests: cpu: 600m memory: 1G Liveness: exec [/usr/local/bin/liveness-check.sh] delay=120s timeout=5s period=30s #success=1 #failure=4 Readiness: exec [/usr/local/bin/readiness-check.sh] delay=135s timeout=1s period=5s #success=1 #failure=3 Environment Variables from: mytest-cluster-env-vars-haproxy Secret Optional: true Environment: PXC\_SERVICE: mytest-cluster-pxc LIVENESS\_CHECK\_TIMEOUT: 5 READINESS\_CHECK\_TIMEOUT: 1 Mounts: /etc/haproxy-custom/ from haproxy-custom (rw) /etc/haproxy/pxc from haproxy-auto (rw) /etc/mysql/haproxy-env-secret from mytest-cluster-env-vars-haproxy (rw) /etc/mysql/mysql-users-secret from mysql-users-secret-file (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-9s5n6 (ro) pxc-monit: Image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy Port: Host Port: Args: /usr/bin/peer-list -on-change=/usr/bin/add\_pxc\_nodes.sh -service=$(PXC\_SERVICE) Limits: cpu: 600m memory: 2G Requests: cpu: 500m memory: 1G Environment Variables from: mytest-cluster-env-vars-haproxy Secret Optional: true Environment: PXC\_SERVICE: mytest-cluster-pxc Mounts: /etc/haproxy-custom/ from haproxy-custom (rw) /etc/haproxy/pxc from haproxy-auto (rw) /etc/mysql/haproxy-env-secret from mytest-cluster-env-vars-haproxy (rw) /etc/mysql/mysql-users-secret from mysql-users-secret-file (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-9s5n6 (ro) my-sidecar-1: Image: busybox Port: Host Port: Command: /bin/sh Args: -c while true; do trap ‘exit 0’ SIGINT SIGTERM SIGQUIT SIGKILL; done; Limits: cpu: 200m memory: 200M Requests: cpu: 100m memory: 100M Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-9s5n6 (ro) Volumes: haproxy-custom: Type: ConfigMap (a volume populated by a ConfigMap) Name: mytest-cluster-haproxy Optional: true haproxy-auto: Type: EmptyDir (a temporary directory that shares a pod’s lifetime) Medium: SizeLimit: mysql-users-secret-file: Type: Secret (a volume populated by a Secret) SecretName: internal-mytest-cluster Optional: false mytest-cluster-env-vars-haproxy: Type: Secret (a volume populated by a Secret) SecretName: mytest-cluster-env-vars-haproxy Optional: true default-token-9s5n6: Type: Secret (a volume populated by a Secret) SecretName: default-token-9s5n6 Optional: false QoS Class: Burstable Node-Selectors: Tolerations: [node.kubernetes.io/not-ready:NoExecute](http://node.kubernetes.io/not-ready:NoExecute) op=Exists for 300s [node.kubernetes.io/unreachable:NoExecute](http://node.kubernetes.io/unreachable:NoExecute) op=Exists for 300s Events: ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [February 7, 2023, 6:38pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/7 "2023-02-07T18:38:13Z")
@Ravi_Kumar_Pokala Do you see any events? You need to understand why this pod has a Pending state. Also, the output of this comment can be helpful: `kubectl get events --sort-by=.metadata.creationTimestamp -w` ---
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [February 7, 2023, 7:49pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/8 "2023-02-07T19:49:00Z")
> [@Slava\_Sarzhan](#): > > kubectl get events --sort-by=.metadata.creationTimestamp -w @Slava_Sarzhan : below is the event log related to cluster-haproxy-0, it doesn’t show any issues as such. I don’t see PVC created for HAPROXY as well. Let me know if you need to any other info to find the cause. 0s Normal SuccessfulCreate statefulset/mytest-cluster-haproxy create Pod mytest-cluster-haproxy-0 in StatefulSet mytest-cluster-haproxy successful ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [February 7, 2023, 7:58pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/9 "2023-02-07T19:58:46Z")
HaProxy is stainless. We do not have PVCs for these pods. In your output of the following command ` **>kubectl describe pod/mytest-cluster-haproxy-0** ` I do not see any info in `Evenst:` section. Did you remove it? If yes, please resent it. ---
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [February 7, 2023, 10:13pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/10 "2023-02-07T22:13:47Z")
@Slava_Sarzhan : Nothing captured in the Events: it is none Tolerations: [node.kubernetes.io/not-ready:NoExecute](http://node.kubernetes.io/not-ready:NoExecute) op=Exists for 300s [node.kubernetes.io/unreachable:NoExecute](http://node.kubernetes.io/unreachable:NoExecute) op=Exists for 300s Events: \ ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [February 8, 2023, 1:43pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/11 "2023-02-08T13:43:32Z")
@Ravi_Kumar_Pokala try to check the output of the following command > kubectl describe sts mytest-cluster-haproxy It can help you to understand what is wrong. And also provide your CR. ---
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [February 8, 2023, 7:02pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/12 "2023-02-08T19:02:11Z")
> [@Slava\_Sarzhan](#): > > kubectl describe sts mytest-cluster-haproxy @Slava_Sarzhan : I checked the “mytest-cluster-haproxy” statefulset description it doesn’t provide enough information about what the issue is. **Please find the description below. I have attached the CR.yaml for your reference.** > kubectl describe sts mytest-cluster-haproxy > Name: mytest-cluster-haproxy > Namespace: pxc > CreationTimestamp: Tue, 07 Feb 2023 11:31:51 -0800 > Selector: [app.kubernetes.io/component=haproxy,app.kubernetes.io/instance=mytest-cluster,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster](http://app.kubernetes.io/component=haproxy,app.kubernetes.io/instance=mytest-cluster,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster) > Labels: > Annotations: [percona.com/last-config-hash:](http://percona.com/last-config-hash:) > eyJyZXBsaWNhcyI6Mywic2VsZWN0b3IiOnsibWF0Y2hMYWJlbHMiOnsiYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50IjoiaGFwcm94eSIsImFwcC5rdWJlcm5ldGVzLmlvL2luc3… > Replicas: 3 desired | 1 total > Update Strategy: RollingUpdate > Partition: 0 > Pods Status: 0 Running / 1 Waiting / 0 Succeeded / 0 Failed > Pod Template: > Labels: [app.kubernetes.io/component=haproxy](http://app.kubernetes.io/component=haproxy) > [app.kubernetes.io/instance=mytest-cluster](http://app.kubernetes.io/instance=mytest-cluster) > [app.kubernetes.io/managed-by=percona-xtradb-cluster-operator](http://app.kubernetes.io/managed-by=percona-xtradb-cluster-operator) > [app.kubernetes.io/name=percona-xtradb-cluster](http://app.kubernetes.io/name=percona-xtradb-cluster) > [app.kubernetes.io/part-of=percona-xtradb-cluster](http://app.kubernetes.io/part-of=percona-xtradb-cluster) > rack=rack-22 > Annotations: [iam.amazonaws.com/role:](http://iam.amazonaws.com/role:) role-arn > [percona.com/configuration-hash:](http://percona.com/configuration-hash:) f0cf38deede8503f8be79b49927dba41 > Service Account: default > Containers: > haproxy: > Image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy > Ports: 3306/TCP, 3307/TCP, 3309/TCP, 33062/TCP, 33060/TCP > Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP > Limits: > cpu: 700m > memory: 1G > Requests: > cpu: 600m > memory: 1G > Liveness: exec [/usr/local/bin/liveness-check.sh] delay=120s timeout=5s period=30s #success=1 #failure=4 > Readiness: exec [/usr/local/bin/readiness-check.sh] delay=135s timeout=1s period=5s #success=1 #failure=3 > Environment Variables from: > mytest-cluster-env-vars-haproxy Secret Optional: true > Environment: > PXC\_SERVICE: mytest-cluster-pxc > LIVENESS\_CHECK\_TIMEOUT: 5 > READINESS\_CHECK\_TIMEOUT: 1 > Mounts: > /etc/haproxy-custom/ from haproxy-custom (rw) > /etc/haproxy/pxc from haproxy-auto (rw) > /etc/mysql/haproxy-env-secret from mytest-cluster-env-vars-haproxy (rw) > /etc/mysql/mysql-users-secret from mysql-users-secret-file (rw) > pxc-monit: > Image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy > Port: \ > Host Port: \ > Args: > /usr/bin/peer-list > -on-change=/usr/bin/add\_pxc\_nodes.sh > -service=$(PXC\_SERVICE) > Limits: > cpu: 600m > memory: 2G > Requests: > cpu: 500m > memory: 1G > Environment Variables from: > mytest-cluster-env-vars-haproxy Secret Optional: true > Environment: > PXC\_SERVICE: mytest-cluster-pxc > Mounts: > /etc/haproxy-custom/ from haproxy-custom (rw) > /etc/haproxy/pxc from haproxy-auto (rw) > /etc/mysql/haproxy-env-secret from mytest-cluster-env-vars-haproxy (rw) > /etc/mysql/mysql-users-secret from mysql-users-secret-file (rw) > my-sidecar-1: > Image: busybox > Port: \ > Host Port: \ > Command: > /bin/sh > Args: > -c > while true; do trap ‘exit 0’ SIGINT SIGTERM SIGQUIT SIGKILL; done; > Limits: > cpu: 200m > memory: 200M > Requests: > cpu: 100m > memory: 100M > Environment: \ > Mounts: \ > Volumes: > haproxy-custom: > Type: ConfigMap (a volume populated by a ConfigMap) > Name: mytest-cluster-haproxy > Optional: true > haproxy-auto: > Type: EmptyDir (a temporary directory that shares a pod’s lifetime) > Medium: > SizeLimit: \ > mysql-users-secret-file: > Type: Secret (a volume populated by a Secret) > SecretName: internal-mytest-cluster > Optional: false > mytest-cluster-env-vars-haproxy: > Type: Secret (a volume populated by a Secret) > SecretName: mytest-cluster-env-vars-haproxy > Optional: true > Volume Claims: \ > Events: > [mytest-cryaml.txt](https://forums.percona.com/uploads/short-url/80cE0NHeClZ7aOSAfJHjGxK2uau.txt) (15.9 KB) > \ ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [February 9, 2023, 6:40pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/13 "2023-02-09T18:40:33Z")
@Ravi_Kumar_Pokala do you have a custom scheduler? As I can see schedulerName option is uncommented for haproxy. ---
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [February 10, 2023, 9:49pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/14 "2023-02-10T21:49:30Z")
@Slava_Sarzhan : No custom scheduler, I have commented on it then cluster creation works!!, thank you. I am running into another issue while restoring a new cluster from a backup. **Steps followed:** 1. Created a cluster and added new table to MYSQL database. 2. Ran a backup to store it in a PVC. 3. Created another cluster - name “restore-mytest-cluster” 4. Applied restore spec file _apiVersion: [pxc.percona.com/v1](http://pxc.percona.com/v1)_ _kind: PerconaXtraDBClusterRestore_ _metadata:_ - name: restore1\* _spec:_ - pxcCluster: restore-mytest-cluster\* - backupName: backup-to-restore-emp-table\* 1. “restore-mytest-cluster” was stopped and pxc-pod got recreated but failed to run successfully with heath check failure. 2. Event logs shows login failed for user “monitor” **\>kubectl describe pod/restore-mytest-cluster-pxc-0** Name: restore-mytest-cluster-pxc-0 Namespace: pxc Priority: 0 Service Account: default Node: 10.47.244.15/10.47.244.15 Start Time: Fri, 10 Feb 2023 13:31:10 -0800 Labels: [app.kubernetes.io/component=pxc](http://app.kubernetes.io/component=pxc) [app.kubernetes.io/instance=restore-mytest-cluster](http://app.kubernetes.io/instance=restore-mytest-cluster) [app.kubernetes.io/managed-by=percona-xtradb-cluster-operator](http://app.kubernetes.io/managed-by=percona-xtradb-cluster-operator) [app.kubernetes.io/name=percona-xtradb-cluster](http://app.kubernetes.io/name=percona-xtradb-cluster) [app.kubernetes.io/part-of=percona-xtradb-cluster](http://app.kubernetes.io/part-of=percona-xtradb-cluster) controller-revision-hash=restore-mytest-cluster-pxc-6cc754c9f5 [statefulset.kubernetes.io/pod-name=restore-mytest-cluster-pxc-0](http://statefulset.kubernetes.io/pod-name=restore-mytest-cluster-pxc-0) Annotations: [cni.projectcalico.org/podIP:](http://cni.projectcalico.org/podIP:) 10.20.80.154/32 [cni.projectcalico.org/podIPs:](http://cni.projectcalico.org/podIPs:) 10.20.80.154/32 [kubernetes.io/limit-ranger:](http://kubernetes.io/limit-ranger:) LimitRanger plugin set: cpu, memory limit for container logs; cpu, memory limit for container logrotate [Percona](http://percona.com/configuration-hash:) d41d8cd98f00b204e9800998ecf8427e [Percona](http://percona.com/ssl-hash:) eca633ffe87d12049ad0f444e2485101 [Percona](http://percona.com/ssl-internal-hash:) cc20d1bd6e04d4d34816e6f944711423 Status: Running IP: 10.20.80.154 IPs: IP: 10.20.80.154 Controlled By: StatefulSet/restore-mytest-cluster-pxc Init Containers: pxc-init: Container ID: docker://544436ecae60bad7b8de3c88a486d056c1b48c66da3ce2690225c57137a19aad Image: percona/percona-xtradb-cluster-operator:1.11.0 Image ID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:69501813d433aba1b9bd0babf7b7033000696da2a4c7fc582dac00c79a100c82 Port: Host Port: Command: /pxc-init-entrypoint.sh State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 10 Feb 2023 13:31:20 -0800 Finished: Fri, 10 Feb 2023 13:31:20 -0800 Ready: True Restart Count: 0 Limits: cpu: 1 ephemeral-storage: 1400M memory: 1400M Requests: cpu: 600m ephemeral-storage: 1G memory: 1G Environment: Mounts: /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-9s5n6 (ro) Containers: logs: Container ID: docker://34436a072f376763f9d9f3aa00cfe0ac520d57ab3ad27595bf679cc5ca0b894b Image: percona/percona-xtradb-cluster-operator:1.11.0-logcollector Image ID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:fda6ca8f7bf95e86808ae60e305cda8007f7a886688f9bcfa5a566d2b43d05c0 Port: Host Port: State: Running Started: Fri, 10 Feb 2023 13:31:22 -0800 Ready: True Restart Count: 0 Limits: cpu: 1500m memory: 512Mi Requests: cpu: 200m memory: 100M Environment Variables from: restore-mytest-cluster-log-collector Secret Optional: true Environment: LOG\_DATA\_DIR: /var/lib/mysql POD\_NAMESPASE: pxc (v1:metadata.namespace) POD\_NAME: restore-mytest-cluster-pxc-0 (v1:metadata.name) Mounts: /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-9s5n6 (ro) logrotate: Container ID: docker://400e529139c3d32d51383ab16d0373c3aeb90569d8695fc9e87026a3eadefe0b Image: percona/percona-xtradb-cluster-operator:1.11.0-logcollector Image ID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:fda6ca8f7bf95e86808ae60e305cda8007f7a886688f9bcfa5a566d2b43d05c0 Port: Host Port: Args: logrotate State: Running Started: Fri, 10 Feb 2023 13:31:23 -0800 Ready: True Restart Count: 0 Limits: cpu: 1500m memory: 512Mi Requests: cpu: 200m memory: 100M Environment: SERVICE\_TYPE: mysql MONITOR\_PASSWORD: \ Optional: false Mounts: /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-9s5n6 (ro) pxc: Container ID: docker://8e22f96e5325f9531797c7edc8e87c376150c192ecf7c46f7cbe3e03e1977dd9 Image: percona/percona-xtradb-cluster:8.0.27-18.1 Image ID: docker-pullable://percona/percona-xtradb-cluster@sha256:a0fced75ecd2cd164dd9937917440911aed972476d48a2b8a84fe832bc67e43a Ports: 3306/TCP, 4444/TCP, 4567/TCP, 4568/TCP, 33062/TCP, 33060/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP Command: /var/lib/mysql/pxc-entrypoint.sh Args: mysqld State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 10 Feb 2023 13:45:23 -0800 Finished: Fri, 10 Feb 2023 13:47:43 -0800 Ready: False Restart Count: 6 Limits: cpu: 1 ephemeral-storage: 1400M memory: 1400M Requests: cpu: 600m ephemeral-storage: 1G memory: 1G Liveness: exec [/var/lib/mysql/liveness-check.sh] delay=100s timeout=5s period=10s #success=1 #failure=3 Readiness: exec [/var/lib/mysql/readiness-check.sh] delay=125s timeout=15s period=30s #success=1 #failure=5 Environment Variables from: restore-mytest-cluster-env-vars-pxc Secret Optional: true Environment: PXC\_SERVICE: restore-mytest-cluster-pxc-unready MONITOR\_HOST: % MYSQL\_ROOT\_PASSWORD: \ Optional: false XTRABACKUP\_PASSWORD: \ Optional: false MONITOR\_PASSWORD: \ Optional: false LOG\_DATA\_DIR: /var/lib/mysql IS\_LOGCOLLECTOR: yes CLUSTER\_HASH: 2828535 OPERATOR\_ADMIN\_PASSWORD: \ Optional: false LIVENESS\_CHECK\_TIMEOUT: 5 READINESS\_CHECK\_TIMEOUT: 15 Mounts: /etc/my.cnf.d from auto-config (rw) /etc/mysql/mysql-users-secret from mysql-users-secret-file (rw) /etc/mysql/ssl from ssl (rw) /etc/mysql/ssl-internal from ssl-internal (rw) /etc/mysql/vault-keyring-secret from vault-keyring-secret (rw) /etc/percona-xtradb-cluster.conf.d from config (rw) /tmp from tmp (rw) /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-9s5n6 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: datadir: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: datadir-restore-mytest-cluster-pxc-0 ReadOnly: false tmp: Type: EmptyDir (a temporary directory that shares a pod’s lifetime) Medium: SizeLimit: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: restore-mytest-cluster-pxc Optional: true ssl-internal: Type: Secret (a volume populated by a Secret) SecretName: restore-mytest-cluster-ssl-internal Optional: true ssl: Type: Secret (a volume populated by a Secret) SecretName: restore-mytest-cluster-ssl Optional: false auto-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: auto-restore-mytest-cluster-pxc Optional: true vault-keyring-secret: Type: Secret (a volume populated by a Secret) SecretName: restore-mytest-cluster-vault Optional: true mysql-users-secret-file: Type: Secret (a volume populated by a Secret) SecretName: internal-restore-mytest-cluster Optional: false default-token-9s5n6: Type: Secret (a volume populated by a Secret) SecretName: default-token-9s5n6 Optional: false QoS Class: Burstable Node-Selectors: Tolerations: [node.kubernetes.io/not-ready:NoExecute](http://node.kubernetes.io/not-ready:NoExecute) op=Exists for 300s [node.kubernetes.io/unreachable:NoExecute](http://node.kubernetes.io/unreachable:NoExecute) op=Exists for 300s Events: Type Reason Age From Message * * * Normal Scheduled 16m default-scheduler Successfully assigned pxc/restore-mytest-cluster-pxc-0 to 10.47.244.15 Warning FailedAttachVolume 16m attachdetach-controller Multi-Attach error for volume “pvc-d761e245-3cca-4d15-bfc3-3846556fae35” Volume is already exclusively attached to one node and can’t be attached to another Normal SuccessfulAttachVolume 16m attachdetach-controller AttachVolume.Attach succeeded for volume “pvc-d761e245-3cca-4d15-bfc3-3846556fae35” Normal Pulling 16m kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.11.0” Normal Pulled 16m kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.11.0” in 432.030729ms Normal Created 16m kubelet Created container pxc-init Normal Started 16m kubelet Started container pxc-init Normal Pulled 16m kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.11.0-logcollector” in 379.571455ms Normal Pulling 16m kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.11.0-logcollector” Normal Created 16m kubelet Created container logs Normal Started 16m kubelet Started container logs Normal Pulling 16m kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.11.0-logcollector” Normal Pulled 16m kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.11.0-logcollector” in 382.095939ms Normal Created 16m kubelet Created container logrotate Normal Started 16m kubelet Started container logrotate Normal Pulling 16m kubelet Pulling image “percona/percona-xtradb-cluster:8.0.27-18.1” Normal Pulled 16m kubelet Successfully pulled image “percona/percona-xtradb-cluster:8.0.27-18.1” in 371.509446ms Normal Created 16m kubelet Created container pxc Normal Started 16m kubelet Started container pxc **Warning Unhealthy 14m (x2 over 14m) kubelet Liveness probe failed: ERROR 1045 (28000): Access denied for user ‘monitor’@‘restore-mytest-cluster-pxc-0.restore-mytest-cluster-pxc.pxc.svc.’ (using password: YES)** - [[-n ‘’]] - exit 1 ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [February 15, 2023, 1:08pm UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/15 "2023-02-15T13:08:00Z")
@Ravi_Kumar_Pokala According to our documentation, you need to use the same secrets (user passwords) as in the original cluster. Do you have it? > When restoring to a new Kubernetes-based environment, make sure it has a Secrets object with the same user passwords as in the original cluster. More details about secrets can be found in System Users. ---
**Author:** ![Ravi\_Kumar\_Pokala](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ravi_kumar_pokala/32/8780_2.png) [@Ravi\_Kumar\_Pokala](https://forums.percona.com/u/Ravi_Kumar_Pokala)\ **Post date:** [February 16, 2023, 1:12am UTC](https://forums.percona.com/t/kubernetes-percona-xtradb-cluster-operator-fails-to-initialize-readiness-probe-failed/19680/16 "2023-02-16T01:12:37Z")
@Slava_Sarzhan: Thank you. It worked after using the same secrets.