Kubernetes: percona-xtradb-cluster-operator fails to initialize - readiness probe failed

Hi,

I have followed the instruction to install the operator in Kubernetes @ Generic Kubernetes installation - Percona Operator for MySQL based on Percona XtraDB Cluster

I am able to run the operator pod, but the cluster pod fails to come up. Please let me know how to resolve it.

>kubectl get pods
NAME READY STATUS RESTARTS AGE
percona-xtradb-cluster-operator-5c95894bbf-hwv5h 1/1 Running 0 122m
test-cluster-pxc-0 2/3 Running 0 13m

>kubectl describe pod/test-cluster-pxc-0
Events:
Type Reason Age From Message


Warning FailedScheduling 7m28s (x3 over 7m39s) default-scheduler 0/20 nodes are available: 20 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 7m24s default-scheduler Successfully assigned pxc/test-cluster-pxc-0 to 10.47.244.15
Normal SuccessfulAttachVolume 7m24s attachdetach-controller AttachVolume.Attach succeeded for volume “pvc-ab3faef3-3a15-4592-8d83-774212a9c926”
Normal Pulling 7m6s kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.12.0”
Normal Pulled 7m6s kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.12.0” in 693.444389ms
Normal Created 7m6s kubelet Created container pxc-init
Normal Pulling 7m5s kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.12.0-logcollector”
Normal Started 7m5s kubelet Started container pxc-init
Normal Created 7m4s kubelet Created container logs
Normal Pulled 7m4s kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.12.0-logcollector” in 663.401732ms
Normal Created 7m3s kubelet Created container logrotate
Normal Started 7m3s kubelet Started container logs
Normal Pulling 7m3s kubelet Pulling image “percona/percona-xtradb-cluster-operator:1.12.0-logcollector”
Normal Pulled 7m3s kubelet Successfully pulled image “percona/percona-xtradb-cluster-operator:1.12.0-logcollector” in 645.501357ms
Normal Started 7m2s kubelet Started container logrotate
Normal Pulling 7m2s kubelet Pulling image “percona/percona-xtradb-cluster:8.0.29-21.1”
Normal Pulled 7m2s kubelet Successfully pulled image “percona/percona-xtradb-cluster:8.0.29-21.1” in 646.002133ms
Normal Created 7m2s kubelet Created container pxc
Normal Started 7m1s kubelet Started container pxc
Warning DNSConfigForming 5m52s (x6 over 7m7s) kubelet Search Line limits were exceeded, some search paths have been omitted, the applied search line is: pxc.svc.cluster.local svc.cluster.local cluster.local

Warning Unhealthy 5m17s (x3 over 6m17s) kubelet Readiness probe failed: ERROR 2003 (HY000): Can’t connect to MySQL server on ‘10.20.80.134:33062’ (111)

  • [[ ‘’ == \P\r\i\m\a\r\y ]]
  • exit 1
    Warning Unhealthy 112s kubelet Liveness probe failed: ERROR 2003 (HY000): Can’t connect to MySQL server on ‘10.20.80.134:33062’ (111)
  • [[ -n ‘’ ]]
  • exit 1

>kubect logs pod/test-cluster-pxc-0
Percona XtraDB Cluster: Finding peers
2023/01/19 01:06:22 Peer finder enter
2023/01/19 01:06:22 Determined Domain to be pxc.svc.cluster.local
2023/01/19 01:06:22 lookup test-cluster-pxc-unready on 10.21.0.10:53: no such host
2023/01/19 01:06:23 lookup test-cluster-pxc-unready on 10.21.0.10:53: no such host

1 Like

Hi @Ravi_Kumar_Pokala ,
As I can see you have some issues connected with DNS. We need to know more about your k8s setup. E.g. which k8s version do you use? So, please tell us more about your k8s deployment.

1 Like

@Slava_Sarzhan : We are using the vendor-managed(Platform 9) Kubernetes deployment. please find the version information below.

Server Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.5”, GitCommit:“6b1d87acf3c8253c123756b9e61dac642678305f”, GitTreeState:“clean”, BuildDate:“2021-03-18T01:02:01Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}

Cluster creations fails with below error:

>kubect get pods
NAME READY STATUS RESTARTS AGE
percona-xtradb-cluster-operator-5c95894bbf-7g52j 1/1 Running 0 5m24s
test-cluster-haproxy-0 0/2 Pending 0 2m26s

Operator logs

k logs pod/percona-xtradb-cluster-operator-5c95894bbf-7g52j

2023-02-04T01:48:40.334Z ERROR Reconciler error {“controller”: “perconaxtradbcluster-controller”, “object”: {“name”:“test-cluster”,“namespace”:“pxc”}, “namespace”: “pxc”, “name”: “test-cluster”, “reconcileID”: “75d74f89-417d-4015-b720-74714e3725ae”, “error”: “PodDisruptionBudget for test-cluster-haproxy: reconcile pdb: get object: no matches for kind "PodDisruptionBudget" in version "policy/v1"”, “errorVerbose”: “no matches for kind "PodDisruptionBudget" in version "policy/v1"\nget object\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).createOrUpdate\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1299\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).reconcilePDB\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:945\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).deploy\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:685\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:312\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nreconcile pdb\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).reconcilePDB\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:945\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).deploy\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:685\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:312\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nPodDisruptionBudget for test-cluster-haproxy\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).deploy\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:687\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:312\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594”}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234