Infinite database initialization on creation

Percona Everest
1

Hi, I’m experimenting with Percona Everest and ran into a problem.

I have created several databases for my experiments.

When I started to create the next database, it didn’t get created, for several hours now the database is in Initialization status.

Please advise what I need to do and how to identify the problem.

Screenshot 2024-01-19 at 5.43.14 PM
Screenshot 2024-01-19 at 5.43.14 PM2044×1026 121 KB

2

I checked the pods running

kubectl -n percona-everest get pods

It turned out that I had run out of cluster resources, because the pxc pods were also running haproxy pods, for which I had not calculated the resources.

3

I am having the same problem with latest release of Percona Everest 0.10.1.

The cluster is set up using minikube with 3 nodes, 6 cpu cores and 16GB RAM.

I am able to create mongodb database.

Cluster resources use remains moderate:

$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ha3 604m 5% 1889Mi 5%
ha3-m02 665m 5% 1772Mi 5%
ha3-m03 614m 5% 2485Mi 7%

When mysql database is created it remains in initializing with portainer monitoring showing almost infinite memory reservation.

When mysql database is not in creation / initialization process:

Memory reservation
19989 / 99801 MB - 20%

However when mysql database instance is created:

Memory reservation reaches astronomical value:

Memory reservation
5186173034815 / 99801 MB - 5196514098%

It appears haproxy pod keeps restarting and is unable to complete what it is supposed to do to make the database cluster functional.

Any help would be appreciated.

4

Do you happen to have a MacBook with an ARM (M1,2,3) processor?

5

No. I am working on a Dell XPS 17 with x86-64, 6 Cores, 12 Threads, 32 GB RAM.

Kubernetes cluster created with minikube:

minikube start -p ha3 --cpus=6 --memory=16384 --nodes=3 --driver=docker --ha=true

6

I’m not an expert, but it seems Minikube with 3 nodes and 6 CPUs requires 3*6=18 CPUs in the system.

I think you need to reduce the number of Minikube nodes to 1.

You can also try Kind or k3s
Our developers use this too

7

In my case, the problem was the Mac, as the operators don’t support ARM architecture yet.

8

Ok.

It sounds like a reasonable suggestion.

For information the output of

kubectl describe pod -n dev mysql-eay-pxc-1

on a stuck up pod is:

                         node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:
Type Reason Age From Message

Warning FailedScheduling 5m52s default-scheduler running Bind plugin “DefaultBinder”: Operation cannot be fulfilled on pods/binding “mysql-eay-pxc-1”: pod mysql-eay-pxc-1 is already assigned to node “ha3-m03”
Warning FailedScheduling 5m52s default-scheduler running Bind plugin “DefaultBinder”: Operation cannot be fulfilled on pods/binding “mysql-eay-pxc-1”: pod mysql-eay-pxc-1 is already assigned to node “ha3-m03”
Normal Scheduled 5m53s default-scheduler Successfully assigned dev/mysql-eay-pxc-1 to ha3-m03
Normal Pulled 5m50s kubelet Successfully pulled image “docker.io/percona/percona-xtradb-cluster-operator:1.13.0” in 2.666s (2.666s including waiting). Image size: 140547877 bytes.
Normal Pulled 5m46s kubelet Successfully pulled image “docker.io/percona/percona-xtradb-cluster-operator:1.13.0” in 2.681s (2.681s including waiting). Image size: 140547877 bytes.
Normal Pulled 5m29s kubelet Successfully pulled image “docker.io/percona/percona-xtradb-cluster-operator:1.13.0” in 2.657s (2.657s including waiting). Image size: 140547877 bytes.
Normal Pulled 4m58s kubelet Successfully pulled image “docker.io/percona/percona-xtradb-cluster-operator:1.13.0” in 2.631s (2.631s including waiting). Image size: 140547877 bytes.
Normal Created 4m57s (x4 over 5m50s) kubelet Created container pxc-init
Normal Started 4m57s (x4 over 5m49s) kubelet Started container pxc-init
Normal Pulling 4m16s (x5 over 5m52s) kubelet Pulling image “docker.io/percona/percona-xtradb-cluster-operator:1.13.0”
Normal Pulled 4m13s kubelet Successfully pulled image “docker.io/percona/percona-xtradb-cluster-operator:1.13.0” in 2.946s (2.946s including waiting). Image size: 140547877 bytes.
Warning BackOff 49s (x24 over 5m46s) kubelet Back-off restarting failed container pxc-init in pod mysql-eay-pxc-1_dev(045c88ce-1bc7-46e9-b98c-40cb5f7924bf)