Hello, I have 3 nodes with 1 master and 2 worker nodes on k8s.
I setup by this tutorial Install Percona XtraDB Cluster using Helm
with same namespace for pxc-db & operator.
But I got error. some time:
pod/my-db-pxc-db-haproxy-0 1/2 CrashLoopBackOff 165 7h54m
pod/my-db-pxc-db-haproxy-1 1/2 Running 161 7h47m
pod/my-db-pxc-db-haproxy-2 0/2 Pending 0 7h43m
and
pod/my-db-pxc-db-pxc-0 3/3 Running 0 7h54m
pod/my-db-pxc-db-pxc-1 0/3 Pending 0 7h51m
can you help me?
Hi @thanh,
Could you please share the haproxy logs from crashed pod?
kubectl logs my-db-pxc-db-haproxy-0
Here my log:
kubectl -n pxc logs my-db-pxc-db-haproxy-0
error: a container name must be specified for pod my-db-pxc-db-haproxy-0, choose one of: [haproxy pxc-monit]
Please share the output of kubectl -n pxc logs my-db-pxc-db-haproxy-0 haproxy
Here my logs:
- β[β haproxy = haproxy β]β
- β[β β!β -f /etc/haproxy/pxc/haproxy.cfg β]β
- custom_conf=/etc/haproxy-custom/haproxy-global.cfg
- β[β -f /etc/haproxy-custom/haproxy-global.cfg β]β
- haproxy_opt=β-W -db β
- β[β -f /etc/haproxy-custom/haproxy-global.cfg -a -z ββ β]β
- haproxy_opt+=β-f /etc/haproxy/haproxy-global.cfg β
- haproxy_opt+=β-f /etc/haproxy/pxc/haproxy.cfg -p /etc/haproxy/pxc/haproxy.pid -S /etc/haproxy/pxc/haproxy-main.sock β
- exec haproxy -W -db -f /etc/haproxy/haproxy-global.cfg -f /etc/haproxy/pxc/haproxy.cfg -p /etc/haproxy/pxc/haproxy.pid -S /etc/haproxy/pxc/haproxy-main.sock
[NOTICE] 210/071749 (1) : New worker #1 (9) forked
and mysql-pxc-db is still pending
pod/mysql-pxc-db-pxc-0 0/3 Pending
@thanh Do you know why it is Pending?
You need to get more information about it:
kubectl describe pods my-db-pxc-db-pxc-1
Also I can see your haproxy pods canβt connect to βpxc-0β. You need to check logs from βpxc-0β pod using the following commands:
kubectl logs my-db-pxc-db-pxc-0 -c pxc
kubectl logs my-db-pxc-db-pxc-0 -c logs
And all logs from haproxy pods:
kubectl logs my-db-pxc-db-haproxy-0 -c haproxy
kubectl logs my-db-pxc-db-haproxy-0 -c pxc-monit
Only after that you will have the whole picture why the cluster canβt start.
my error:
Volumes:
datadir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: datadir-cluster1-pxc-0
ReadOnly: false
Warning FailedScheduling 61s (x5 over 3m40s) default-scheduler 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
persistentvolumeclaim/datadir-cluster1-pxc-0 Pending 6m37s
how to fix this?
@SlavaSarzhan Same problem using official page instructions β Install Percona XtraDB Cluster on Kubernetes
Problem is related with waiting pending from PVC , there is no PV error;
Events:
Type Reason Age From Message
Normal FailedBinding 34s (x102 over 25m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
If your storage does not support dynamic provisioning of PVs you can use local storage Local Storage support for the Percona Distribution for MySQL Operator feature or you can deploy OpenEBS for these needs. More detailed information you can find on our blogpost: Deploying Percona Kubernetes Operators with OpenEBS Local Storage - Percona Database Performance Blog