Can you help me setup pxc cluster on k8s

MySQL & MariaDB Percona Operator for MySQL
1

Hello, I have 3 nodes with 1 master and 2 worker nodes on k8s.
I setup by this tutorial Install Percona XtraDB Cluster using Helm
with same namespace for pxc-db & operator.
But I got error. some time:
pod/my-db-pxc-db-haproxy-0 1/2 CrashLoopBackOff 165 7h54m
pod/my-db-pxc-db-haproxy-1 1/2 Running 161 7h47m
pod/my-db-pxc-db-haproxy-2 0/2 Pending 0 7h43m
and
pod/my-db-pxc-db-pxc-0 3/3 Running 0 7h54m
pod/my-db-pxc-db-pxc-1 0/3 Pending 0 7h51m
can you help me?

error db
error db1201×467 80.3 KB

1 Like
2

Hi @thanh,

Could you please share the haproxy logs from crashed pod?

kubectl logs my-db-pxc-db-haproxy-0
1 Like
3

Here my log:
kubectl -n pxc logs my-db-pxc-db-haproxy-0
error: a container name must be specified for pod my-db-pxc-db-haproxy-0, choose one of: [haproxy pxc-monit]

1 Like
4

Please share the output of kubectl -n pxc logs my-db-pxc-db-haproxy-0 haproxy

1 Like
5

Here my logs:

  • ‘[’ haproxy = haproxy ‘]’
  • ‘[’ ‘!’ -f /etc/haproxy/pxc/haproxy.cfg ‘]’
  • custom_conf=/etc/haproxy-custom/haproxy-global.cfg
  • ‘[’ -f /etc/haproxy-custom/haproxy-global.cfg ‘]’
  • haproxy_opt='-W -db ’
  • ‘[’ -f /etc/haproxy-custom/haproxy-global.cfg -a -z ‘’ ‘]’
  • haproxy_opt+='-f /etc/haproxy/haproxy-global.cfg ’
  • haproxy_opt+='-f /etc/haproxy/pxc/haproxy.cfg -p /etc/haproxy/pxc/haproxy.pid -S /etc/haproxy/pxc/haproxy-main.sock ’
  • exec haproxy -W -db -f /etc/haproxy/haproxy-global.cfg -f /etc/haproxy/pxc/haproxy.cfg -p /etc/haproxy/pxc/haproxy.pid -S /etc/haproxy/pxc/haproxy-main.sock
    [NOTICE] 210/071749 (1) : New worker #1 (9) forked
1 Like
6

and mysql-pxc-db is still pending

pod/mysql-pxc-db-pxc-0 0/3 Pending

1 Like
7

@thanh Do you know why it is Pending?
You need to get more information about it:
kubectl describe pods my-db-pxc-db-pxc-1

Also I can see your haproxy pods can’t connect to ‘pxc-0’. You need to check logs from ‘pxc-0’ pod using the following commands:

kubectl logs my-db-pxc-db-pxc-0 -c pxc
kubectl logs my-db-pxc-db-pxc-0 -c logs

And all logs from haproxy pods:

kubectl logs my-db-pxc-db-haproxy-0 -c haproxy
kubectl logs my-db-pxc-db-haproxy-0 -c pxc-monit

Only after that you will have the whole picture why the cluster can’t start.

1 Like
8

my error:

Volumes:
datadir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: datadir-cluster1-pxc-0
ReadOnly: false

Warning FailedScheduling 61s (x5 over 3m40s) default-scheduler 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.

persistentvolumeclaim/datadir-cluster1-pxc-0 Pending 6m37s

how to fix this?

1 Like
9

error:

errrr
errrr1068×81 16.3 KB

des-cluster1-pxc-0.txt (8.3 KB)

1 Like
10

@Slava_Sarzhan Same problem using official page instructions → Install Percona XtraDB Cluster on Kubernetes

Problem is related with waiting pending from PVC , there is no PV error;

Events:
Type Reason Age From Message

Normal FailedBinding 34s (x102 over 25m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set

2 Likes
11

@iso
do you have a storage class?
Looks like some issue with CSI driver.

1 Like
12

@thanh @iso

If your storage does not support dynamic provisioning of PVs you can use local storage Local Storage support for the Percona Operator for MySQL feature or you can deploy OpenEBS for these needs. More detailed information you can find on our blogpost: Deploying Percona Kubernetes Operators with OpenEBS Local Storage - Percona Database Performance Blog

1 Like