MySQL Cluster using Percona Mysql Operator does not start

MySQL & MariaDB Percona Operator for MySQL
1

Hello, we increased the size of our PV volume. We have 3 nodes. We followed the procedure: Scale Percona Distribution for MySQL on Kubernetes

When we try to start the cluster, it does not start. The problem is that the first node is a slave, and it’s trying to reach the third node that it hasn’t created yet.

If the first node cannot start, the pod switches to crashloopbackoff and stops creating the cluster.

The logs from cluster-br-prd-mysql-0:

+ exec mysqld
/etc/mysql/config/my-secret.cnf
2022-02-15T03:03:03.389215Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.25-15) starting as process 1
2022-02-15T03:03:03.396924Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-02-15T03:03:03.541457Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-02-15T03:03:03.650357Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql/mysqlx.sock
2022-02-15T03:03:03.721919Z 0 [Warning] [MY-010068] [Server] CA certificate /etc/mysql/mysql-tls-secret/ca.crt is self signed.
2022-02-15T03:03:03.722325Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-02-15T03:03:03.732541Z 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_admin. See below for the description of exact issue.
2022-02-15T03:03:03.732908Z 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2022-02-15T03:03:03.733226Z 0 [System] [MY-013603] [Server] No TLS configuration was given for channel mysql_admin; re-using TLS configuration of channel mysql_main.
2022-02-15T03:03:03.765248Z 0 [Warning] [MY-010604] [Repl] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=cluster-br-prd-mysql-0-relay-bin' to avoid this problem.
2022-02-15T03:03:03.775811Z 5 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2022-02-15T03:03:03.785469Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.25-15' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 15, Revision a558ec2.
2022-02-15T03:03:03.785714Z 0 [System] [MY-013292] [Server] Admin interface ready for connections, address: '10.0.1.185' port: 33062
2022-02-15T03:03:03.795268Z 5 [ERROR] [MY-010584] [Repl] Slave I/O for channel '': error connecting to master 'replication@cluster-br-prd-mysql-2.cluster-br-prd-mysql.percona:3306' - retry-time: 60 retries: 1 message: Unknown MySQL server host 'cluster-br-prd-mysql-2.cluster-br-prd-mysql.percona' (2), Error_code: MY-002005
2022-02-15T03:03:19.968117Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.25-15).
2022-02-15T03:03:21.168350Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.25-15) Percona Server (GPL), Release 15, Revision a558ec2.

Logs from operator:

2022-02-15T03:03:57.432Z        INFO    controller.perconaservermysql.reconcileUsers    MySQL is not ready      {"reconciler group": "ps.percona.com", "reconciler kind": "PerconaServerMySQL", "name": "cluster-br-prd", "namespace": "percona"}
2022-02-15T03:03:57.460Z        DEBUG   controller.perconaservermysql.reconcileReplicationPrimaryPod    got 1 pods      {"reconciler group": "ps.percona.com", "reconciler kind": "PerconaServerMySQL", "name": "cluster-br-prd", "namespace": "percona"}
2022-02-15T03:03:57.484Z        ERROR   controller.perconaservermysql   Reconciler error        {"reconciler group": "ps.percona.com", "reconciler kind": "PerconaServerMySQL", "name": "cluster-br-prd", "namespace": "percona", "error": "reconcile: replication: reconcile primary pod: get cluster primary: request failed with 500 Internal Server Error", "errorVerbose": "request failed with 500 Internal Server Error\ngithub.com/percona/percona-server-mysql-
..
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/src/github.com/percona/percona-server-mysql-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

Can you help me start the cluster?

I really appreciate your help. Thanks!

1 Like
2

Hi @Sebastian_Migliorisi,

Most probably you hit a bug, the new MySQL Operator is not considered stable and production ready at the moment. Could you please share the exact steps you performed to resize the PV?

1 Like
3

Hi @Ege_Gunes, thanks for your reply.

Yes, first I tried to restart all the nodes:


kubectl rollout restart statefulset -n percona cluster-br-prd-mysql

Then I tried to increase the PV size:


kubectl get -n percona PerconaServerMySQL -o yaml > CR_backup.yaml

kubectl delete -f CR_backup.yaml

kubectl edit pvc -n percona datadir-cluster-br-prd-mysql-0

kubectl edit pvc -n percona datadir-cluster-br-prd-mysql-1

kubectl edit pvc -n percona datadir-cluster-br-prd-mysql-2

Then I edited CR_backup.yaml to change the new size and applied the configuration.


kubectl apply -f CR_backup.yaml

This last step didn’t work, and I am unable to start the cluster anymore.

Do you recommend using XtraDB Cluster Operator instead?

Best Regards,

1 Like