Percona-xtradb-cluster-57 fails start upg 5.7.18-29.20 > 5.7.19-29.22. All nodes

After upgrading everything stop and can’t roll back.
/my.cnf

The Percona XtraDB Cluster 5.7 configuration file.

* IMPORTANT: Additional settings that can override those from this file!

The files must end with ‘.cnf’, otherwise they’ll be ignored.

Please make any edits and changes to the appropriate sectional files

included below.

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/percona-xtradb-cluster.conf.d/


/etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf

Template my.cnf for PXC

Edit to your requirements.

[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-bin
log_slave_updates
expire_logs_days=7

Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0


/etc/mysql/percona-xtradb-cluster.conf.d/wsrep.cnf

[mysqld]

Path to Galera library

wsrep_provider=/usr/lib/galera3/libgalera_smm.so

Cluster connection URL contains IPs of nodes

#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.1.45,192.168.1.46,192.168.1.47

In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

Slave thread to use

wsrep_slave_threads=8

wsrep_log_conflicts

This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

Node IP address

wsrep_node_address=192.168.1.45

Cluster name

wsrep_cluster_name=imglx-db-cluster

#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=prod45

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

pxc_strict_mode=ENFORCING

SST method

wsrep_sst_method=xtrabackup-v2

#Authentication for SST method
#wsrep_sst_auth=“sstuser:password”

#wsrep_replicate_myisam = ON

~# service mysql start
Job for mysql.service failed. See ‘systemctl status mysql.service’ and ‘journalctl -xn’ for details.

~# systemctl status mysql.service

  • mysql.service - Start and stop the mysql (Percona XtraDB Cluster) daemon
    Loaded: loaded (/lib/systemd/system/mysql.service; enabled)
    Active: activating (start) since Wed 2017-09-27 22:46:28 WEST; 40s ago
    Process: 4599 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
    Control: 4686 (mysql)
    CGroup: /system.slice/mysql.service
    |-4686 /bin/bash /etc/init.d/mysql start
    |-4722 /bin/sh /usr/bin/mysqld_safe
    |-5106 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --wsrep-provider=/usr/lib/ga…
    `-5273 sleep 1

Sep 27 22:46:28 PROD45 systemd[1]: Stopping Start and stop the mysql (Percona XtraDB Cluster) daemon…
Sep 27 22:46:28 PROD45 systemd[1]: Starting Start and stop the mysql (Percona XtraDB Cluster) daemon…
Sep 27 22:46:31 PROD45 mysql[4681]: WSREP: Recovered position dcb3260d-5d9a-11e7-b715-cf577114ed32:13578

Help!

grastate.dat

Node: 0

GALERA saved state

version: 2.1
uuid: dcb3260d-5d9a-11e7-b715-cf577114ed32
seqno: 13578
safe_to_bootstrap: 0

Node: 1

GALERA saved state

version: 2.1
uuid: 00000000-0000-0000-0000-000000000000
seqno: -1
safe_to_bootstrap: 0

Node:2

GALERA saved state

version: 2.1
uuid: 00000000-0000-0000-0000-000000000000
seqno: -1
safe_to_bootstrap: 0

Tried everything but still won’t start.
(Scenario 6)

https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/
It was running perfectly until upgrade.