Unable to start XtraDB cluster after host failure

Unable to start XtraDB cluster after host failure.

Hi folks,

After some power issue in DC all hosts with cluster guests were down.
Now unable to start mysqld service on each node. Here is an output from the syslog

Dec 22 14:22:16 percona-1.cloud.dimanetz.net systemd[1]: Starting Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap…
– Subject: Unit mysql@bootstrap.service has begun start-up
– Defined-By: systemd
– Support: [url]http://lists.freedesktop.org/mailman/listinfo/systemd-devel[/url]

– Unit mysql@bootstrap.service has begun starting up.
Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysql-systemd[12421]: State transfer in progress, setting sleep higher
Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysqld_safe[12420]: 2017-12-22T13:22:16.359286Z mysqld_safe Logging to ‘/var/log/mysqld.log’.
Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysqld_safe[12420]: 2017-12-22T13:22:16.360737Z mysqld_safe Logging to ‘/var/log/mysqld.log’.
Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysqld_safe[12420]: 2017-12-22T13:22:16.376960Z mysqld_safe Starting mysqld daemon with databases from /data/mysqldata
Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysqld_safe[12420]: 2017-12-22T13:22:16.389082Z mysqld_safe WSREP: Running position recovery with --log_error=‘/data/mysqldata/wsrep_recovery.xB4hbS’ –
Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12421]: /usr/bin/mysql-systemd: line 140: kill: (12420) - No such process
Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12421]: ERROR! mysqld_safe with PID 12420 has already exited: FAILURE
Dec 22 14:22:26 percona-1.cloud.dimanetz.net systemd[1]: mysql@bootstrap.service: control process exited, code=exited status=1
Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12922]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12922]: ERROR! mysql already dead
Dec 22 14:22:26 percona-1.cloud.dimanetz.net systemd[1]: mysql@bootstrap.service: control process exited, code=exited status=2
Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12953]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12953]: WARNING: mysql may be already dead
Dec 22 14:22:26 percona-1.cloud.dimanetz.net systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap.
– Subject: Unit mysql@bootstrap.service has failed
– Defined-By: systemd
– Support: [url]http://lists.freedesktop.org/mailman/listinfo/systemd-devel[/url]

– Unit mysql@bootstrap.service has failed.

– The result is failed.

Tried to bootstrap cluster - got same errors. Any new lines in /var/log/mysqld.log.

ll /var/run/mysqld/
total 0

Same at all 3 nodes.

GALERA saved state

version: 2.1
uuid: 6a3ccb37-c7ff-11e7-9ce1-16e4a152fdf7
seqno: -1
safe_to_bootstrap: 0