3rd node cannot be started after restart (power off and power on).

Hello,

I have a cluster from 3 nodes. After power off of 3rd node and power on it can not be started.

Here is a status of the node:

systemctl status mysql

● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; vendor preset: disabled)
Active: activating (start-post) since Fri 2016-12-09 22:39:19 EET; 1h 48min left
Process: 915 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Main PID: 994 (mysqld_safe); : 995 (mysql-systemd)
CGroup: /system.slice/mysql.service
├─ 994 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
├─1669 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsr…
└─control
├─ 995 /bin/bash -ue /usr/bin/mysql-systemd start-post 994
└─5471 sleep 1

Dec 09 22:39:19 GlusterDC1_3 systemd[1]: Starting Percona XtraDB Cluster…
Dec 09 22:39:21 GlusterDC1_3 mysqld_safe[994]: 2016-12-09T20:39:21.938827Z mysqld_safe Logging to ‘/var/log/mysqld.log’.
Dec 09 22:39:22 GlusterDC1_3 mysqld_safe[994]: 2016-12-09T20:39:22.142700Z mysqld_safe Starting mysqld daemon with databases…/mysql
Dec 09 22:39:22 GlusterDC1_3 mysqld_safe[994]: 2016-12-09T20:39:22.193207Z mysqld_safe Skipping wsrep-recover for 43de3d74-b…5 pair
Dec 09 22:39:22 GlusterDC1_3 mysqld_safe[994]: 2016-12-09T20:39:22.213707Z mysqld_safe Assigning 43de3d74-bca8-11e6-a178-57b…sition
Hint: Some lines were ellipsized, use -l to show in full.

Here is the status of the node after 2 hours:

systemctl status mysql

● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; vendor preset: disabled)
Active: activating (start-post) since Fri 2016-12-09 21:09:27 EET; 6min ago
Process: 6370 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
Process: 6154 ExecStop=/usr/bin/mysql-systemd stop (code=exited, status=2)
Process: 6400 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Main PID: 6441 (mysqld_safe); : 6442 (mysql-systemd)
CGroup: /system.slice/mysql.service
├─6441 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
├─6841 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsr…
└─control
├─6442 /bin/bash -ue /usr/bin/mysql-systemd start-post 6441
└─8167 sleep 1

Dec 09 21:09:27 GlusterDC1_3 systemd[1]: mysql.service holdoff time over, scheduling restart.
Dec 09 21:09:27 GlusterDC1_3 systemd[1]: Starting Percona XtraDB Cluster…
Dec 09 21:09:28 GlusterDC1_3 mysqld_safe[6441]: 2016-12-09T19:09:28.030105Z mysqld_safe Logging to ‘/var/log/mysqld.log’.
Dec 09 21:09:28 GlusterDC1_3 mysqld_safe[6441]: 2016-12-09T19:09:28.050046Z mysqld_safe Starting mysqld daemon with database…/mysql
Dec 09 21:09:28 GlusterDC1_3 mysqld_safe[6441]: 2016-12-09T19:09:28.058749Z mysqld_safe Skipping wsrep-recover for 43de3d74-…5 pair
Dec 09 21:09:28 GlusterDC1_3 mysqld_safe[6441]: 2016-12-09T19:09:28.060037Z mysqld_safe Assigning 43de3d74-bca8-11e6-a178-57…sition
Hint: Some lines were ellipsized, use -l to show in full.

I can not connect to the node. Here is what it says:

mysql -u root -p

Enter password:
ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)

Other 2 nodes works fine.

Does somebody know what is the problem?

Sincerely,
Alexandr

As said in the newer thread, the MySQL error log would provide more information on the issue - do you have it?