# 3 node cluster, 1 updated stopped working **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [November 14, 2017, 3:13pm UTC](https://forums.percona.com/t/3-node-cluster-1-updated-stopped-working/6001 "2017-11-14T15:13:57Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![pedropilla](https://avatars.discourse-cdn.com/v4/letter/p/cdc98d/32.png) [@pedropilla](https://forums.percona.com/u/pedropilla)\ **Post date:** [November 14, 2017, 3:13pm UTC](https://forums.percona.com/t/3-node-cluster-1-updated-stopped-working/6001/1 "2017-11-14T15:13:57Z")
Need some help here, I was using 5.7.19-29.22.1.el7 and decided to update trying to resolve some issues that happens aleatory, like node shutdown. I picked node1 and updated through yum update, after this this node cant join the cluster anymore, below I’ll put some info: SElinux is disabled on all nodes, firewalld is with the right zone for the eth, nothing has changed, only yum update mess everything. [root@noc-db-01 pedro]# cat /etc/my.cnf [client] socket=/var/lib/mysql1/mysql.sock [mysqld] server-id=1 datadir=/var/lib/mysql1 user=mysql socket=/var/lib/mysql1/mysql.sock log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid log-bin log\_slave\_updates expire\_logs\_days=7 symbolic-links=0 wsrep\_provider=/usr/lib64/galera3/libgalera\_smm.so wsrep\_cluster\_address=gcomm://192.168.100.1,192.168.100.2,192.168.100.3 binlog\_format=ROW default\_storage\_engine=InnoDB wsrep\_slave\_threads= 8 wsrep\_log\_conflicts wsrep\_node\_address=192.168.100.1 wsrep\_cluster\_name=noc-db wsrep\_node\_name=noc-db-01 pxc\_strict\_mode=DISABLED wsrep\_sst\_method=xtrabackup-v2 wsrep\_sst\_auth=“sstuser:s3cret” log\_timestamps=SYSTEM ignore-db-dir=lost+found thread\_cache\_size = 50 max\_connections = 512 query\_cache\_limit = 0 query\_cache\_size = 0 query\_cache\_type = 0 expire\_logs\_days = 10 max\_binlog\_size = 400M max\_allowed\_packet = 128M innodb\_buffer\_pool\_size = 24G innodb\_buffer\_pool\_instances = 24 innodb\_flush\_log\_at\_trx\_commit = 0 innodb\_flush\_method = O\_DIRECT innodb\_locks\_unsafe\_for\_binlog = 1 innodb\_autoinc\_lock\_mode = 2 innodb\_log\_file\_size = 400M open\_files\_limit = 65535 table\_open\_cache = 16384 table-definition-cache = 4096 join\_buffer\_size = 512K tmp-table-size = 32M max-heap-table-size = 32M [root@noc-db-01 pedro]# systemctl status mysql.service ● mysql.service - Percona XtraDB Cluster Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2017-11-14 18:34:42 -02; 1min 12s ago Process: 12890 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS) Process: 12860 ExecStop=/usr/bin/mysql-systemd stop (code=exited, status=2) Process: 12120 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=1/FAILURE) Process: 12119 ExecStart=/usr/bin/mysqld\_safe --basedir=/usr (code=exited, status=0/SUCCESS) Process: 12078 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS) Main PID: 12119 (code=exited, status=0/SUCCESS) Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: mysql.service: control process exited, code=exited status=1 Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12120]: ERROR! mysqld\_safe with PID 12119 has already exited: FAILURE Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12860]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12860]: ERROR! mysql already dead Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: mysql.service: control process exited, code=exited status=2 Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12890]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12890]: WARNING: mysql may be already dead Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: Failed to start Percona XtraDB Cluster. Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: Unit mysql.service entered failed state. Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: mysql.service failed. ## Nov 14 18:34:32 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: Starting Percona XtraDB Cluster… – Subject: Unit mysql.service has begun start-up – Defined-By: systemd – Support: [http://lists.freedesktop.org/mailman/listinfo/systemd-devel](http://lists.freedesktop.org/mailman/listinfo/systemd-devel) ## – Unit mysql.service has begun starting up. Nov 14 18:34:32 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12120]: State transfer in progress, setting sleep higher Nov 14 18:34:32 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysqld\_safe[12119]: mysqld\_safe Adding ‘/usr/lib64/libjemalloc.so.1’ to LD\_PRELOAD for mysqld Nov 14 18:34:32 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysqld\_safe[12119]: 2017-11-14T20:34:32.919982Z mysqld\_safe Logging to ‘/var/log/mysqld.log’. Nov 14 18:34:32 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysqld\_safe[12119]: 2017-11-14T20:34:32.923291Z mysqld\_safe Logging to ‘/var/log/mysqld.log’. Nov 14 18:34:32 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysqld\_safe[12119]: 2017-11-14T20:34:32.967654Z mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql1 Nov 14 18:34:32 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysqld\_safe[12119]: 2017-11-14T20:34:32.993454Z mysqld\_safe WSREP: Running position recovery with --log\_error=‘/var/lib/mysql1/wsrep\_recovery.H90Nca’ --pid-file='/var/lib/m Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12120]: /usr/bin/mysql-systemd: line 140: kill: (12119) - No such process Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: mysql.service: control process exited, code=exited status=1 Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12120]: ERROR! mysqld\_safe with PID 12119 has already exited: FAILURE Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12860]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12860]: ERROR! mysql already dead Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: mysql.service: control process exited, code=exited status=2 Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12890]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) mysql-systemd[12890]: WARNING: mysql may be already dead Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: Failed to start Percona XtraDB Cluster. – Subject: Unit mysql.service has failed – Defined-By: systemd – Support: [http://lists.freedesktop.org/mailman/listinfo/systemd-devel](http://lists.freedesktop.org/mailman/listinfo/systemd-devel) ## – Unit mysql.service has failed. – The result is failed. Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: Unit mysql.service entered failed state. Nov 14 18:34:42 [noc-db-01.vetorial.net](http://noc-db-01.vetorial.net) systemd[1]: mysql.service failed. ---
**Author:** ![pedropilla](https://avatars.discourse-cdn.com/v4/letter/p/cdc98d/32.png) [@pedropilla](https://forums.percona.com/u/pedropilla)\ **Post date:** [November 16, 2017, 8:17am UTC](https://forums.percona.com/t/3-node-cluster-1-updated-stopped-working/6001/2 "2017-11-16T08:17:08Z")
As I need full cluster working I removed the 5.7.19-29.22.3.el7 and reinstalled the 5.7.18-29.20.1.el7, everything working again. ---
**Author:** ![cbegin](https://avatars.discourse-cdn.com/v4/letter/c/c6cbf5/32.png) [@cbegin](https://forums.percona.com/u/cbegin)\ **Post date:** [November 30, 2017, 10:37am UTC](https://forums.percona.com/t/3-node-cluster-1-updated-stopped-working/6001/3 "2017-11-30T10:37:07Z")
Hi! This sounds very similar to an issue we’re seeing. I posted about it in another thread: [url][https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/50141-open-table-locks-incrementing-on-one-cluster-node-requires-mysqld-cycle-to-correct[/url]](https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/50141-open-table-locks-incrementing-on-one-cluster-node-requires-mysqld-cycle-to-correct%5B/url%5D) ---
**Author:** ![cbegin](https://avatars.discourse-cdn.com/v4/letter/c/c6cbf5/32.png) [@cbegin](https://forums.percona.com/u/cbegin)\ **Post date:** [November 30, 2017, 10:41am UTC](https://forums.percona.com/t/3-node-cluster-1-updated-stopped-working/6001/4 "2017-11-30T10:41:03Z")
We’re also running Percona-XtraDB-Cluster-server-57-5.7.19-29.22.1.el7.x86\_64 and we’ll try downgrading to see if that helps, since we can recreate the problem. ---
**Author:** ![krunalbauskar](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@krunalbauskar](https://forums.percona.com/u/krunalbauskar)\ **Post date:** [December 4, 2017, 1:05am UTC](https://forums.percona.com/t/3-node-cluster-1-updated-stopped-working/6001/5 "2017-12-04T01:05:18Z")
Can you please attach log file to understand what exactly failure on the said node.