# Can start mysql or resync SST after last update **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [January 11, 2016, 9:01am UTC](https://forums.percona.com/t/can-start-mysql-or-resync-sst-after-last-update/4659 "2016-01-11T09:01:35Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![PercUse](https://avatars.discourse-cdn.com/v4/letter/p/dec6dc/32.png) [@PercUse](https://forums.percona.com/u/PercUse)\ **Post date:** [January 11, 2016, 9:01am UTC](https://forums.percona.com/t/can-start-mysql-or-resync-sst-after-last-update/4659/1 "2016-01-11T09:01:35Z")
Hi Guys, We just updated our mysql cluster from 5.6.26-25.12-1.precise to 5.6.27-25.13-1.precise Package involved: percona-xtradb-cluster-client-5.6 (5.6.26-25.12-1.precise =\> 5.6.27-25.13-1.precise) percona-xtradb-cluster-common-5.6 (5.6.26-25.12-1.precise =\> 5.6.27-25.13-1.precise) percona-xtradb-cluster-galera-3.x (3.12.2-1.precise =\> 3.13-1.precise) percona-xtradb-cluster-server-5.6 (5.6.26-25.12-1.precise =\> 5.6.27-25.13-1.precise) Since the update, we have been able to start MySQL on 2 nodes but the third one don’t want to start or sync > [@](#): > > /etc/init.d/mysql start > > - Starting MySQL (Percona XtraDB Cluster) database server mysqld > > 160111 15:15:03 mysqld\_safe Starting mysqld daemon with databases from /srv/mysql/data > 160111 15:15:03 mysqld\_safe WSREP: Running position recovery with --log\_error=‘/srv/mysql/data/wsrep\_recovery.I1fypn’ --pid-file=‘/srv/mysql/data/server2-recover.pid’ > 2016-01-11 15:15:05 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 5251 … > 160111 15:15:48 mysqld\_safe WSREP: Recovered position 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1305553 > Log of wsrep recovery (–wsrep-recover): > 2016-01-11 15:15:05 5251 [Note] Plugin ‘FEDERATED’ is disabled. > InnoDB: Warning: innodb\_log\_block\_size has been changed from default value 512. (###EXPERIMENTAL### operation) > InnoDB: The log block size is set to 4096. > 2016-01-11 15:15:05 5251 [Note] InnoDB: Using atomics to ref count buffer pool pages > 2016-01-11 15:15:05 5251 [Note] InnoDB: The InnoDB memory heap is disabled > 2016-01-11 15:15:05 5251 [Note] InnoDB: Mutexes and rw\_locks use GCC atomic builtins > 2016-01-11 15:15:05 5251 [Note] InnoDB: Memory barrier is not used > 2016-01-11 15:15:05 5251 [Note] InnoDB: Compressed tables use zlib 1.2.3.4 > 2016-01-11 15:15:05 5251 [Note] InnoDB: Using Linux native AIO > 2016-01-11 15:15:05 5251 [Note] InnoDB: Using CPU crc32 instructions > 2016-01-11 15:15:05 5251 [Note] InnoDB: Initializing buffer pool, size = 10.0G > 2016-01-11 15:15:06 5251 [Note] InnoDB: Completed initialization of buffer pool > 2016-01-11 15:15:06 5251 [Note] InnoDB: Highest supported file format is Barracuda. > 2016-01-11 15:15:45 5251 [Note] InnoDB: 128 rollback segment(s) are active. > 2016-01-11 15:15:45 5251 [Note] InnoDB: Waiting for purge to start > 2016-01-11 15:15:45 5251 [Warning] InnoDB: Setting thread 5395 nice to -10 failed, current nice 0, errno 13 > 2016-01-11 15:15:45 5251 [Note] InnoDB: Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 5.6.27-76.0 started; log sequence number 5680797833 > 2016-01-11 15:15:45 5251 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery. > 2016-01-11 15:15:45 5251 [Warning] InnoDB: Setting thread 5396 nice to -10 failed, current nice 0, errno 13 > 2016-01-11 15:15:45 5251 [Note] RSA private key file not found: /srv/mysql/data//private\_key.pem. Some authentication plugins will not work. > 2016-01-11 15:15:45 5251 [Note] RSA public key file not found: /srv/mysql/data//public\_key.pem. Some authentication plugins will not work. > 2016-01-11 15:15:45 5251 [Note] Server hostname (bind-address): ‘192.168.10.12’; port: 3306 > 2016-01-11 15:15:45 5251 [Note] - ‘192.168.10.12’ resolves to ‘192.168.10.12’; > 2016-01-11 15:15:45 5251 [Note] Server socket created on IP: ‘192.168.10.12’. > 2016-01-11 15:15:45 5251 [Note] WSREP: Recovered position: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1305553 > 2016-01-11 15:15:45 5251 [Note] Binlog end > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘partition’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘BLACKHOLE’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘ARCHIVE’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘PERFORMANCE\_SCHEMA’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_CHANGED\_PAGES’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_DATAFILES’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESPACES’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN\_COLS’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_FIELDS’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_COLUMNS’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_INDEXES’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESTATS’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLES’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_TABLE’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_CACHE’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_FT\_CONFIG’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_FT\_BEING\_DELETED’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_FT\_DELETED’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_FT\_DEFAULT\_STOPWORD’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_METRICS’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_BUFFER\_POOL\_STATS’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE\_LRU’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX\_RESET’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_CMPMEM\_RESET’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_CMPMEM’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_CMP\_RESET’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_CMP’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_LOCK\_WAITS’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_LOCKS’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘INNODB\_TRX’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘XTRADB\_RSEG’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘XTRADB\_INTERNAL\_HASH\_TABLES’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘XTRADB\_READ\_VIEW’ > 2016-01-11 15:15:45 5251 [Note] Shutting down plugin ‘InnoDB’ > 2016-01-11 15:15:45 5251 [Note] InnoDB: FTS optimize thread exiting. > 2016-01-11 15:15:45 5251 [Note] InnoDB: Starting shutdown… > 2016-01-11 15:15:47 5251 [Note] InnoDB: Shutdown completed; log sequence number 5680801480 > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘MyISAM’ > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘MRG\_MYISAM’ > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘MEMORY’ > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘CSV’ > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘sha256\_password’ > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘mysql\_old\_password’ > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘mysql\_native\_password’ > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘wsrep’ > 2016-01-11 15:15:47 5251 [Note] Shutting down plugin ‘binlog’ > 2016-01-11 15:15:47 5251 [Note] /usr/sbin/mysqld: Shutdown complete > > 2016-01-11 15:15:49 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 5413 … > 2016-01-11 15:15:49 5413 [Note] WSREP: Read nil XID from storage engines, skipping position init > 2016-01-11 15:15:49 5413 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib/libgalera\_smm.so’ > 2016-01-11 15:15:49 5413 [Note] WSREP: wsrep\_load(): Galera 3.13(rb4bea65) by Codership Oy \ loaded successfully. > 2016-01-11 15:15:49 5413 [Note] WSREP: CRC-32C: using hardware acceleration. > 2016-01-11 15:15:49 5413 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 > 2016-01-11 15:15:49 5413 [Note] WSREP: Passing config to GCS: base\_dir = /srv/mysql/data/; base\_host = 192.168.10.12; base\_port = 4567; cert.log\_conflicts = no; debug = no; evs.auto\_evict = 0; evs.delay\_margin = PT1S; evs.delayed\_keep\_period = PT30S; evs.inactive\_check\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 4; evs.stats\_report\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 2; evs.view\_forget\_timeout = PT24H; gcache.dir = /srv/mysql/data/; gcache.keep\_pages\_count = 0; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /srv/mysql/data//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1.0; gcs.fc\_limit = 16; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignore\_quor > 2016-01-11 15:15:49 5413 [Note] WSREP: Service thread queue flushed. > 2016-01-11 15:15:49 5413 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 > 2016-01-11 15:15:49 5413 [Note] WSREP: wsrep\_sst\_grab() > 2016-01-11 15:15:49 5413 [Note] WSREP: Start replication > 2016-01-11 15:15:49 5413 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 > 2016-01-11 15:15:49 5413 [Note] WSREP: protonet asio version 0 > 2016-01-11 15:15:49 5413 [Note] WSREP: Using CRC-32C for message checksums. > 2016-01-11 15:15:49 5413 [Note] WSREP: backend: asio > 2016-01-11 15:15:49 5413 [Note] WSREP: restore pc from disk successfully > 2016-01-11 15:15:49 5413 [Note] WSREP: GMCast version 0 > 2016-01-11 15:15:49 5413 [Note] WSREP: (00000000, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 > 2016-01-11 15:15:49 5413 [Note] WSREP: (00000000, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 > 2016-01-11 15:15:49 5413 [ERROR] WSREP: failed to open gcomm backend connection: 131: invalid UUID: 00000000 (FATAL) > at gcomm/src/pc.cpp:PC():271 > 2016-01-11 15:15:49 5413 [ERROR] WSREP: gcs/src/gcs\_core.cpp:gcs\_core\_open():208: Failed to open backend connection: -131 (State not recoverable) > 2016-01-11 15:15:49 5413 [ERROR] WSREP: gcs/src/gcs.cpp:gcs\_open():1379: Failed to open channel ‘prestaserver\_cluster’ at ‘gcomm://192.168.10.11,192.168.10.12,192.168.10.13’: -131 (State not recoverable) > 2016-01-11 15:15:49 5413 [ERROR] WSREP: gcs connect failed: State not recoverable > 2016-01-11 15:15:49 5413 [ERROR] WSREP: wsrep::connect(gcomm://192.168.10.11,192.168.10.12,192.168.10.13) failed: 7 > 2016-01-11 15:15:49 5413 [ERROR] Aborting > > 2016-01-11 15:15:49 5413 [Note] WSREP: Service disconnected. > 2016-01-11 15:15:50 5413 [Note] WSREP: Some threads may fail to exit. > 2016-01-11 15:15:50 5413 [Note] Binlog end > 2016-01-11 15:15:50 5413 [Note] /usr/sbin/mysqld: Shutdown complete > > 160111 15:15:50 mysqld\_safe mysqld from pid file /srv/mysql/data/server2.pid ended > > - The server quit without updating PID file (/srv/mysql/data/server2.pid). > - The server quit without updating PID file (/srv/mysql/data/server2.pid). Tried to start mysql and forcing the donor to be the server1 but still the same issue On the failed server: -rw-rw---- 1 mysql mysql 104 Jan 11 15:29 grastate.dat -rw-rw---- 1 mysql mysql 0 Jan 11 14:20 gvwstate.dat cat grastate.dat # GALERA saved state version: 2.1 uuid: 00000000-0000-0000-0000-000000000000 seqno: -1 cert\_index: Any idea how to force the server to rsync and start? ---
**Author:** ![PercUse](https://avatars.discourse-cdn.com/v4/letter/p/dec6dc/32.png) [@PercUse](https://forums.percona.com/u/PercUse)\ **Post date:** [January 11, 2016, 10:01am UTC](https://forums.percona.com/t/can-start-mysql-or-resync-sst-after-last-update/4659/2 "2016-01-11T10:01:52Z")
Tried again to restart the server specifying the donor to be server3 Logs from the server2 (broken): > [@](#): > > 16:24:17 mysqld\_safe Starting mysqld daemon with databases from /srv/mysql/data > 160111 16:24:17 mysqld\_safe WSREP: Running position recovery with --log\_error=‘/srv/mysql/data/wsrep\_recovery.aUOCr9’ --pid-file=‘/srv/mysql/data/server2-recover.pid’ > 2016-01-11 16:24:19 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 16836 … > 160111 16:24:55 mysqld\_safe WSREP: Recovered position 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1305553 > Log of wsrep recovery (–wsrep-recover): > 2016-01-11 16:24:19 16836 [Note] Plugin ‘FEDERATED’ is disabled. > InnoDB: Warning: innodb\_log\_block\_size has been changed from default value 512. (###EXPERIMENTAL### operation) > InnoDB: The log block size is set to 4096. > 2016-01-11 16:24:19 16836 [Note] InnoDB: Using atomics to ref count buffer pool pages > 2016-01-11 16:24:19 16836 [Note] InnoDB: The InnoDB memory heap is disabled > 2016-01-11 16:24:19 16836 [Note] InnoDB: Mutexes and rw\_locks use GCC atomic builtins > 2016-01-11 16:24:19 16836 [Note] InnoDB: Memory barrier is not used > 2016-01-11 16:24:19 16836 [Note] InnoDB: Compressed tables use zlib 1.2.3.4 > 2016-01-11 16:24:19 16836 [Note] InnoDB: Using Linux native AIO > 2016-01-11 16:24:19 16836 [Note] InnoDB: Using CPU crc32 instructions > 2016-01-11 16:24:19 16836 [Note] InnoDB: Initializing buffer pool, size = 10.0G > 2016-01-11 16:24:19 16836 [Note] InnoDB: Completed initialization of buffer pool > 2016-01-11 16:24:19 16836 [Note] InnoDB: Highest supported file format is Barracuda. > 2016-01-11 16:24:52 16836 [Note] InnoDB: 128 rollback segment(s) are active. > 2016-01-11 16:24:52 16836 [Note] InnoDB: Waiting for purge to start > 2016-01-11 16:24:52 16836 [Note] InnoDB: Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 5.6.27-76.0 started; log sequence number 5680802744 > 2016-01-11 16:24:52 16836 [Warning] InnoDB: Setting thread 16865 nice to -10 failed, current nice 0, errno 13 > 2016-01-11 16:24:52 16836 [Warning] InnoDB: Setting thread 16866 nice to -10 failed, current nice 0, errno 13 > 2016-01-11 16:24:52 16836 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery. > 2016-01-11 16:24:52 16836 [Note] RSA private key file not found: /srv/mysql/data//private\_key.pem. Some authentication plugins will not work. > 2016-01-11 16:24:52 16836 [Note] RSA public key file not found: /srv/mysql/data//public\_key.pem. Some authentication plugins will not work. > 2016-01-11 16:24:52 16836 [Note] Server hostname (bind-address): ‘192.168.10.12’; port: 3306 > 2016-01-11 16:24:52 16836 [Note] - ‘192.168.10.12’ resolves to ‘192.168.10.12’; > 2016-01-11 16:24:52 16836 [Note] Server socket created on IP: ‘192.168.10.12’. > 2016-01-11 16:24:52 16836 [Note] WSREP: Recovered position: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1305553 > 2016-01-11 16:24:52 16836 [Note] Binlog end > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘partition’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘BLACKHOLE’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘ARCHIVE’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘PERFORMANCE\_SCHEMA’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_CHANGED\_PAGES’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_DATAFILES’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESPACES’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN\_COLS’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_FIELDS’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_COLUMNS’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_INDEXES’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESTATS’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLES’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_TABLE’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_CACHE’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_FT\_CONFIG’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_FT\_BEING\_DELETED’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_FT\_DELETED’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_FT\_DEFAULT\_STOPWORD’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_METRICS’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_BUFFER\_POOL\_STATS’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE\_LRU’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX\_RESET’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_CMPMEM\_RESET’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_CMPMEM’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_CMP\_RESET’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_CMP’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_LOCK\_WAITS’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_LOCKS’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘INNODB\_TRX’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘XTRADB\_RSEG’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘XTRADB\_INTERNAL\_HASH\_TABLES’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘XTRADB\_READ\_VIEW’ > 2016-01-11 16:24:52 16836 [Note] Shutting down plugin ‘InnoDB’ > 2016-01-11 16:24:52 16836 [Note] InnoDB: FTS optimize thread exiting. > 2016-01-11 16:24:52 16836 [Note] InnoDB: Starting shutdown… > 2016-01-11 16:24:55 16836 [Note] InnoDB: Shutdown completed; log sequence number 5680802754 > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘MyISAM’ > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘MRG\_MYISAM’ > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘MEMORY’ > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘CSV’ > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘sha256\_password’ > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘mysql\_old\_password’ > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘mysql\_native\_password’ > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘wsrep’ > 2016-01-11 16:24:55 16836 [Note] Shutting down plugin ‘binlog’ > 2016-01-11 16:24:55 16836 [Note] /usr/sbin/mysqld: Shutdown complete > > 2016-01-11 16:24:56 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 16880 … > 2016-01-11 16:24:56 16880 [Note] WSREP: Read nil XID from storage engines, skipping position init > 2016-01-11 16:24:56 16880 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib/libgalera\_smm.so’ > 2016-01-11 16:24:56 16880 [Note] WSREP: wsrep\_load(): Galera 3.13(rb4bea65) by Codership Oy \ loaded successfully. > 2016-01-11 16:24:56 16880 [Note] WSREP: CRC-32C: using hardware acceleration. > 2016-01-11 16:24:56 16880 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 > 2016-01-11 16:24:56 16880 [Note] WSREP: Passing config to GCS: base\_dir = /srv/mysql/data/; base\_host = 192.168.10.12; base\_port = 4567; cert.log\_conflicts = no; debug = no; evs.auto\_evict = 0; evs.delay\_margin = PT1S; evs.delayed\_keep\_period = PT30S; evs.inactive\_check\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 4; evs.stats\_report\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 2; evs.view\_forget\_timeout = PT24H; gcache.dir = /srv/mysql/data/; gcache.keep\_pages\_count = 0; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /srv/mysql/data//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1.0; gcs.fc\_limit = 16; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignore\_quor > 2016-01-11 16:24:56 16880 [Note] WSREP: Service thread queue flushed. > 2016-01-11 16:24:56 16880 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 > 2016-01-11 16:24:56 16880 [Note] WSREP: wsrep\_sst\_grab() > 2016-01-11 16:24:56 16880 [Note] WSREP: Start replication > 2016-01-11 16:24:56 16880 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 > 2016-01-11 16:24:56 16880 [Note] WSREP: protonet asio version 0 > 2016-01-11 16:24:56 16880 [Note] WSREP: Using CRC-32C for message checksums. > 2016-01-11 16:24:56 16880 [Note] WSREP: backend: asio > 2016-01-11 16:24:56 16880 [Warning] WSREP: access file(/srv/mysql/data//gvwstate.dat) failed(No such file or directory) > 2016-01-11 16:24:56 16880 [Note] WSREP: restore pc from disk failed > 2016-01-11 16:24:56 16880 [Note] WSREP: GMCast version 0 > 2016-01-11 16:24:56 16880 [Note] WSREP: (789476f2, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 > 2016-01-11 16:24:56 16880 [Note] WSREP: (789476f2, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 > 2016-01-11 16:24:56 16880 [Note] WSREP: EVS version 0 > 2016-01-11 16:24:56 16880 [Note] WSREP: gcomm: connecting to group ‘prestaserver\_cluster’, peer ‘192.168.10.11:,192.168.10.12:,192.168.10.13:’ > 2016-01-11 16:24:56 16880 [Warning] WSREP: (789476f2, ‘tcp://0.0.0.0:4567’) address ‘tcp://192.168.10.12:4567’ points to own listening address, blacklisting > 2016-01-11 16:24:56 16880 [Note] WSREP: (789476f2, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: > 2016-01-11 16:24:57 16880 [Note] WSREP: declaring 0ee12c0b at tcp://192.168.10.13:4567 stable > 2016-01-11 16:24:57 16880 [Note] WSREP: declaring dae17fee at tcp://192.168.10.11:4567 stable > 2016-01-11 16:24:57 16880 [Note] WSREP: Node 0ee12c0b state prim > 2016-01-11 16:24:57 16880 [Note] WSREP: view(view\_id(PRIM,0ee12c0b,10) memb { > 0ee12c0b,0 > 789476f2,0 > dae17fee,0 > } joined { > } left { > } partitioned { > }) > 2016-01-11 16:24:57 16880 [Note] WSREP: save pc into disk > 2016-01-11 16:24:57 16880 [Note] WSREP: gcomm: connected > 2016-01-11 16:24:57 16880 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 > 2016-01-11 16:24:57 16880 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) > 2016-01-11 16:24:57 16880 [Note] WSREP: Opened channel ‘prestaserver\_cluster’ > 2016-01-11 16:24:57 16880 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 1, memb\_num = 3 > 2016-01-11 16:24:57 16880 [Note] WSREP: Waiting for SST to complete. > 2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. > 2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: sent state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 > 2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 0 (server3in) > 2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 2 (server1in) > 2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 1 (server2in) > 2016-01-11 16:24:57 16880 [Note] WSREP: Quorum results: > version = 3, > component = PRIMARY, > conf\_id = 8, > members = 2/3 (joined/total), > act\_id = 1309360, > last\_appl. = -1, > protocols = 0/7/3 (gcs/repl/appl), > group UUID = 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524 > 2016-01-11 16:24:57 16880 [Note] WSREP: Flow-control interval: [28, 28] > 2016-01-11 16:24:57 16880 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 1309360) > 2016-01-11 16:24:57 16880 [Note] WSREP: State transfer required: > Group state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360 > Local state: 00000000-0000-0000-0000-000000000000:-1 > 2016-01-11 16:24:57 16880 [Note] WSREP: New cluster view: global state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360, view# 9: Primary, number of nodes: 3, my index: 1, protocol version 3 > 2016-01-11 16:24:57 16880 [Warning] WSREP: Gap in state sequence. Need state transfer. > 2016-01-11 16:24:57 16880 [Note] WSREP: Running: 'wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.10.12’ --datadir ‘/srv/mysql/data/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16880’ ‘’ ’ > WSREP\_SST: [INFO] Streaming with xbstream (20160111 16:24:57.947) > WSREP\_SST: [INFO] Using socat as streamer (20160111 16:24:57.948) > WSREP\_SST: [INFO] Stale sst\_in\_progress file: /srv/mysql/data//sst\_in\_progress (20160111 16:24:57.952) > WSREP\_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160111 16:24:58.061) > 2016-01-11 16:24:58 16880 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.10.12:4444/xtrabackup\_sst//1 > 2016-01-11 16:24:58 16880 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. > 2016-01-11 16:24:58 16880 [Note] WSREP: REPL Protocols: 7 (3, 2) > 2016-01-11 16:24:58 16880 [Note] WSREP: Service thread queue flushed. > 2016-01-11 16:24:58 16880 [Note] WSREP: Assign initial position for certification: 1309360, protocol version: 3 > 2016-01-11 16:24:58 16880 [Note] WSREP: Service thread queue flushed. > 2016-01-11 16:24:58 16880 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (51c9a3db-aeaa-11e3-a5e6-bffb1faf3524): 1 (Operation not permitted) > at galera/src/replicator\_str.cpp:prepare\_for\_IST():489. IST will be unavailable. > 2016-01-11 16:24:58 16880 [Note] WSREP: Member 1.0 (server2in) requested state transfer from ‘_any_’. Selected 0.0 (server3in)(SYNCED) as donor. > 2016-01-11 16:24:58 16880 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 1309360) > 2016-01-11 16:24:58 16880 [Note] WSREP: Requesting state transfer: success, donor: 0 > WSREP\_SST: [ERROR] Parent mysqld process (PID:16880) terminated unexpectedly. (20160111 16:24:58.443) > WSREP\_SST: [ERROR] Cleanup after exit with status:32 (20160111 16:24:58.446) > 2016-01-11 16:24:58 16880 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.10.12’ --datadir ‘/srv/mysql/data/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16880’ ‘’ : 32 (Broken pipe) > 2016-01-11 16:24:58 16880 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. > 2016-01-11 16:24:58 16880 [ERROR] WSREP: SST script aborted with error 32 (Broken pipe) > 2016-01-11 16:24:58 16880 [ERROR] WSREP: SST failed: 32 (Broken pipe) > 2016-01-11 16:24:58 16880 [ERROR] Aborting > > 2016-01-11 16:25:00 16880 [Note] WSREP: (789476f2, ‘tcp://0.0.0.0:4567’) turning message relay requesting off > 2016-01-11 16:25:00 16880 [Note] WSREP: Closing send monitor… > 2016-01-11 16:25:00 16880 [Note] WSREP: Closed send monitor. > 2016-01-11 16:25:00 16880 [Note] WSREP: gcomm: terminating thread > 2016-01-11 16:25:00 16880 [Note] WSREP: gcomm: joining thread > 2016-01-11 16:25:00 16880 [Note] WSREP: gcomm: closing backend > 2016-01-11 16:25:00 16880 [Note] WSREP: view(view\_id(NON\_PRIM,0ee12c0b,10) memb { > 789476f2,0 > } joined { > } left { > } partitioned { > 0ee12c0b,0 > dae17fee,0 > }) > 2016-01-11 16:25:00 16880 [Note] WSREP: view((empty)) > 2016-01-11 16:25:00 16880 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1 > 2016-01-11 16:25:00 16880 [Note] WSREP: gcomm: closed > 2016-01-11 16:25:00 16880 [Note] WSREP: Flow-control interval: [16, 16] > 2016-01-11 16:25:00 16880 [Note] WSREP: Received NON-PRIMARY. > 2016-01-11 16:25:00 16880 [Note] WSREP: Shifting JOINER → OPEN (TO: 1309360) > 2016-01-11 16:25:00 16880 [Note] WSREP: Received self-leave message. > 2016-01-11 16:25:00 16880 [Note] WSREP: Flow-control interval: [0, 0] > 2016-01-11 16:25:00 16880 [Note] WSREP: Received SELF-LEAVE. Closing connection. > 2016-01-11 16:25:00 16880 [Note] WSREP: Shifting OPEN → CLOSED (TO: 1309360) > 2016-01-11 16:25:00 16880 [Note] WSREP: RECV thread exiting 0: Success > 2016-01-11 16:25:00 16880 [Note] WSREP: recv\_thread() joined. > 2016-01-11 16:25:00 16880 [Note] WSREP: Closing replication queue. > 2016-01-11 16:25:00 16880 [Note] WSREP: Closing slave action queue. > 2016-01-11 16:25:00 16880 [Note] WSREP: Service disconnected. > 2016-01-11 16:25:00 16880 [Note] WSREP: rollbacker thread exiting > 2016-01-11 16:25:01 16880 [Note] WSREP: Some threads may fail to exit. > 2016-01-11 16:25:01 16880 [Note] Binlog end > 2016-01-11 16:25:01 16880 [Note] /usr/sbin/mysqld: Shutdown complete > > Error in my\_thread\_global\_end(): 1 threads didn’t exit > 160111 16:25:06 mysqld\_safe mysqld from pid file /srv/mysql/data/server2.pid ended Logs from server3 (donor): > [@](#): > > 2016-01-11 16:24:56 6540 [Note] WSREP: (0ee12c0b, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: > 2016-01-11 16:24:57 6540 [Note] WSREP: declaring 789476f2 at tcp://192.168.10.12:4567 stable > 2016-01-11 16:24:57 6540 [Note] WSREP: declaring dae17fee at tcp://192.168.10.11:4567 stable > 2016-01-11 16:24:57 6540 [Note] WSREP: Node 0ee12c0b state prim > 2016-01-11 16:24:57 6540 [Note] WSREP: view(view\_id(PRIM,0ee12c0b,10) memb { > 0ee12c0b,0 > 789476f2,0 > dae17fee,0 > } joined { > } left { > } partitioned { > }) > 2016-01-11 16:24:57 6540 [Note] WSREP: save pc into disk > 2016-01-11 16:24:57 6540 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 0, memb\_num = 3 > 2016-01-11 16:24:57 6540 [Note] WSREP: STATE\_EXCHANGE: sent state UUID: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 > 2016-01-11 16:24:57 6540 [Note] WSREP: STATE EXCHANGE: sent state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 > 2016-01-11 16:24:57 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 0 (server3in) > 2016-01-11 16:24:57 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 2 (server1in) > 2016-01-11 16:24:57 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 1 (server2in) > 2016-01-11 16:24:57 6540 [Note] WSREP: Quorum results: > version = 3, > component = PRIMARY, > conf\_id = 8, > members = 2/3 (joined/total), > act\_id = 1309360, > last\_appl. = 1309329, > protocols = 0/7/3 (gcs/repl/appl), > group UUID = 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524 > 2016-01-11 16:24:57 6540 [Note] WSREP: Flow-control interval: [28, 28] > 2016-01-11 16:24:57 6540 [Note] WSREP: New cluster view: global state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360, view# 9: Primary, number of nodes: 3, my index: 0, protocol version 3 > 2016-01-11 16:24:57 6540 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. > 2016-01-11 16:24:57 6540 [Note] WSREP: REPL Protocols: 7 (3, 2) > 2016-01-11 16:24:57 6540 [Note] WSREP: Service thread queue flushed. > 2016-01-11 16:24:57 6540 [Note] WSREP: Assign initial position for certification: 1309360, protocol version: 3 > 2016-01-11 16:24:57 6540 [Note] WSREP: Service thread queue flushed. > 2016-01-11 16:24:58 6540 [Note] WSREP: Member 1.0 (server2in) requested state transfer from ‘_any_’. Selected 0.0 (server3in)(SYNCED) as donor. > 2016-01-11 16:24:58 6540 [Note] WSREP: Shifting SYNCED → DONOR/DESYNCED (TO: 1309360) > 2016-01-11 16:24:58 6540 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. > 2016-01-11 16:24:58 6540 [Note] WSREP: Running: ‘wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.10.12:4444/xtrabackup\_sst//1’ --socket ‘/var/run/mysqld/mysqld.sock’ --datadir ‘/srv/mysql/data/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360’’ > 2016-01-11 16:24:58 6540 [Note] WSREP: sst\_donor\_thread signaled with 0 > WSREP\_SST: [INFO] Streaming with xbstream (20160111 16:24:58.364) > WSREP\_SST: [INFO] Using socat as streamer (20160111 16:24:58.367) > WSREP\_SST: [INFO] Using /tmp/tmp.oPsbIOa4Wn as xtrabackup temporary directory (20160111 16:24:58.384) > WSREP\_SST: [INFO] Using /tmp/tmp.Hek7hwKeX2 as innobackupex temporary directory (20160111 16:24:58.387) > WSREP\_SST: [INFO] Streaming GTID file before SST (20160111 16:24:58.393) > WSREP\_SST: [INFO] Evaluating xbstream -c ${INFO\_FILE} | socat -u stdio TCP:192.168.10.12:4444; RC=( ${PIPESTATUS[@]} ) (20160111 16:24:58.399) > WSREP\_SST: [INFO] Sleeping before data transfer for SST (20160111 16:24:58.407) > 2016-01-11 16:25:00 6540 [Note] WSREP: (0ee12c0b, ‘tcp://0.0.0.0:4567’) turning message relay requesting off > 2016-01-11 16:25:00 6540 [Note] WSREP: declaring dae17fee at tcp://192.168.10.11:4567 stable > 2016-01-11 16:25:00 6540 [Note] WSREP: forgetting 789476f2 (tcp://192.168.10.12:4567) > 2016-01-11 16:25:00 6540 [Note] WSREP: Node 0ee12c0b state prim > 2016-01-11 16:25:00 6540 [Note] WSREP: view(view\_id(PRIM,0ee12c0b,11) memb { > 0ee12c0b,0 > dae17fee,0 > } joined { > } left { > } partitioned { > 789476f2,0 > }) > 2016-01-11 16:25:00 6540 [Note] WSREP: save pc into disk > 2016-01-11 16:25:00 6540 [Note] WSREP: forgetting 789476f2 (tcp://192.168.10.12:4567) > 2016-01-11 16:25:00 6540 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 0, memb\_num = 2 > 2016-01-11 16:25:00 6540 [Note] WSREP: STATE\_EXCHANGE: sent state UUID: 7abdc036-b877-11e5-88e6-9a7ccc4b0bda > 2016-01-11 16:25:00 6540 [Note] WSREP: STATE EXCHANGE: sent state msg: 7abdc036-b877-11e5-88e6-9a7ccc4b0bda > 2016-01-11 16:25:00 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 7abdc036-b877-11e5-88e6-9a7ccc4b0bda from 0 (server3in) > 2016-01-11 16:25:00 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 7abdc036-b877-11e5-88e6-9a7ccc4b0bda from 1 (server1in) > 2016-01-11 16:25:00 6540 [Note] WSREP: Quorum results: > version = 3, > component = PRIMARY, > conf\_id = 9, > members = 2/2 (joined/total), > act\_id = 1309360, > last\_appl. = 1309329, > protocols = 0/7/3 (gcs/repl/appl), > group UUID = 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524 > 2016-01-11 16:25:00 6540 [Note] WSREP: Flow-control interval: [23, 23] > 2016-01-11 16:25:00 6540 [Note] WSREP: New cluster view: global state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360, view# 10: Primary, number of nodes: 2, my index: 0, protocol version 3 > 2016-01-11 16:25:00 6540 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. > 2016-01-11 16:25:00 6540 [Note] WSREP: REPL Protocols: 7 (3, 2) > 2016-01-11 16:25:00 6540 [Note] WSREP: Service thread queue flushed. > 2016-01-11 16:25:00 6540 [Note] WSREP: Assign initial position for certification: 1309360, protocol version: 3 > 2016-01-11 16:25:00 6540 [Note] WSREP: Service thread queue flushed. > 2016-01-11 16:25:05 6540 [Note] WSREP: cleaning up 789476f2 (tcp://192.168.10.12:4567) > WSREP\_SST: [INFO] Streaming the backup to joiner at 192.168.10.12 4444 (20160111 16:25:08.412) > WSREP\_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2\>${DATA}/innobackup.backup.log | socat -u stdio TCP:192.168.10.12:4444; RC=( ${PIPESTATUS[@]} ) (20160111 16:25:08.416) > 2016/01/11 16:25:08 socat[27142] E connect(3, AF=2 192.168.10.12:4444, 16): Connection refused > WSREP\_SST: [ERROR] innobackupex finished with error: 1. Check /srv/mysql/data//innobackup.backup.log (20160111 16:25:09.574) > WSREP\_SST: [ERROR] Cleanup after exit with status:22 (20160111 16:25:09.578) > WSREP\_SST: [INFO] Cleaning up temporary directories (20160111 16:25:09.582) > 2016-01-11 16:25:09 6540 [ERROR] WSREP: Failed to read from: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.10.12:4444/xtrabackup\_sst//1’ --socket ‘/var/run/mysqld/mysqld.sock’ --datadir ‘/srv/mysql/data/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360’ > 2016-01-11 16:25:09 6540 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.10.12:4444/xtrabackup\_sst//1’ --socket ‘/var/run/mysqld/mysqld.sock’ --datadir ‘/srv/mysql/data/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360’: 22 (Invalid argument) > 2016-01-11 16:25:09 6540 [ERROR] WSREP: Command did not run: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.10.12:4444/xtrabackup\_sst//1’ --socket ‘/var/run/mysqld/mysqld.sock’ --datadir ‘/srv/mysql/data/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360’ > 2016-01-11 16:25:09 6540 [Warning] WSREP: Could not find peer: 789476f2-b877-11e5-b548-fee1e647e6b6 > 2016-01-11 16:25:09 6540 [Warning] WSREP: 0.0 (server3in): State transfer to -1.-1 (left the group) failed: -22 (Invalid argument) > 2016-01-11 16:25:09 6540 [Note] WSREP: Shifting DONOR/DESYNCED → JOINED (TO: 1309360) > 2016-01-11 16:25:09 6540 [Note] WSREP: Member 0.0 (server3in) synced with group. > 2016-01-11 16:25:09 6540 [Note] WSREP: Shifting JOINED → SYNCED (TO: 1309360) > 2016-01-11 16:25:09 6540 [Note] WSREP: Synchronized with group, ready for connections > 2016-01-11 16:25:09 6540 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. ---
**Author:** ![PercUse](https://avatars.discourse-cdn.com/v4/letter/p/dec6dc/32.png) [@PercUse](https://forums.percona.com/u/PercUse)\ **Post date:** [January 12, 2016, 8:27am UTC](https://forums.percona.com/t/can-start-mysql-or-resync-sst-after-last-update/4659/3 "2016-01-12T08:27:56Z")
Any help or idea will be welcome. 😉 I tried to force SST sync but it fail each time (open file limit ?) Seems to be hard to recover this server :s > [@](#): > > 2016-01-12 15:14:13 28030 [Warning] InnoDB: Cannot open table web567/ps\_smarty\_lazy\_cache from the internal data dictionary of InnoDB though the .frm file for the table exists. See [http://dev.mysql.com/doc/refman/5.6/…eshooting.html](http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html) for how you can resolve the problem. > 2016-01-12 15:14:13 28030 [Warning] WSREP: BF applier failed to open\_and\_lock\_tables: 1146, fatal: 0 wsrep = (exec\_mode: 1 conflict\_state: 0 seqno: 1320051) > 2016-01-12 15:14:13 28030 [ERROR] Slave SQL: Error executing row event: ‘Table ‘web567.ps\_smarty\_lazy\_cache’ doesn’t exist’, Error\_code: 1146 > 2016-01-12 15:14:13 28030 [Warning] WSREP: RBR event 3 Write\_rows apply warning: 1146, 1320051 > 2016-01-12 15:14:13 28030 [ERROR] WSREP: Failed to apply trx: source: e8566194-b935-11e5-94d5-139a15c6d8c2 version: 3 local: 0 state: APPLYING flags: 1 conn\_id: 738 trx\_id: 7013954 seqnos (l: 5, g: 1320051, s: 1320050, d: 1320050, ts: 2987010053581795) > 2016-01-12 15:14:13 28030 [ERROR] WSREP: Failed to apply trx 1320051 4 times > 2016-01-12 15:14:13 28030 [ERROR] WSREP: Node consistency compromized, aborting… > 2016-01-12 15:14:13 28030 [Note] WSREP: Closing send monitor… > 2016-01-12 15:14:13 28030 [Note] WSREP: Closed send monitor. > 2016-01-12 15:14:13 28030 [Note] WSREP: gcomm: terminating thread > 2016-01-12 15:14:13 28030 [Note] WSREP: gcomm: joining thread > 2016-01-12 15:14:13 28030 [Note] WSREP: gcomm: closing backend > 2016-01-12 15:14:13 28030 [Note] WSREP: view(view\_id(NON\_PRIM,4eb463f5,33) memb { > 94582432,0 > } joined { > } left { > } partitioned { > 4eb463f5,0 > e8566194,0 > }) > 2016-01-12 15:14:13 28030 [Note] WSREP: view((empty)) > 2016-01-12 15:14:13 28030 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1 > 2016-01-12 15:14:13 28030 [Note] WSREP: gcomm: closed > 2016-01-12 15:14:13 28030 [Note] WSREP: Flow-control interval: [16, 16] > 2016-01-12 15:14:13 28030 [Note] WSREP: Received NON-PRIMARY. > 2016-01-12 15:14:13 28030 [Note] WSREP: Shifting SYNCED → OPEN (TO: 1320051) > 2016-01-12 15:14:13 28030 [Note] WSREP: Received self-leave message. > 2016-01-12 15:14:13 28030 [Note] WSREP: Flow-control interval: [0, 0] > 2016-01-12 15:14:13 28030 [Note] WSREP: Received SELF-LEAVE. Closing connection. > 2016-01-12 15:14:13 28030 [Note] WSREP: Shifting OPEN → CLOSED (TO: 1320051) > 2016-01-12 15:14:13 28030 [Note] WSREP: RECV thread exiting 0: Success > 2016-01-12 15:14:13 28030 [Note] WSREP: recv\_thread() joined. > 2016-01-12 15:14:13 28030 [Note] WSREP: Closing replication queue. > 2016-01-12 15:14:13 28030 [Note] WSREP: Closing slave action queue. > 2016-01-12 15:14:13 28030 [Note] WSREP: /usr/sbin/mysqld: Terminated. > Aborted (core dumped) > 160112 15:14:13 mysqld\_safe Number of processes running now: 0 > 160112 15:14:13 mysqld\_safe WSREP: not restarting wsrep node automatically And It seems that we have this error with all DB on the affected node The affected node got kicked and die when triying to restart it: > [@](#): > > root@server2:~# tail -f /srv/mysql/data/wsrep\_recovery.gKOov7 > 2016-01-12 15:17:30 29954 [Note] InnoDB: Compressed tables use zlib 1.2.3.4 > 2016-01-12 15:17:30 29954 [Note] InnoDB: Using Linux native AIO > 2016-01-12 15:17:30 29954 [Note] InnoDB: Using CPU crc32 instructions > 2016-01-12 15:17:30 29954 [Note] InnoDB: Initializing buffer pool, size = 10.0G > 2016-01-12 15:17:30 29954 [Note] InnoDB: Completed initialization of buffer pool > 2016-01-12 15:17:31 29954 [Note] InnoDB: Highest supported file format is Barracuda. > 2016-01-12 15:17:31 29954 [Note] InnoDB: The log sequence numbers 94245707 and 94245707 in ibdata files do not match the log sequence number 94245871 in the ib\_logfiles! > 2016-01-12 15:17:31 29954 [Note] InnoDB: Database was not shutdown normally! > 2016-01-12 15:17:31 29954 [Note] InnoDB: Starting crash recovery. > 2016-01-12 15:17:31 29954 [Note] InnoDB: Reading tablespace information from the .ibd files… > 2016-01-12 15:18:46 29954 [Note] InnoDB: Restoring possible half-written data pages > 2016-01-12 15:18:46 29954 [Note] InnoDB: from the doublewrite buffer… > 2016-01-12 15:18:46 29954 [Note] InnoDB: 128 rollback segment(s) are active. > 2016-01-12 15:18:46 29954 [Note] InnoDB: Waiting for purge to start > 2016-01-12 15:18:46 29954 [Warning] InnoDB: Setting thread 30372 nice to -10 failed, current nice 0, errno 13 > 2016-01-12 15:18:46 29954 [Warning] InnoDB: Setting thread 30373 nice to -10 failed, current nice 0, errno 13 > 2016-01-12 15:18:46 29954 [Note] InnoDB: Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 5.6.27-76.0 started; log sequence number 94245871 > 2016-01-12 15:18:46 29954 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery. > 2016-01-12 15:18:46 29954 [Note] RSA private key file not found: /srv/mysql/data//private\_key.pem. Some authentication plugins will not work. > 2016-01-12 15:18:46 29954 [Note] RSA public key file not found: /srv/mysql/data//public\_key.pem. Some authentication plugins will not work. > 2016-01-12 15:18:46 29954 [Note] Server hostname (bind-address): ‘192.168.10.12’; port: 3306 > 2016-01-12 15:18:46 29954 [Note] - ‘192.168.10.12’ resolves to ‘192.168.10.12’; > 2016-01-12 15:18:46 29954 [Note] Server socket created on IP: ‘192.168.10.12’. > 2016-01-12 15:18:46 29954 [Note] WSREP: Recovered position: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320050 > 2016-01-12 15:18:46 29954 [Note] Binlog end > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘partition’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘BLACKHOLE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘ARCHIVE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘PERFORMANCE\_SCHEMA’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CHANGED\_PAGES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_DATAFILES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESPACES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN\_COLS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_FIELDS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_COLUMNS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_INDEXES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESTATS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_TABLE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_CACHE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_CONFIG’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_BEING\_DELETED’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_DELETED’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_DEFAULT\_STOPWORD’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_METRICS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_BUFFER\_POOL\_STATS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE\_LRU’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX\_RESET’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMPMEM\_RESET’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMPMEM’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMP\_RESET’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMP’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_LOCK\_WAITS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_LOCKS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_TRX’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘XTRADB\_RSEG’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘XTRADB\_INTERNAL\_HASH\_TABLES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘XTRADB\_READ\_VIEW’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘InnoDB’ > 2016-01-12 15:18:46 29954 [Note] InnoDB: FTS optimize thread exiting. > 2016-01-12 15:18:46 29954 [Note] InnoDB: Starting shutdown… > 2016-01-12 15:18:49 29954 [Note] InnoDB: Shutdown completed; log sequence number 94245881 > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘MyISAM’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘MRG\_MYISAM’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘MEMORY’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘CSV’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘sha256\_password’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘mysql\_old\_password’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘mysql\_native\_password’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘wsrep’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘binlog’ > 2016-01-12 15:18:49 29954 [Note] /usr/sbin/mysqld: Shutdown complete > > 160112 15:18:49 mysqld\_safe WSREP: Recovered position 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320050 > Log of wsrep recovery (–wsrep-recover): > 2016-01-12 15:17:30 29954 [Note] Plugin ‘FEDERATED’ is disabled. > InnoDB: Warning: innodb\_log\_block\_size has been changed from default value 512. (###EXPERIMENTAL### operation) > InnoDB: The log block size is set to 4096. > 2016-01-12 15:17:30 29954 [Note] InnoDB: Using atomics to ref count buffer pool pages > 2016-01-12 15:17:30 29954 [Note] InnoDB: The InnoDB memory heap is disabled > 2016-01-12 15:17:30 29954 [Note] InnoDB: Mutexes and rw\_locks use GCC atomic builtins > 2016-01-12 15:17:30 29954 [Note] InnoDB: Memory barrier is not used > 2016-01-12 15:17:30 29954 [Note] InnoDB: Compressed tables use zlib 1.2.3.4 > 2016-01-12 15:17:30 29954 [Note] InnoDB: Using Linux native AIO > 2016-01-12 15:17:30 29954 [Note] InnoDB: Using CPU crc32 instructions > 2016-01-12 15:17:30 29954 [Note] InnoDB: Initializing buffer pool, size = 10.0G > 2016-01-12 15:17:30 29954 [Note] InnoDB: Completed initialization of buffer pool > 2016-01-12 15:17:31 29954 [Note] InnoDB: Highest supported file format is Barracuda. > 2016-01-12 15:17:31 29954 [Note] InnoDB: The log sequence numbers 94245707 and 94245707 in ibdata files do not match the log sequence number 94245871 in the ib\_logfiles! > 2016-01-12 15:17:31 29954 [Note] InnoDB: Database was not shutdown normally! > 2016-01-12 15:17:31 29954 [Note] InnoDB: Starting crash recovery. > 2016-01-12 15:17:31 29954 [Note] InnoDB: Reading tablespace information from the .ibd files… > 2016-01-12 15:18:46 29954 [Note] InnoDB: Restoring possible half-written data pages > 2016-01-12 15:18:46 29954 [Note] InnoDB: from the doublewrite buffer… > 2016-01-12 15:18:46 29954 [Note] InnoDB: 128 rollback segment(s) are active. > 2016-01-12 15:18:46 29954 [Note] InnoDB: Waiting for purge to start > 2016-01-12 15:18:46 29954 [Warning] InnoDB: Setting thread 30372 nice to -10 failed, current nice 0, errno 13 > 2016-01-12 15:18:46 29954 [Warning] InnoDB: Setting thread 30373 nice to -10 failed, current nice 0, errno 13 > 2016-01-12 15:18:46 29954 [Note] InnoDB: Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 5.6.27-76.0 started; log sequence number 94245871 > 2016-01-12 15:18:46 29954 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery. > 2016-01-12 15:18:46 29954 [Note] RSA private key file not found: /srv/mysql/data//private\_key.pem. Some authentication plugins will not work. > 2016-01-12 15:18:46 29954 [Note] RSA public key file not found: /srv/mysql/data//public\_key.pem. Some authentication plugins will not work. > 2016-01-12 15:18:46 29954 [Note] Server hostname (bind-address): ‘192.168.10.12’; port: 3306 > 2016-01-12 15:18:46 29954 [Note] - ‘192.168.10.12’ resolves to ‘192.168.10.12’; > 2016-01-12 15:18:46 29954 [Note] Server socket created on IP: ‘192.168.10.12’. > 2016-01-12 15:18:46 29954 [Note] WSREP: Recovered position: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320050 > 2016-01-12 15:18:46 29954 [Note] Binlog end > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘partition’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘BLACKHOLE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘ARCHIVE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘PERFORMANCE\_SCHEMA’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CHANGED\_PAGES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_DATAFILES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESPACES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN\_COLS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_FIELDS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_COLUMNS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_INDEXES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESTATS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_TABLE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_CACHE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_CONFIG’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_BEING\_DELETED’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_DELETED’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_FT\_DEFAULT\_STOPWORD’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_METRICS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_BUFFER\_POOL\_STATS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE\_LRU’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX\_RESET’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMPMEM\_RESET’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMPMEM’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMP\_RESET’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_CMP’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_LOCK\_WAITS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_LOCKS’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘INNODB\_TRX’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘XTRADB\_RSEG’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘XTRADB\_INTERNAL\_HASH\_TABLES’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘XTRADB\_READ\_VIEW’ > 2016-01-12 15:18:46 29954 [Note] Shutting down plugin ‘InnoDB’ > 2016-01-12 15:18:46 29954 [Note] InnoDB: FTS optimize thread exiting. > 2016-01-12 15:18:46 29954 [Note] InnoDB: Starting shutdown… > 2016-01-12 15:18:49 29954 [Note] InnoDB: Shutdown completed; log sequence number 94245881 > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘MyISAM’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘MRG\_MYISAM’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘MEMORY’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘CSV’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘sha256\_password’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘mysql\_old\_password’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘mysql\_native\_password’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘wsrep’ > 2016-01-12 15:18:49 29954 [Note] Shutting down plugin ‘binlog’ > 2016-01-12 15:18:49 29954 [Note] /usr/sbin/mysqld: Shutdown complete > > 2016-01-12 15:18:50 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 30388 … > 2016-01-12 15:18:50 30388 [Note] WSREP: Read nil XID from storage engines, skipping position init > 2016-01-12 15:18:50 30388 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib/libgalera\_smm.so’ > 2016-01-12 15:18:50 30388 [Note] WSREP: wsrep\_load(): Galera 3.13(rb4bea65) by Codership Oy \ loaded successfully. > 2016-01-12 15:18:50 30388 [Note] WSREP: CRC-32C: using hardware acceleration. > 2016-01-12 15:18:50 30388 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 > 2016-01-12 15:18:50 30388 [Note] WSREP: Passing config to GCS: base\_dir = /srv/mysql/data/; base\_host = 192.168.10.12; base\_port = 4567; cert.log\_conflicts = no; debug = no; evs.auto\_evict = 0; evs.delay\_margin = PT1S; evs.delayed\_keep\_period = PT30S; evs.inactive\_check\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 4; evs.stats\_report\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 2; evs.view\_forget\_timeout = PT24H; gcache.dir = /srv/mysql/data/; gcache.keep\_pages\_count = 0; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /srv/mysql/data//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1.0; gcs.fc\_limit = 16; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignore\_quor > 2016-01-12 15:18:51 30388 [Note] WSREP: Service thread queue flushed. > 2016-01-12 15:18:51 30388 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 > 2016-01-12 15:18:51 30388 [Note] WSREP: wsrep\_sst\_grab() > 2016-01-12 15:18:51 30388 [Note] WSREP: Start replication > 2016-01-12 15:18:51 30388 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 > 2016-01-12 15:18:51 30388 [Note] WSREP: protonet asio version 0 > 2016-01-12 15:18:51 30388 [Note] WSREP: Using CRC-32C for message checksums. > 2016-01-12 15:18:51 30388 [Note] WSREP: backend: asio > 2016-01-12 15:18:51 30388 [Warning] WSREP: access file(/srv/mysql/data//gvwstate.dat) failed(No such file or directory) > 2016-01-12 15:18:51 30388 [Note] WSREP: restore pc from disk failed > 2016-01-12 15:18:51 30388 [Note] WSREP: GMCast version 0 > 2016-01-12 15:18:51 30388 [Note] WSREP: (672f38ea, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 > 2016-01-12 15:18:51 30388 [Note] WSREP: (672f38ea, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 > 2016-01-12 15:18:51 30388 [Note] WSREP: EVS version 0 > 2016-01-12 15:18:51 30388 [Note] WSREP: gcomm: connecting to group ‘prestaserver\_cluster’, peer ‘192.168.10.11:,192.168.10.12:,192.168.10.13:’ > 2016-01-12 15:18:51 30388 [Warning] WSREP: (672f38ea, ‘tcp://0.0.0.0:4567’) address ‘tcp://192.168.10.12:4567’ points to own listening address, blacklisting > 2016-01-12 15:18:51 30388 [Note] WSREP: (672f38ea, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: > 2016-01-12 15:18:51 30388 [Note] WSREP: declaring 4eb463f5 at tcp://192.168.10.13:4567 stable > 2016-01-12 15:18:51 30388 [Note] WSREP: declaring e8566194 at tcp://192.168.10.11:4567 stable > 2016-01-12 15:18:51 30388 [Note] WSREP: Node 4eb463f5 state prim > 2016-01-12 15:18:51 30388 [Note] WSREP: view(view\_id(PRIM,4eb463f5,35) memb { > 4eb463f5,0 > 672f38ea,0 > e8566194,0 > } joined { > } left { > } partitioned { > }) > 2016-01-12 15:18:51 30388 [Note] WSREP: save pc into disk > 2016-01-12 15:18:52 30388 [Note] WSREP: gcomm: connected > 2016-01-12 15:18:52 30388 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 > 2016-01-12 15:18:52 30388 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) > 2016-01-12 15:18:52 30388 [Note] WSREP: Opened channel ‘prestaserver\_cluster’ > 2016-01-12 15:18:52 30388 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 1, memb\_num = 3 > 2016-01-12 15:18:52 30388 [Note] WSREP: Waiting for SST to complete. > 2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. > 2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: sent state msg: 677c4c97-b937-11e5-ac6f-3f661196e91d > 2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: got state msg: 677c4c97-b937-11e5-ac6f-3f661196e91d from 0 (server3in) > 2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: got state msg: 677c4c97-b937-11e5-ac6f-3f661196e91d from 2 (server1in) > 2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: got state msg: 677c4c97-b937-11e5-ac6f-3f661196e91d from 1 (server2in) > 2016-01-12 15:18:52 30388 [Note] WSREP: Quorum results: > version = 3, > component = PRIMARY, > conf\_id = 34, > members = 2/3 (joined/total), > act\_id = 1320125, > last\_appl. = -1, > protocols = 0/7/3 (gcs/repl/appl), > group UUID = 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524 > 2016-01-12 15:18:52 30388 [Note] WSREP: Flow-control interval: [28, 28] > 2016-01-12 15:18:52 30388 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 1320125) > 2016-01-12 15:18:52 30388 [Note] WSREP: State transfer required: > Group state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320125 > Local state: 00000000-0000-0000-0000-000000000000:-1 > 2016-01-12 15:18:52 30388 [Note] WSREP: New cluster view: global state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320125, view# 35: Primary, number of nodes: 3, my index: 1, protocol version 3 > 2016-01-12 15:18:52 30388 [Warning] WSREP: Gap in state sequence. Need state transfer. > 2016-01-12 15:18:52 30388 [Note] WSREP: Running: 'wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.10.12’ --datadir ‘/srv/mysql/data/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘30388’ ‘’ ’ > WSREP\_SST: [INFO] Streaming with xbstream (20160112 15:18:52.134) > WSREP\_SST: [INFO] Using socat as streamer (20160112 15:18:52.135) > WSREP\_SST: [INFO] Stale sst\_in\_progress file: /srv/mysql/data//sst\_in\_progress (20160112 15:18:52.139) > WSREP\_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160112 15:18:52.158) > 2016-01-12 15:18:52 30388 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.10.12:4444/xtrabackup\_sst//1 > 2016-01-12 15:18:52 30388 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. > 2016-01-12 15:18:52 30388 [Note] WSREP: REPL Protocols: 7 (3, 2) > 2016-01-12 15:18:52 30388 [Note] WSREP: Service thread queue flushed. > 2016-01-12 15:18:52 30388 [Note] WSREP: Assign initial position for certification: 1320125, protocol version: 3 > 2016-01-12 15:18:52 30388 [Note] WSREP: Service thread queue flushed. > 2016-01-12 15:18:52 30388 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (51c9a3db-aeaa-11e3-a5e6-bffb1faf3524): 1 (Operation not permitted) > at galera/src/replicator\_str.cpp:prepare\_for\_IST():489. IST will be unavailable. > 2016-01-12 15:18:52 30388 [Note] WSREP: Member 1.0 (server2in) requested state transfer from ‘_any_’. Selected 0.0 (server3in)(SYNCED) as donor. > 2016-01-12 15:18:52 30388 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 1320125) > 2016-01-12 15:18:52 30388 [Note] WSREP: Requesting state transfer: success, donor: 0 > WSREP\_SST: [ERROR] Parent mysqld process (PID:30388) terminated unexpectedly. (20160112 15:18:52.565) > WSREP\_SST: [ERROR] Cleanup after exit with status:32 (20160112 15:18:52.567) > 2016-01-12 15:18:52 30388 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.10.12’ --datadir ‘/srv/mysql/data/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘30388’ ‘’ : 32 (Broken pipe) > 2016-01-12 15:18:52 30388 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. > 2016-01-12 15:18:52 30388 [ERROR] WSREP: SST script aborted with error 32 (Broken pipe) > 2016-01-12 15:18:52 30388 [ERROR] WSREP: SST failed: 32 (Broken pipe) > 2016-01-12 15:18:52 30388 [ERROR] Aborting > > 2016-01-12 15:18:54 30388 [Note] WSREP: (672f38ea, ‘tcp://0.0.0.0:4567’) turning message relay requesting off > 2016-01-12 15:18:54 30388 [Note] WSREP: Closing send monitor… > 2016-01-12 15:18:54 30388 [Note] WSREP: Closed send monitor. > 2016-01-12 15:18:54 30388 [Note] WSREP: gcomm: terminating thread > 2016-01-12 15:18:54 30388 [Note] WSREP: gcomm: joining thread > 2016-01-12 15:18:54 30388 [Note] WSREP: gcomm: closing backend > 2016-01-12 15:18:54 30388 [Note] WSREP: view(view\_id(NON\_PRIM,4eb463f5,35) memb { > 672f38ea,0 > } joined { > } left { > } partitioned { > 4eb463f5,0 > e8566194,0 > }) > 2016-01-12 15:18:54 30388 [Note] WSREP: view((empty)) > 2016-01-12 15:18:54 30388 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1 > 2016-01-12 15:18:54 30388 [Note] WSREP: gcomm: closed > 2016-01-12 15:18:54 30388 [Note] WSREP: Flow-control interval: [16, 16] > 2016-01-12 15:18:54 30388 [Note] WSREP: Received NON-PRIMARY. > 2016-01-12 15:18:54 30388 [Note] WSREP: Shifting JOINER → OPEN (TO: 1320125) > 2016-01-12 15:18:54 30388 [Note] WSREP: Received self-leave message. > 2016-01-12 15:18:54 30388 [Note] WSREP: Flow-control interval: [0, 0] > 2016-01-12 15:18:54 30388 [Note] WSREP: Received SELF-LEAVE. Closing connection. > 2016-01-12 15:18:54 30388 [Note] WSREP: Shifting OPEN → CLOSED (TO: 1320125) > 2016-01-12 15:18:54 30388 [Note] WSREP: RECV thread exiting 0: Success > 2016-01-12 15:18:54 30388 [Note] WSREP: recv\_thread() joined. > 2016-01-12 15:18:54 30388 [Note] WSREP: Closing replication queue. > 2016-01-12 15:18:54 30388 [Note] WSREP: Closing slave action queue. > 2016-01-12 15:18:54 30388 [Note] WSREP: Service disconnected. > 2016-01-12 15:18:54 30388 [Note] WSREP: rollbacker thread exiting > 2016-01-12 15:18:55 30388 [Note] WSREP: Some threads may fail to exit. > 2016-01-12 15:18:55 30388 [Note] Binlog end > 2016-01-12 15:18:55 30388 [Note] /usr/sbin/mysqld: Shutdown complete > > Error in my\_thread\_global\_end(): 1 threads didn’t exit > 160112 15:19:00 mysqld\_safe mysqld from pid file /srv/mysql/data/server2.pid ended Thanks in advance ---
**Author:** ![PercUse](https://avatars.discourse-cdn.com/v4/letter/p/dec6dc/32.png) [@PercUse](https://forums.percona.com/u/PercUse)\ **Post date:** [February 5, 2016, 3:52am UTC](https://forums.percona.com/t/can-start-mysql-or-resync-sst-after-last-update/4659/4 "2016-02-05T03:52:43Z")
Hi Guys, I have the feeling that there’s not anymore someone who can answer our questions and it’s the case for different topic here not only mine… Regards,