# Nodes don't join after upgrade to XtraDB-Cluster-5.6.22-72.0 and xtrabackup-2.2.10 **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [May 26, 2015, 9:42am UTC](https://forums.percona.com/t/nodes-dont-join-after-upgrade-to-xtradb-cluster-5-6-22-72-0-and-xtrabackup-2-2-10/4240 "2015-05-26T09:42:29Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![ceyu](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@ceyu](https://forums.percona.com/u/ceyu)\ **Post date:** [May 26, 2015, 9:42am UTC](https://forums.percona.com/t/nodes-dont-join-after-upgrade-to-xtradb-cluster-5-6-22-72-0-and-xtrabackup-2-2-10/4240/1 "2015-05-26T09:42:29Z")
Hi all, you see below the my.cnf and extractions of the logs on the joiner and donor. Unfortunately, I can’t find out the problem, so I do need your help. I started the first node with /etc/init.d/mysql bootstrap-pxc. Thank you in advance. Cemil DONOR - my.cnf [mysqld] binlog\_format = ROW default\_storage\_engine = InnoDB innodb\_autoinc\_lock\_mode = 2 innodb\_locks\_unsafe\_for\_binlog = 1 innodb\_log\_file\_size = 512M innodb\_data\_file\_path = ibdata1:24M:autoextend innodb\_buffer\_pool\_size = 12G #innodb\_flush\_log\_at\_trx\_commit = 2 query\_cache\_size = 0 query\_cache\_type = 0 bind-address = 0.0.0.0 log\_slave\_updates ## ## WSREP options ## wsrep\_provider = /services/mysql/lib/libgalera\_smm.so wsrep\_provider\_options = “gcache.size=256M” wsrep\_cluster\_name = mysql\_cluster wsrep\_cluster\_address = gcomm://192.168.128.196,192.168.128.197,192.168.128.198 wsrep\_node\_name = db1-private wsrep\_node\_address = 192.168.128.196 wsrep\_sst\_method = xtrabackup-v2 wsrep\_sst\_auth = puser:secret # Optional wsrep Provider Settings wsrep\_node\_incoming\_address = 192.168.128.196 wsrep\_sst\_receive\_address = 192.168.128.196 wsrep\_slave\_threads = 8 #wsrep\_replicate\_myisam = 1 JOINER - my.cnf [mysqld] binlog\_format = ROW default\_storage\_engine = InnoDB innodb\_autoinc\_lock\_mode = 2 innodb\_locks\_unsafe\_for\_binlog = 1 innodb\_log\_file\_size = 512M innodb\_data\_file\_path = ibdata1:24M:autoextend innodb\_buffer\_pool\_size = 12G #innodb\_flush\_log\_at\_trx\_commit = 2 query\_cache\_size = 0 query\_cache\_type = 0 bind-address = 0.0.0.0 log\_slave\_updates ## ## WSREP options ## wsrep\_provider = /services/mysql/lib/libgalera\_smm.so wsrep\_provider\_options = “gcache.size=256M” wsrep\_cluster\_name = mysql\_cluster wsrep\_cluster\_address = gcomm://192.168.128.196,192.168.128.197,192.168.128.198 wsrep\_node\_name = db2-private wsrep\_node\_address = 192.168.128.197 wsrep\_sst\_method = xtrabackup-v2 wsrep\_sst\_auth = puser:secret # Optional wsrep Provider Settings wsrep\_node\_incoming\_address = 192.168.128.197 wsrep\_sst\_receive\_address = 192.168.128.197 wsrep\_slave\_threads = 8 #wsrep\_replicate\_myisam = 1 JOINER - ERROR-LOG WSREP\_SST: [INFO] Waiting for SST streaming to complete! (20150526 16:15:27.N) xbstream: Can’t create/write to file ‘././backup-my.cnf’ (Errcode: 2 - No such file or directory) xbstream: failed to create file. 2015/05/26 16:15:38 socat[82119] E write(1, 0x802451000, 41): Broken pipe WSREP\_SST: [ERROR] Error while getting data from donor node: exit codes: 1 1 (20150526 16:15:38.N) WSREP\_SST: [ERROR] Cleanup after exit with status:32 (20150526 16:15:38.N) 2015-05-26 16:15:38 81866 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.128.197’ --auth ‘puser:secret’ --datadir ‘/services/mysql/data/’ --defaults-file ‘/services/mysql/my.cnf’ --parent ‘81866’ ‘’ : 32 (Broken pipe) 2015-05-26 16:15:38 81866 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 2015-05-26 16:15:38 81866 [ERROR] WSREP: SST failed: 32 (Broken pipe) 2015-05-26 16:15:38 81866 [ERROR] Aborting 2015-05-26 16:15:38 81866 [Warning] WSREP: 0.0 (db1-private): State transfer to 1.0 (db2-private) failed: -22 (Invalid argument) 2015-05-26 16:15:38 81866 [ERROR] WSREP: gcs/src/gcs\_group.cpp:int gcs\_group\_handle\_join\_msg(gcs\_group\_t\*, const gcs\_recv\_msg\_t\*)():731: Will never receive state. Need to abort. 2015-05-26 16:15:38 81866 [Note] WSREP: gcomm: terminating thread 2015-05-26 16:15:38 81866 [Note] WSREP: gcomm: joining thread 2015-05-26 16:15:38 81866 [Note] WSREP: gcomm: closing backend 2015-05-26 16:15:39 81866 [Note] WSREP: view(view\_id(NON\_PRIM,6641ad6c,18) memb { a60e3d2b,0 } joined { } left { } partitioned { 6641ad6c,0 }) 2015-05-26 16:15:39 81866 [Note] WSREP: view((empty)) 2015-05-26 16:15:39 81866 [Note] WSREP: gcomm: closed 2015-05-26 16:15:39 81866 [Note] WSREP: mysqld: Terminated. … DONOR - ERROR-LOG … WSREP\_SST: [INFO] Streaming with xbstream (20150526 16:15:27.N) WSREP\_SST: [INFO] Using socat as streamer (20150526 16:15:27.N) WSREP\_SST: [INFO] Using /tmp/tmp.4O9pcm11 as xtrabackup temporary directory (20150526 16:15:27.N) WSREP\_SST: [INFO] Using /tmp/tmp.2zGAaXD6 as innobackupex temporary directory (20150526 16:15:27.N) WSREP\_SST: [INFO] Streaming GTID file before SST (20150526 16:15:27.N) WSREP\_SST: [INFO] Evaluating xbstream -c ${INFO\_FILE} | socat -u stdio TCP:192.168.128.197:4444; RC=( ${PIPESTATUS[@]} ) (20150526 16:15:27.N) WSREP\_SST: [INFO] Sleeping before data transfer for SST (20150526 16:15:27.N) WSREP\_SST: [INFO] Streaming the backup to joiner at 192.168.128.197 4444 (20150526 16:15:37.N) WSREP\_SST: [INFO] Evaluating innobackupex --defaults-file=/services/mysql/my.cnf --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2\>${DATA}/innobackup.backup.log | socat -u stdio TCP:192.168.128.197:4444; RC=( ${PIPESTATUS[@]} ) (20150526 16:15:37.N) 2015/05/26 16:15:38 socat[25151] E write(5, 0x802451000, 8192): Broken pipe WSREP\_SST: [ERROR] innobackupex finished with error: 1. Check /services/mysql/data//innobackup.backup.log (20150526 16:15:38.N) WSREP\_SST: [ERROR] Cleanup after exit with status:22 (20150526 16:15:38.N) WSREP\_SST: [INFO] Cleaning up temporary directories (20150526 16:15:38.N) 2015-05-26 16:15:38 57661 [ERROR] WSREP: Failed to read from: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.128.197:4444/xtrabackup\_sst//1’ --auth ‘puser:secret’ --socket ‘/tmp/mysql. sock’ --datadir ‘/services/mysql/data/’ --defaults-file ‘/services/mysql/my.cnf’ ‘’ --gtid ‘8e7b1114-c27b-11e4-a13f-f6862ce824b1:4625349’ 2015-05-26 16:15:38 57661 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.128.197:4444/xtrabackup\_sst//1’ --auth ‘puser:secret’ --socket ‘/t mp/mysql.sock’ --datadir ‘/services/mysql/data/’ --defaults-file ‘/services/mysql/my.cnf’ ‘’ --gtid ‘8e7b1114-c27b-11e4-a13f-f6862ce824b1:4625349’: 22 (Invalid argument) 2015-05-26 16:15:38 57661 [ERROR] WSREP: Command did not run: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.128.197:4444/xtrabackup\_sst//1’ --auth ‘puser:secret’ --socket ‘/tmp/mysql. sock’ --datadir ‘/services/mysql/data/’ --defaults-file ‘/services/mysql/my.cnf’ ‘’ --gtid ‘8e7b1114-c27b-11e4-a13f-f6862ce824b1:4625349’ 2015-05-26 16:15:38 57661 [Warning] WSREP: 0.0 (db1-private): State transfer to 1.0 (db2-private) failed: -22 (Invalid argument) … DONOR - innobackup.backup.log 150526 16:15:37 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql\_read\_default\_file=/services/mysql/my.cnf;mysql\_read\_default\_group=xtrabackup;mysql\_socket=/tmp/mysql.sock’ as ‘pu ser’ (using password: YES). 150526 16:15:37 innobackupex: Connected to MySQL server 150526 16:15:37 innobackupex: Starting the backup operation innobackupex: Using server version 5.6.22-72.0 innobackupex: Created backup directory /tmp/tmp.2zGAaXD6 150526 16:15:37 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file=“/services/mysql/my.cnf” --defaults-group=“mysqld” --backup --suspend-at-end --target-dir=/tmp/tmp.4O9pcm11 –tmpdir=/tmp/tmp.4O9pcm11 --extra-lsndir=‘/tmp/tmp.4O9pcm11’ --stream=xbstream innobackupex: Waiting for ibbackup (pid=25154) to suspend innobackupex: Suspend file ‘/tmp/tmp.4O9pcm11/xtrabackup\_suspended\_2’ xtrabackup version 2.2.10 based on MySQL server 5.6.22 FreeBSD10.1 (amd64) (revision id: ) xtrabackup: uses posix\_fadvise(). xtrabackup: cd to /services/mysql/data xtrabackup: open files limit requested 0, set to 942318 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb\_data\_home\_dir = ./ xtrabackup: innodb\_data\_file\_path = ibdata1:24M:autoextend xtrabackup: innodb\_log\_group\_home\_dir = ./ xtrabackup: innodb\_log\_files\_in\_group = 2 xtrabackup: innodb\_log\_file\_size = 536870912 > > log scanned up to (58176722719) > > xtrabackup: Generating a list of tablespaces > > [01] Streaming ./ibdata1 > > ^Gxtrabackup: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe) > > xb\_stream\_write\_data() failed. > > ^Gxtrabackup: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe) > > [01] xtrabackup: Error: xtrabackup\_copy\_datafile() failed. > > [01] xtrabackup: Error: failed to copy datafile. > > innobackupex: got a fatal error with the following stacktrace: at /services/mysql/bin/innobackupex line 2703. > > main::wait\_for\_ibbackup\_file\_create(‘/tmp/tmp.4O9pcm11/xtrabackup\_suspended\_2’) called at /services/mysql/bin/innobackupex line 2723 > > main::wait\_for\_ibbackup\_suspend(‘/tmp/tmp.4O9pcm11/xtrabackup\_suspended\_2’) called at /services/mysql/bin/innobackupex line 1976 > > main::backup() called at /services/mysql/bin/innobackupex line 1601 > > innobackupex: Error: The xtrabackup child process has died at /services/mysql/bin/innobackupex line 2703. ---
**Author:** ![Llew](https://avatars.discourse-cdn.com/v4/letter/l/e274bd/32.png) [@Llew](https://forums.percona.com/u/Llew)\ **Post date:** [June 15, 2015, 7:24am UTC](https://forums.percona.com/t/nodes-dont-join-after-upgrade-to-xtradb-cluster-5-6-22-72-0-and-xtrabackup-2-2-10/4240/2 "2015-06-15T07:24:26Z")
Hi Ceyu, I had exactly the same problem. It turns out that I had to delete the folder $datadir/.sst (on the joiner) to solve this problem. Hope this can help you. Best regards. Llew. ---
**Author:** ![mirfan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mirfan/32/16890_2.png) [@mirfan](https://forums.percona.com/u/mirfan)\ **Post date:** [June 16, 2015, 1:19am UTC](https://forums.percona.com/t/nodes-dont-join-after-upgrade-to-xtradb-cluster-5-6-22-72-0-and-xtrabackup-2-2-10/4240/3 "2015-06-16T01:19:40Z")
As Liew described, you may delete $datadir/.sst directory and try again. I found this bug too [url][https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1431101[/url]](https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1431101%5B/url%5D)