# 2nd node unable to join cluster **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [December 3, 2014, 3:39pm UTC](https://forums.percona.com/t/2nd-node-unable-to-join-cluster/3928 "2014-12-03T15:39:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sanjchatterj](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@sanjchatterj](https://forums.percona.com/u/sanjchatterj)\ **Post date:** [December 3, 2014, 3:39pm UTC](https://forums.percona.com/t/2nd-node-unable-to-join-cluster/3928/1 "2014-12-03T15:39:56Z")
I’m setting up a 3 node cluster . The first node starts up fine with bootstrap , the second node fails to join the cluster [root@mysql-test02 mysql]# /etc/init.d/mysql start MySQL (Percona XtraDB Cluster) is not running, but lock fil[FAILED]lock/subsys/mysql) exists Starting MySQL (Percona XtraDB Cluster)…State transfer in progress, setting sleep higher …The server quit without updating PID file (/var/lib/mysql[FAILED]est02.bo3.e-dialog.com.pid). MySQL (Percona XtraDB Cluster) server startup failed! [FAILED] [root@mysql-test02 mysql]# Here is the relevant information from the node 2 error log - 2014-12-03 16:02:09 51655 [Note] WSREP: Running: 'wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘10.100.7.28’ --auth ‘sstuser:s3cret’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘51655’ ‘’ ’ WSREP\_SST: [INFO] Streaming with tar (20141203 16:02:10.030) WSREP\_SST: [INFO] Using socat as streamer (20141203 16:02:10.031) WSREP\_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | pv -f -i 10 -N joiner 2\>\>/tmp/sstl.progress | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20141203 16:02:10.041) 2014-12-03 16:02:10 51655 [Note] WSREP: Prepared SST request: xtrabackup|10.100.7.28:4444/xtrabackup\_sst 2014-12-03 16:02:10 51655 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2014-12-03 16:02:10 51655 [Note] WSREP: REPL Protocols: 6 (3, 2) 2014-12-03 16:02:10 51655 [Note] WSREP: Service thread queue flushed. 2014-12-03 16:02:10 51655 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3 2014-12-03 16:02:10 51655 [Note] WSREP: Service thread queue flushed. 2014-12-03 16:02:10 51655 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (60c7c2b6-7b1d-11e4-9d8f-3723cbbcdb67): 1 (Operation not permitted) at galera/src/replicator\_str.cpp:prepare\_for\_IST():456. IST will be unavailable. 2014-12-03 16:02:10 51655 [Note] WSREP: Member 1.0 ([mysql-test02.bo3.e-dialog.com](http://mysql-test02.bo3.e-dialog.com)) requested state transfer from ‘_any_’. Selected 0.0 ([mysql-test01.bo3.e-dialog.com](http://mysql-test01.bo3.e-dialog.com))(SYNCED) as donor. 2014-12-03 16:02:10 51655 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 3) 2014-12-03 16:02:10 51655 [Note] WSREP: Requesting state transfer: success, donor: 0 2014-12-03 16:02:12 51655 [Note] WSREP: (a4faecfe, ‘tcp://0.0.0.0:4567’) turning message relay requesting off 2014-12-03 16:02:13 51655 [Note] WSREP: 0.0 ([mysql-test01.bo3.e-dialog.com](http://mysql-test01.bo3.e-dialog.com)): State transfer to 1.0 ([mysql-test02.bo3.e-dialog.com](http://mysql-test02.bo3.e-dialog.com)) complete. 2014-12-03 16:02:13 51655 [Note] WSREP: Member 0.0 ([mysql-test01.bo3.e-dialog.com](http://mysql-test01.bo3.e-dialog.com)) synced with group. WSREP\_SST: [ERROR] xtrabackup process ended without creating ‘/var/lib/mysql//xtrabackup\_galera\_info’ (20141203 16:02:13.684) WSREP\_SST: [INFO] Contents of datadir (20141203 16:02:13.685) WSREP\_SST: [INFO] -rw------- 1 mysql mysql 8820 Dec 3 13:09 /var/lib/mysql//mysql/columns\_priv.frm -rw------- 1 mysql mysql 0 Dec 3 13:09 /var/lib/mysql//mysql/columns\_priv.MYD -rw------- 1 mysql mysql 4096 Dec 3 13:09 /var/lib/mysql//mysql/columns\_priv.MYI -rw------- 1 mysql mysql 9582 Dec 3 13:09 /var/lib/mysql//mysql/db.frm -rw------- 1 mysql mysql 880 Dec 3 13:09 /var/lib/mysql//mysql/db.MYD -rw------- 1 mysql mysql 5120 Dec 3 13:09 /var/lib/mysql//mysql/db.MYI -rw------- 1 mysql mysql 10223 Dec 3 13:09 /var/lib/mysql//mysql/event.frm -rw------- 1 mysql mysql 0 Dec 3 13:09 /var/lib/mysql//mysql/event.MYD . . . -rw-rw---- 1 mysql mysql 13518 Dec 3 13:09 /var/lib/mysql//performance\_schema/table\_lock\_waits\_summary\_by\_table.frm -rw-rw---- 1 mysql mysql 9201 Dec 3 13:09 /var/lib/mysql//performance\_schema/threads.frm -rw-rw---- 1 mysql mysql 8676 Dec 3 13:09 /var/lib/mysql//performance\_schema/users.frm -rw-rw---- 1 mysql mysql 0 Dec 3 16:02 /var/lib/mysql//test/db.opt (20141203 16:02:13.694) WSREP\_SST: [ERROR] Cleanup after exit with status:32 (20141203 16:02:13.698) WSREP\_SST: [INFO] Removing the sst\_in\_progress file (20141203 16:02:13.699) 2014-12-03 16:02:13 51655 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘10.100.7.28’ --auth ‘sstuser:s3cret’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘51655’ ‘’ : 32 (Broken pipe) 2014-12-03 16:02:13 51655 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 2014-12-03 16:02:13 51655 [ERROR] WSREP: SST failed: 32 (Broken pipe) 2014-12-03 16:02:13 51655 [ERROR] Aborting On node 1 the the innobackup.backup.log log looks clean - [root@mysql-test01 mysql]# cat innobackup.backup.log InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. This software is published under the GNU GENERAL PUBLIC LICENSE Version 2, June 1991. Get the latest version of Percona XtraBackup, documentation, and help resources: [http://www.percona.com/xb/p](http://www.percona.com/xb/p) 141203 16:02:10 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql\_read\_default\_file=/etc/my.cnf;mysql\_read\_default\_group=xtrabackup;mysql\_s ocket=/var/lib/mysql/mysql.sock’ as ‘sstuser’ (using password: YES). 141203 16:02:10 innobackupex: Connected to MySQL server 141203 16:02:10 innobackupex: Executing a version check against the server… 141203 16:02:10 innobackupex: Done. 141203 16:02:10 innobackupex: Starting the backup operation IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints “completed OK!”. innobackupex: Using server version 5.6.21-70.1-56 innobackupex: Created backup directory /tmp 141203 16:02:10 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file=“/etc/my.cnf” --defaults-group=“mysqld” --backup --suspend-at-end --target-dir=/tmp --innodb\_log\_file\_size=“50331648” --innodb\_data\_file\_path=“ibdata1:12M:autoextend” --tmpdir=/tmp --extra-lsndir=‘/tmp’ --stream=tar innobackupex: Waiting for ibbackup (pid=53974) to suspend innobackupex: Suspend file ‘/tmp/xtrabackup\_suspended\_2’ xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86\_64) (revision id: ) xtrabackup: uses posix\_fadvise(). xtrabackup: cd to /var/lib/mysql xtrabackup: open files limit requested 0, set to 131072 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb\_data\_home\_dir = ./ xtrabackup: innodb\_data\_file\_path = ibdata1:12M:autoextend xtrabackup: innodb\_log\_group\_home\_dir = ./ xtrabackup: innodb\_log\_files\_in\_group = 2 xtrabackup: innodb\_log\_file\_size = 50331648 > > log scanned up to (1626819) > > xtrabackup: Generating a list of tablespaces > > [01] Streaming ./ibdata1 > > [01] …done > > [01] Streaming ./mysql/slave\_worker\_info.ibd > > [01] …done > > [01] Streaming ./mysql/slave\_relay\_log\_info.ibd > > [01] …done > > [01] Streaming ./mysql/innodb\_table\_stats.ibd > > [01] …done > > [01] Streaming ./mysql/innodb\_index\_stats.ibd > > [01] …done > > [01] Streaming ./mysql/slave\_master\_info.ibd > > [01] …done > > log scanned up to (1626819) > > xtrabackup: Creating suspend file ‘/tmp/xtrabackup\_suspended\_2’ with pid ‘53974’ 141203 16:02:12 innobackupex: Continuing after ibbackup has suspended 141203 16:02:12 innobackupex: Executing LOCK TABLES FOR BACKUP… 141203 16:02:12 innobackupex: Backup tables lock acquired 141203 16:02:12 innobackupex: Starting to backup non-InnoDB tables and files innobackupex: in subdirectories of ‘/var/lib/mysql/’ innobackupex: Backing up files ‘/var/lib/mysql//mysql/\*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}’ (74 files) > > log scanned up to (1626819) > > innobackupex: Backing up files ‘/var/lib/mysql//performance\_schema/\*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}’ (53 files) > > 141203 16:02:12 innobackupex: Finished backing up non-InnoDB tables and files 141203 16:02:12 innobackupex: Executing LOCK BINLOG FOR BACKUP… 141203 16:02:12 innobackupex: Executing FLUSH ENGINE LOGS… 141203 16:02:12 innobackupex: Waiting for log copying to finish xtrabackup: The latest check point (for incremental): ‘1626819’ xtrabackup: Stopping log copying thread. .\>\> log scanned up to (1626819) xtrabackup: Creating suspend file ‘/tmp/xtrabackup\_log\_copied’ with pid ‘53974’ xtrabackup: Transaction log of lsn (1626819) to (1626819) was copied. 141203 16:02:13 innobackupex: Executing UNLOCK BINLOG 141203 16:02:13 innobackupex: Executing UNLOCK TABLES 141203 16:02:13 innobackupex: All tables unlocked innobackupex: Backup created in directory ‘/tmp’ 141203 16:02:13 innobackupex: Connection to database server closed innobackupex: You must use -i (–ignore-zeros) option for extraction of the tar stream. 141203 16:02:13 innobackupex: completed OK! [root@mysql-test01 mysql]# ---
**Author:** ![sanjchatterj](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@sanjchatterj](https://forums.percona.com/u/sanjchatterj)\ **Post date:** [December 3, 2014, 3:44pm UTC](https://forums.percona.com/t/2nd-node-unable-to-join-cluster/3928/2 "2014-12-03T15:44:44Z")
Contents of /etc/my.cnf on node 2 - [mysqld] datadir=/var/lib/mysql user=mysql # Path to Galera library wsrep\_provider=/usr/lib64/libgalera\_smm.so # Cluster connection URL contains the IPs of node#1, node#2 and node#3 wsrep\_cluster\_address=gcomm://10.100.7.27,10.100.7.28,10.100.7.29 # In order for Galera to work correctly binlog format should be ROW binlog\_format=ROW # MyISAM storage engine has only experimental support default\_storage\_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb\_autoinc\_lock\_mode=2 # Node #2 address wsrep\_node\_address=10.100.7.28 # SST method wsrep\_sst\_method=xtrabackup # Cluster name wsrep\_cluster\_name=mysql\_test\_cluster # Authentication for SST method wsrep\_sst\_auth=“sstuser:s3cret” ---
**Author:** ![SouthernBelle](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@SouthernBelle](https://forums.percona.com/u/SouthernBelle)\ **Post date:** [December 10, 2014, 8:11am UTC](https://forums.percona.com/t/2nd-node-unable-to-join-cluster/3928/3 "2014-12-10T08:11:17Z")
Any news on this issue? I am struggling with the same problem. ☹ ---
**Author:** ![sanjchatterj](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@sanjchatterj](https://forums.percona.com/u/sanjchatterj)\ **Post date:** [December 16, 2014, 11:14am UTC](https://forums.percona.com/t/2nd-node-unable-to-join-cluster/3928/4 "2014-12-16T11:14:58Z")
No solution yet . I see from your thread that you reverted to 5.6.20 , going to try that . ---
**Author:** ![sanjchatterj](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@sanjchatterj](https://forums.percona.com/u/sanjchatterj)\ **Post date:** [December 16, 2014, 11:42am UTC](https://forums.percona.com/t/2nd-node-unable-to-join-cluster/3928/5 "2014-12-16T11:42:21Z")
Scratch that . This is what worked - commenting out the line “wsrep\_sst\_method=xtrabackup” in the my.cnf file ---
**Author:** ![scott.nemes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/scott.nemes/32/5_2.png) [@scott.nemes](https://forums.percona.com/u/scott.nemes)\ **Post date:** [December 16, 2014, 3:05pm UTC](https://forums.percona.com/t/2nd-node-unable-to-join-cluster/3928/6 "2014-12-16T15:05:14Z")
Looks similar to the answer to this thread: [url][http://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/28266-not-possible-for-joiner-to-join-the-cluster[/url]](http://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/28266-not-possible-for-joiner-to-join-the-cluster%5B/url%5D) The wsrep\_sst\_method=xtrabackup method is apparently being depricated, so commenting out that line made it use the default (probably the xtrabackup-v2 mentioned in the above thread) which then worked. ---
**Author:** ![wagnerbianchi](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/wagnerbianchi/32/802_2.png) [@wagnerbianchi](https://forums.percona.com/u/wagnerbianchi)\ **Post date:** [December 22, 2014, 7:04am UTC](https://forums.percona.com/t/2nd-node-unable-to-join-cluster/3928/7 "2014-12-22T07:04:21Z")
That’s right! I’ve just tested it, commenting out the option wsrep\_sst\_method, it has worked finely well. It seems that the variable’s default is xtrabackup-v2 as you well said.