2nd node unable to join cluster

I’m setting up a 3 node cluster . The first node starts up fine with bootstrap , the second node fails to join the cluster

[root@mysql-test02 mysql]# /etc/init.d/mysql start
MySQL (Percona XtraDB Cluster) is not running, but lock fil[FAILED]lock/subsys/mysql) exists
Starting MySQL (Percona XtraDB Cluster)…State transfer in progress, setting sleep higher
…The server quit without updating PID file (/var/lib/mysql[FAILED]est02.bo3.e-dialog.com.pid).
MySQL (Percona XtraDB Cluster) server startup failed! [FAILED]
[root@mysql-test02 mysql]#

Here is the relevant information from the node 2 error log -

2014-12-03 16:02:09 51655 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role ‘joiner’ --address ‘10.100.7.28’ --auth ‘sstuser:s3cret’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘51655’ ‘’ ’
WSREP_SST: [INFO] Streaming with tar (20141203 16:02:10.030)
WSREP_SST: [INFO] Using socat as streamer (20141203 16:02:10.031)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | pv -f -i 10 -N joiner 2>>/tmp/sstl.progress | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20141203 16:02:10.041)
2014-12-03 16:02:10 51655 [Note] WSREP: Prepared SST request: xtrabackup|10.100.7.28:4444/xtrabackup_sst
2014-12-03 16:02:10 51655 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-03 16:02:10 51655 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-12-03 16:02:10 51655 [Note] WSREP: Service thread queue flushed.
2014-12-03 16:02:10 51655 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
2014-12-03 16:02:10 51655 [Note] WSREP: Service thread queue flushed.
2014-12-03 16:02:10 51655 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (60c7c2b6-7b1d-11e4-9d8f-3723cbbcdb67): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
2014-12-03 16:02:10 51655 [Note] WSREP: Member 1.0 (mysql-test02.bo3.e-dialog.com) requested state transfer from ‘any’. Selected 0.0 (mysql-test01.bo3.e-dialog.com)(SYNCED) as donor.
2014-12-03 16:02:10 51655 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 3)
2014-12-03 16:02:10 51655 [Note] WSREP: Requesting state transfer: success, donor: 0
2014-12-03 16:02:12 51655 [Note] WSREP: (a4faecfe, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2014-12-03 16:02:13 51655 [Note] WSREP: 0.0 (mysql-test01.bo3.e-dialog.com): State transfer to 1.0 (mysql-test02.bo3.e-dialog.com) complete.
2014-12-03 16:02:13 51655 [Note] WSREP: Member 0.0 (mysql-test01.bo3.e-dialog.com) synced with group.
WSREP_SST: [ERROR] xtrabackup process ended without creating ‘/var/lib/mysql//xtrabackup_galera_info’ (20141203 16:02:13.684)
WSREP_SST: [INFO] Contents of datadir (20141203 16:02:13.685)
WSREP_SST: [INFO] -rw------- 1 mysql mysql 8820 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.frm
-rw------- 1 mysql mysql 0 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.MYD
-rw------- 1 mysql mysql 4096 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.MYI
-rw------- 1 mysql mysql 9582 Dec 3 13:09 /var/lib/mysql//mysql/db.frm
-rw------- 1 mysql mysql 880 Dec 3 13:09 /var/lib/mysql//mysql/db.MYD
-rw------- 1 mysql mysql 5120 Dec 3 13:09 /var/lib/mysql//mysql/db.MYI
-rw------- 1 mysql mysql 10223 Dec 3 13:09 /var/lib/mysql//mysql/event.frm
-rw------- 1 mysql mysql 0 Dec 3 13:09 /var/lib/mysql//mysql/event.MYD
.
.
.
-rw-rw---- 1 mysql mysql 13518 Dec 3 13:09 /var/lib/mysql//performance_schema/table_lock_waits_summary_by_table.frm
-rw-rw---- 1 mysql mysql 9201 Dec 3 13:09 /var/lib/mysql//performance_schema/threads.frm
-rw-rw---- 1 mysql mysql 8676 Dec 3 13:09 /var/lib/mysql//performance_schema/users.frm
-rw-rw---- 1 mysql mysql 0 Dec 3 16:02 /var/lib/mysql//test/db.opt (20141203 16:02:13.694)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20141203 16:02:13.698)
WSREP_SST: [INFO] Removing the sst_in_progress file (20141203 16:02:13.699)
2014-12-03 16:02:13 51655 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role ‘joiner’ --address ‘10.100.7.28’ --auth ‘sstuser:s3cret’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘51655’ ‘’ : 32 (Broken pipe)
2014-12-03 16:02:13 51655 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-12-03 16:02:13 51655 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2014-12-03 16:02:13 51655 [ERROR] Aborting

On node 1 the the innobackup.backup.log log looks clean -

[root@mysql-test01 mysql]# cat innobackup.backup.log

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

141203 16:02:10 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_s ocket=/var/lib/mysql/mysql.sock’ as ‘sstuser’ (using password: YES).
141203 16:02:10 innobackupex: Connected to MySQL server
141203 16:02:10 innobackupex: Executing a version check against the server…
141203 16:02:10 innobackupex: Done.
141203 16:02:10 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.

innobackupex: Using server version 5.6.21-70.1-56

innobackupex: Created backup directory /tmp

141203 16:02:10 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file=“/etc/my.cnf” --defaults-group=“mysqld” --backup --suspend-at-end --target-dir=/tmp --innodb_log_file_size=“50331648” --innodb_data_file_path=“ibdata1:12M:autoextend” --tmpdir=/tmp --extra-lsndir=‘/tmp’ --stream=tar
innobackupex: Waiting for ibbackup (pid=53974) to suspend
innobackupex: Suspend file ‘/tmp/xtrabackup_suspended_2’

xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 0, set to 131072
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648

log scanned up to (1626819)
xtrabackup: Generating a list of tablespaces
[01] Streaming ./ibdata1
[01] …done
[01] Streaming ./mysql/slave_worker_info.ibd
[01] …done
[01] Streaming ./mysql/slave_relay_log_info.ibd
[01] …done
[01] Streaming ./mysql/innodb_table_stats.ibd
[01] …done
[01] Streaming ./mysql/innodb_index_stats.ibd
[01] …done
[01] Streaming ./mysql/slave_master_info.ibd
[01] …done
log scanned up to (1626819)
xtrabackup: Creating suspend file ‘/tmp/xtrabackup_suspended_2’ with pid ‘53974’

141203 16:02:12 innobackupex: Continuing after ibbackup has suspended
141203 16:02:12 innobackupex: Executing LOCK TABLES FOR BACKUP…
141203 16:02:12 innobackupex: Backup tables lock acquired

141203 16:02:12 innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of ‘/var/lib/mysql/’
innobackupex: Backing up files ‘/var/lib/mysql//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}’ (74 files)

log scanned up to (1626819)
innobackupex: Backing up files ‘/var/lib/mysql//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}’ (53 files)
141203 16:02:12 innobackupex: Finished backing up non-InnoDB tables and files

141203 16:02:12 innobackupex: Executing LOCK BINLOG FOR BACKUP…
141203 16:02:12 innobackupex: Executing FLUSH ENGINE LOGS…
141203 16:02:12 innobackupex: Waiting for log copying to finish

xtrabackup: The latest check point (for incremental): ‘1626819’
xtrabackup: Stopping log copying thread.
.>> log scanned up to (1626819)

xtrabackup: Creating suspend file ‘/tmp/xtrabackup_log_copied’ with pid ‘53974’
xtrabackup: Transaction log of lsn (1626819) to (1626819) was copied.
141203 16:02:13 innobackupex: Executing UNLOCK BINLOG
141203 16:02:13 innobackupex: Executing UNLOCK TABLES
141203 16:02:13 innobackupex: All tables unlocked

innobackupex: Backup created in directory ‘/tmp’
141203 16:02:13 innobackupex: Connection to database server closed
innobackupex: You must use -i (–ignore-zeros) option for extraction of the tar stream.
141203 16:02:13 innobackupex: completed OK!
[root@mysql-test01 mysql]#

Contents of /etc/my.cnf on node 2 -

[mysqld]

datadir=/var/lib/mysql
user=mysql

Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://10.100.7.27,10.100.7.28,10.100.7.29

In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

Node #2 address

wsrep_node_address=10.100.7.28

SST method

wsrep_sst_method=xtrabackup

Cluster name

wsrep_cluster_name=mysql_test_cluster

Authentication for SST method

wsrep_sst_auth=“sstuser:s3cret”

Any news on this issue? I am struggling with the same problem. :frowning:

No solution yet . I see from your thread that you reverted to 5.6.20 , going to try that .

Scratch that . This is what worked -

commenting out the line “wsrep_sst_method=xtrabackup” in the my.cnf file

Looks similar to the answer to this thread:
[url]http://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/28266-not-possible-for-joiner-to-join-the-cluster[/url]

The wsrep_sst_method=xtrabackup method is apparently being depricated, so commenting out that line made it use the default (probably the xtrabackup-v2 mentioned in the above thread) which then worked.

That’s right! I’ve just tested it, commenting out the option wsrep_sst_method, it has worked finely well. It seems that the variable’s default is xtrabackup-v2 as you well said.