I currently have a 3 node cluster and was hoping to add a 4th node. The first 3 installs went rather easily though this one is having a hang up. Any help would be greatly appreciated.
Everything is running cent 6.4 x64
Current Nodes:
vm02db1.domain.com
vm03db1.domain.com ← Started with /etc/init.d/mysql start --wsrep-cluster-address=“gcomm://”
vm04db1.domain.com
Trying to add Node:
vm01db1.domain.com
my.cnf
[mysqld]
datadir=/Array/perconadb/data
tmpdir=/Array/perconadb/tmp
user=mysql
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_slave_threads=8
wsrep_cluster_address=gcomm://vm01db1.domain.com,vm02db1.domain.com,vm03db1.domain.com,vm04db1.domain.com
binlog_format=ROW
default_storage_engine=InnoDB
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
wsrep_node_address=vm01db1.domain.com
wsrep_sst_method=xtrabackup
wsrep_sst_auth=sstuser:mysupersecretpasswordhere ← Thats not really the password i use 
From the DONOR (vm03db1) - This is the sync trying to kick off
130613 10:02:03 [Note] WSREP: Flow-control interval: [32, 32]
130613 10:02:03 [Note] WSREP: New cluster view: global state: 2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388, view# 24: Primary, number of nodes: 4, my index: 0, protocol version 2
130613 10:02:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130613 10:02:04 [Note] WSREP: Assign initial position for certification: 1191388, protocol version: 2
130613 10:02:05 [Note] WSREP: Node 3 (vm01db1) requested state transfer from ‘any’. Selected 0 (vm03db1)(SYNCED) as donor.
130613 10:02:05 [Note] WSREP: Shifting SYNCED → DONOR/DESYNCED (TO: 1191388)
130613 10:02:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
From the DONOR (vm03db1) - This is the error I see
130613 10:02:05 [Note] WSREP: Running: ‘wsrep_sst_xtrabackup --role ‘donor’ --address ‘vm01db1.domain.com:4444/xtrabackup_sst’ --auth ‘sstuser:mysupersecretpasswordhere’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/Array/perconadb/data/’ --defaults-file ‘/etc/my.cnf’ --gtid ‘2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388’’
130613 10:02:05 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [ERROR] innobackupex finished with error: 25. Check /Array/perconadb/data//innobackup.backup.log (20130613 10:02:06.307)
130613 10:02:06 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup --role ‘donor’ --address ‘vm01db1.domain.com:4444/xtrabackup_sst’ --auth ‘sstuser:mysupersecretpasswordhere’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/Array/perconadb/data/’ --defaults-file ‘/etc/my.cnf’ --gtid ‘2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388’
130613 10:02:06 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role ‘donor’ --address ‘vm01db1.domain.com:4444/xtrabackup_sst’ --auth ‘sstuser:mysupersecretpasswordhere’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/Array/perconadb/data/’ --defaults-file ‘/etc/my.cnf’ --gtid ‘2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388’: 22 (Invalid argument)
130613 10:02:06 [Warning] WSREP: 0 (vm03db1): State transfer to 3 (vm01db1) failed: -1 (Operation not permitted)
From the DONOR (vm03db1) - This is the innobackupex log
30613 10:02:06 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock’ as ‘sstuser’ (using password: YES).
130613 10:02:06 innobackupex: Connected to MySQL server
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.
innobackupex: Using mysql server version 5.5.30
innobackupex: Created backup directory /tmp
tar: -: Cannot write: Broken pipe
tar: Error is not recoverable: exiting now
innobackupex: Error: Failed to stream ‘backup-my.cnf’: Inappropriate ioctl for device at /usr//bin/innobackupex line 389.
From the JOINER (vm01cv1) - This is the error I see
130613 10:02:03 [Note] WSREP: State transfer required:
Group state: 2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388
Local state: 00000000-0000-0000-0000-000000000000:-1
130613 10:02:03 [Note] WSREP: New cluster view: global state: 2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388, view# 24: Primary, number of nodes: 4, my index: 3, protocol version 2
130613 10:02:03 [Warning] WSREP: Gap in state sequence. Need state transfer.
130613 10:02:05 [Note] WSREP: Running: ‘wsrep_sst_xtrabackup --role ‘joiner’ --address ‘vm01db1.domain.com’ --auth ‘sstuser:mysupersecretpasswordhere’ --datadir ‘/Array/perconadb/data/’ --defaults-file ‘/etc/my.cnf’ --parent ‘3024’’
130613 10:02:05 [Note] WSREP: Prepared SST request: xtrabackup|vm01db1.domain.com:4444/xtrabackup_sst
130613 10:02:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130613 10:02:05 [Note] WSREP: Assign initial position for certification: 1191388, protocol version: 2
130613 10:02:05 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (2c96604b-ca07-11e2-0800-c45d1fb1ae6f): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():436. IST will be unavailable.
130613 10:02:05 [Note] WSREP: Node 3 (vm01db1) requested state transfer from ‘any’. Selected 0 (vm03db1)(SYNCED) as donor.
130613 10:02:05 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 1191388)
130613 10:02:05 [Note] WSREP: Requesting state transfer: success, donor: 0
130613 10:02:06 [Warning] WSREP: 0 (vm03db1): State transfer to 3 (vm01db1) failed: -1 (Operation not permitted)
130613 10:02:06 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.
130613 10:02:06 [Note] WSREP: gcomm: terminating thread
130613 10:02:06 [Note] WSREP: gcomm: joining thread
130613 10:02:06 [Note] WSREP: gcomm: closing backend