error join cluster

hi,
i am building a percona cluster on[COLOR=#FF0000]Percona-XtraDB-Cluster-5.6.24-72.2
and got error on join the cluster.

pls,help me out…
tks.

[COLOR=#FF0000]error.log on node1

2015-07-15 16:19:07 21106 [Note] WSREP: Tables flushed.
rsync: recv_generator: mkdir "mysql" (in rsync_sst) failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "performance_schema" (in rsync_sst) failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "test" (in rsync_sst) failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: open "ibdata01" (in rsync_sst) failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
WSREP_SST: [ERROR] rsync returned code 23: (20150715 16:19:09.434)
2015-07-15 16:19:09 21106 [ERROR] WSREP: Failed to read from: wsrep_sst_rsync --role 'donor' --address '192.168.6.4:4444/rsync_sst' --auth '(null)' --socket '/mysql/data/mysql.sock' --datadir '/mysql/data/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'b8940f5c-2ac2-11e5-b81c-9aa06090fe52:0'
2015-07-15 16:19:09 21106 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'donor' --address '192.168.6.4:4444/rsync_sst' --auth '(null)' --socket '/mysql/data/mysql.sock' --datadir '/mysql/data/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'b8940f5c-2ac2-11e5-b81c-9aa06090fe52:0': 255 (Unknown error 255)
2015-07-15 16:19:09 21106 [Note] WSREP: resuming provider at 5
2015-07-15 16:19:09 21106 [Note] WSREP: Provider resumed.
2015-07-15 16:19:09 21106 [ERROR] WSREP: Command did not run: wsrep_sst_rsync --role 'donor' --address '192.168.6.4:4444/rsync_sst' --auth '(null)' --socket '/mysql/data/mysql.sock' --datadir '/mysql/data/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'b8940f5c-2ac2-11e5-b81c-9aa06090fe52:0'
2015-07-15 16:19:09 21106 [Warning] WSREP: 0.0 (ip63): State transfer to 1.0 (ip64) failed: -255 (Unknown error 255)
2015-07-15 16:19:09 21106 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)

[COLOR=#FF0000]and here is my conf file:

[root@mytest1 lib]# cat /etc/my.cnf 
[client]
port = 3306
socket = /mysql/data/mysql.sock
#prompt = "\u@\d> "
#pager = more


[mysqld]
# *************** main ***************
port = 3306
user = mysql
basedir = /usr/local/mysql
datadir = /mysql/data
socket = /mysql/data/mysql.sock
pid-file = /mysql/data/mysql.pid
log-error = error.err

# *************** General ***************
memlock
skip-name-resolve
explicit_defaults_for_timestamp

max_connections = 1000
back_log = 500
max_connect_errors = 10000
table_open_cache = 16384
thread_cache_size = 64
query_cache_type = 0
query_cache_size = 0
max_allowed_packet = 1M

# ************* Galera & Innodb & binlog **************
wsrep_provider = /usr/local/mysql/lib/libgalera_smm.so
wsrep_cluster_name = my_cluster
wsrep_cluster_address = gcomm://192.168.6.3,192.168.6.4,192.168.6.5,192.168.6.6
wsrep_node_name = ip63
wsrep_node_address = 192.168.6.3
wsrep_sst_method = rsync
#wsrep_sst_method = xtrabackup
#wsrep_sst_auth = "sst:sst"

log-bin = mysql-bin
binlog_format = ROW
expire_logs_days = 3

default_storage_engine = Innodb
innodb_autoinc_lock_mode = 2
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_buffer_pool_size = 2G
innodb_log_buffer_size = 10M
innodb_data_file_path = ibdata01:200M:autoextend
innodb_log_file_size = 256M
innodb_log_files_in_group = 2
innodb_open_files = 8192
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 60

# ************* slow log **************
slow_query_log
long_query_time = 3

# *************** MyIsam ***************
key_buffer_size = 128M 
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 32M

[COLOR=#FF0000]on NODE1,i startup mysql with:

/etc/init.d/mysql start --wsrep_cluster_address=gcomm://

Do you have the correct permissions on “/mysql/data” ? It should be owned by your user name in the userfield in your config. “user = mysql” “mysql” should be the owner in this case.

[root@mytest1 ~]# ls -ld /mysql
drwxr-xr-x. 4 mysql mysql 4096 Jul 15 10:55 /mysql
[root@mytest1 ~]# ls -ld /mysql/data
drwxr-xr-x 2 mysql mysql 4096 Jul 15 17:07 /mysql/data

with Percona-XtraDB-Cluster-5.5.41-37.0.tar.gz and percona-xtradb-cluster-galera-2-2.12-2682.tar.gz
join cluster is ok

node2

[root@mytest2 mysql]# service mysql start
Starting MySQL (Percona XtraDB Cluster)..State transfer in progress, setting sleep higher
. SUCCESS! 

node1.log

150716 10:45:13 [Note] WSREP: (b47c4ac9, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
150716 10:45:14 [Note] WSREP: declaring bbfe894e at tcp://192.168.6.4:4567 stable
150716 10:45:14 [Note] WSREP: Node b47c4ac9 state prim
150716 10:45:14 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
150716 10:45:14 [Note] WSREP: STATE_EXCHANGE: sent state UUID: af50045b-2b64-11e5-a8bf-6240c0a90c00
150716 10:45:14 [Note] WSREP: STATE EXCHANGE: sent state msg: af50045b-2b64-11e5-a8bf-6240c0a90c00
150716 10:45:14 [Note] WSREP: STATE EXCHANGE: got state msg: af50045b-2b64-11e5-a8bf-6240c0a90c00 from 0 (ip_6_3)
150716 10:45:14 [Note] WSREP: STATE EXCHANGE: got state msg: af50045b-2b64-11e5-a8bf-6240c0a90c00 from 1 (ip_6_4)
150716 10:45:14 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 1,
members = 1/2 (joined/total),
act_id = 0,
last_appl. = 0,
protocols = 0/4/3 (gcs/repl/appl),
group UUID = b47cf834-2b63-11e5-8b21-e7b8f418e80e
150716 10:45:14 [Note] WSREP: Flow-control interval: [23, 23]
150716 10:45:14 [Note] WSREP: New cluster view: global state: b47cf834-2b63-11e5-8b21-e7b8f418e80e:0, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
150716 10:45:14 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
150716 10:45:14 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2
150716 10:45:14 [Note] WSREP: Node 1 (ip_6_4) requested state transfer from '*any*'. Selected 0 (ip_6_3)(SYNCED) as donor.
150716 10:45:14 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 0)
150716 10:45:14 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
150716 10:45:14 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.6.4:4444/rsync_sst' --auth '(null)' --socket '/mysql/data/mysql.sock' --datadir '/mysql/data/' --defaults-file '/etc/my.cnf' --gtid 'b47cf834-2b63-11e5-8b21-e7b8f418e80e:0''
150716 10:45:14 [Note] WSREP: sst_donor_thread signaled with 0
150716 10:45:14 [Note] WSREP: Flushing tables for SST...
150716 10:45:14 [Note] WSREP: Provider paused at b47cf834-2b63-11e5-8b21-e7b8f418e80e:0 (5)
150716 10:45:14 [Note] WSREP: Tables flushed.
150716 10:45:17 [Note] WSREP: (b47c4ac9, 'tcp://0.0.0.0:4567') turning message relay requesting off
150716 10:45:21 [Note] WSREP: resuming provider at 5
150716 10:45:21 [Note] WSREP: Provider resumed.
150716 10:45:21 [Note] WSREP: 0 (ip_6_3): State transfer to 1 (ip_6_4) complete.
150716 10:45:21 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)
150716 10:45:21 [Note] WSREP: Member 0 (ip_6_3) synced with group.
150716 10:45:21 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
150716 10:45:21 [Note] WSREP: Synchronized with group, ready for connections
150716 10:45:21 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
150716 10:45:24 [Note] WSREP: 1 (ip_6_4): State transfer from 0 (ip_6_3) complete.
150716 10:45:24 [Note] WSREP: Member 1 (ip_6_4) synced with group.

i would recommend to use wsrep_sst_method=xtrabackup-v2 and install percona xtrabackup on all nodes. xtrabackup creates an innobackup..log on the joiner and donor nodes which helps in troubleshooting SST issues.

IMHO, you should check directory permissions on the joiner node not the donor node and then try SST again.

A note for anyone that stumbles upon this. I had to:

joiner: delete ibdata1, ib_logfile{0,1}, grastate.dat, galera.cache
donor: restart mysql
joiner: restart mysql (to clear all old PID/lock files automatically and start)