Can't join second node (rsync)

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

After bootstrapping nodeA, nodeB fails to start because it can’t SST
Using rsync
Tried 5.6.21 and reverted to 5.6.20 with dsame result

2014-12-14 21:41:40 9277 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 1)
2014-12-14 21:41:40 9277 [Note] WSREP: State transfer required:
Group state: cc13f472-83fa-11e4-8ea3-138a7ef719a7:1
Local state: 00000000-0000-0000-0000-000000000000:-1
2014-12-14 21:41:40 9277 [Note] WSREP: New cluster view: global state: cc13f472-83fa-11e4-8ea3-138a7ef719a7:1, view# 4: Primary, number of nodes: 2, my index: 1, protocol version 3
2014-12-14 21:41:40 9277 [Warning] WSREP: Gap in state sequence. Need state transfer.
2014-12-14 21:41:40 9277 [Note] WSREP: Running: 'wsrep_sst_rsync --role ‘joiner’ --address ‘10.10.1.62’ --auth ‘’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘9277’ ‘’ ’
2014-12-14 21:41:40 9277 [Note] WSREP: Prepared SST request: rsync|10.10.1.62:4444/rsync_sst
2014-12-14 21:41:40 9277 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-14 21:41:40 9277 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-12-14 21:41:40 9277 [Note] WSREP: Service thread queue flushed.
2014-12-14 21:41:40 9277 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
2014-12-14 21:41:40 9277 [Note] WSREP: Service thread queue flushed.
2014-12-14 21:41:40 9277 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (cc13f472-83fa-11e4-8ea3-138a7ef719a7): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
2014-12-14 21:41:40 9277 [Note] WSREP: Member 1.0 (nodeB) requested state transfer from ‘any’. Selected 0.0 (nodeA)(SYNCED) as donor.
2014-12-14 21:41:40 9277 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 1)
2014-12-14 21:41:40 9277 [Note] WSREP: Requesting state transfer: success, donor: 0
2014-12-14 21:41:40 9277 [Warning] WSREP: 0.0 (nodeA): State transfer to 1.0 (nodeB) failed: -255 (Unknown error 255)
2014-12-14 21:41:40 9277 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():722: Will never receive state. Need to abort.
2014-12-14 21:41:40 9277 [Note] WSREP: gcomm: terminating thread
2014-12-14 21:41:40 9277 [Note] WSREP: gcomm: joining thread
2014-12-14 21:41:40 9277 [Note] WSREP: gcomm: closing backend
2014-12-14 21:41:41 9277 [Note] WSREP: view(view_id(NON_PRIM,dcb11fa5,4) memb {
e50b4be3,0
} joined {
} left {
} partitioned {
dcb11fa5,0
})
2014-12-14 21:41:41 9277 [Note] WSREP: view((empty))
2014-12-14 21:41:41 9277 [Note] WSREP: gcomm: closed
2014-12-14 21:41:41 9277 [Note] WSREP: /usr/sbin/mysqld: Terminated.
141214 21:41:41 mysqld_safe mysqld from pid file /var/lib/mysql/mysqlb.pid ended
WSREP_SST: [ERROR] Parent mysqld process (PID:9277) terminated unexpectedly. (20141214 21:41:42.237)
WSREP_SST: [INFO] Joiner cleanup. (20141214 21:41:42.239)
WSREP_SST: [INFO] Joiner cleanup done. (20141214 21:41:42.746)

2

Can you post the MySQL configuration file from both nodes? Additionally, are you able to check if the IPTables/IP6Tables and SELinux is enable?

3

Here’s the oncifugration file of both nodes. Node A was started using “”/etc/init.d/mysql bootstrap-pxc""

[root@mysqla ~]# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql

Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

#manwe for cluster

wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_cluster_address=gcomm://10.10.1.61,10.10.1.62,10.10.1.63
wsrep_cluster_name=MySQLCluster
wsrep_node_name=nodeA
wsrep_node_address=10.10.1.61
wsrep_slave_threads=8
wsrep_sst_method=rsync
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

#########################################################################################################

[root@mysqlb ~]# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql

Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

#manwe for cluster

wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_cluster_address=gcomm://10.10.1.61,10.10.1.62,10.10.1.63
wsrep_cluster_name=MySQLCluster
wsrep_node_name=nodeB
wsrep_node_address=10.10.1.62
wsrep_slave_threads=8
wsrep_sst_method=rsync
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

#######################################################################

Regarding iptables:

[root@mysqlb ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all – 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT icmp – 0.0.0.0/0 0.0.0.0/0
ACCEPT all – 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
REJECT all – 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all – 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[root@mysqla ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

############################################################################

Regarding selinux I executed “setenforce 0” before the installation.

[root@mysqla ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: permissive
Mode from config file: enforcing
Policy version: 24
Policy from config file: targeted

4

Thanks to Percona’s support engineers. They solved my issue in some minutes. Changed to xtrabackup and solved a connectivity issue. Now it works.