I’m trying to build an xtradb cluster following the instructions in the Percona documentation but am running into an error trying to add a node to the cluster. Here is the error message from the joiner:
2018-04-04T23:10:36.778283Z 1 [Note] WSREP: Setting wsrep_ready to false
2018-04-04T23:10:36.778399Z 0 [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.130.35.11' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '28534' --binlog 'f-tier-va-db-1-bin' )
2018-04-04T23:10:36.778948Z 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.130.35.11' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '28534' --binlog 'f-tier-va-db-1-bin'
Read: '(null)'
2018-04-04T23:10:36.778968Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.130.35.11' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '28534' --binlog 'f-tier-va-db-1-bin' : 2 (No such file or directory)
2018-04-04T23:10:36.779013Z 1 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
2018-04-04T23:10:36.779021Z 1 [ERROR] Aborting
On the doner side there is not much
2018-04-04T23:04:41.909344Z 0 [Note] WSREP: (4ad1cc04, 'tcp://0.0.0.0:4567') connection established to 89ef1676 tcp://10.130.35.11:4567
2018-04-04T23:04:41.909388Z 0 [Warning] WSREP: discarding established (time wait) 89ef1676 (tcp://10.130.35.11:4567)
2018-04-04T23:04:42.840177Z 0 [Note] WSREP: cleaning up 89ef1676 (tcp://10.130.35.11:4567)
My guess is that is is saying it can’t find wsrep_sst_xtrabackup-v2 but that file is in /usr/bin. If I try to run it on the command line as the mysql user I get this result
[root@f-tier-va-db-1 percona-xtradb-cluster.conf.d]# sudo su - mysql
Last login: Wed Apr 4 14:51:48 PDT 2018 on pts/0
-bash-4.2$ cd /var/lib/mysql
-bash-4.2$ wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.130.35.11' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '28534' --binlog 'f-tier-va-db-1-bin'
2018-04-04T23:25:10.779548Z WSREP_SST: [DEBUG] The xtrabackup version is 2.4.10
2018-04-04T23:25:11.029867Z WSREP_SST: [DEBUG] Streaming with xbstream
2018-04-04T23:25:11.030998Z WSREP_SST: [DEBUG] Using socat as streamer
2018-04-04T23:25:11.095480Z WSREP_SST: [DEBUG] Evaluating (@ Joiner-Recv-sst-info) timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr,retry=30 stdio | xbstream $xbstreameopts -x; RC=( ${PIPESTATUS[@]} )
ready 10.130.35.11:4444/xtrabackup_sst//1
It looks like a good result returning ready 10.130… etc.
I don’t know how to continue debugging the problem. Other peoples issues here don’t seem to be quite the same as mine.
-bash-4.2$ cat mysqld.cnf wsrep.cnf
# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/lib/mysql/mysql.sock
[mysqld]
server-id=2
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-bin
log_slave_updates
expire_logs_days=7
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
innodb_buffer_pool_size=6G
innodb_log_file_size=128M
enforce_gtid_consistency = 1
gtid_mode = ON
# Path to Galera library
wsrep_provider=/usr/lib64/galera3/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://10.131.35.11,10.130.35.11,10.132.35.11
wsrep_provider_options='gcache.size=512M'
# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB
# Slave thread to use
wsrep_slave_threads= 8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2
# Node IP address
wsrep_node_address=10.130.35.11
# Cluster name
wsrep_cluster_name=trans_db
#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=va-db-1
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING
# SST method
wsrep_sst_method=xtrabackup-v2
#Authentication for SST method
wsrep_sst_auth="sstuser:sstuser"
[sst]
inno-apply-ops="--use-memory=2G"
wsrep_debug=1
#compressor="pigz -p 1"
#decompressor="pigz -d"
Here are my packages
-bash-4.2$ rpm -qa | grep -i percona
Percona-XtraDB-Cluster-garbd-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-shared-compat-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-client-57-5.7.21-29.26.1.el7.x86_64
percona-xtrabackup-24-2.4.10-1.el7.x86_64
Percona-XtraDB-Cluster-57-debuginfo-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-full-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-shared-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-server-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-devel-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-test-57-5.7.21-29.26.1.el7.x86_64
percona-release-0.1-5.noarch
