(No such file or directory) trying to add node to cluster

I’m trying to build an xtradb cluster following the instructions in the Percona documentation but am running into an error trying to add a node to the cluster. Here is the error message from the joiner:

2018-04-04T23:10:36.778283Z 1 [Note] WSREP: Setting wsrep_ready to false
2018-04-04T23:10:36.778399Z 0 [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.130.35.11' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '28534' --binlog 'f-tier-va-db-1-bin' )
2018-04-04T23:10:36.778948Z 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.130.35.11' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '28534' --binlog 'f-tier-va-db-1-bin'
Read: '(null)'
2018-04-04T23:10:36.778968Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.130.35.11' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '28534' --binlog 'f-tier-va-db-1-bin' : 2 (No such file or directory)
2018-04-04T23:10:36.779013Z 1 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
2018-04-04T23:10:36.779021Z 1 [ERROR] Aborting

On the doner side there is not much

2018-04-04T23:04:41.909344Z 0 [Note] WSREP: (4ad1cc04, 'tcp://0.0.0.0:4567') connection established to 89ef1676 tcp://10.130.35.11:4567
2018-04-04T23:04:41.909388Z 0 [Warning] WSREP: discarding established (time wait) 89ef1676 (tcp://10.130.35.11:4567)
2018-04-04T23:04:42.840177Z 0 [Note] WSREP: cleaning up 89ef1676 (tcp://10.130.35.11:4567)

My guess is that is is saying it can’t find wsrep_sst_xtrabackup-v2 but that file is in /usr/bin. If I try to run it on the command line as the mysql user I get this result

[root&#64;f-tier-va-db-1 percona-xtradb-cluster.conf.d]# sudo su - mysql
Last login: Wed Apr 4 14:51:48 PDT 2018 on pts/0
-bash-4.2$ cd /var/lib/mysql
-bash-4.2$ wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.130.35.11' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '28534' --binlog 'f-tier-va-db-1-bin'
2018-04-04T23:25:10.779548Z WSREP_SST: [DEBUG] The xtrabackup version is 2.4.10
2018-04-04T23:25:11.029867Z WSREP_SST: [DEBUG] Streaming with xbstream
2018-04-04T23:25:11.030998Z WSREP_SST: [DEBUG] Using socat as streamer
2018-04-04T23:25:11.095480Z WSREP_SST: [DEBUG] Evaluating (&#64; Joiner-Recv-sst-info) timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr,retry=30 stdio | xbstream $xbstreameopts -x; RC=( ${PIPESTATUS[&#64;]} )
ready 10.130.35.11:4444/xtrabackup_sst//1

It looks like a good result returning ready 10.130… etc.

I don’t know how to continue debugging the problem. Other peoples issues here don’t seem to be quite the same as mine.

-bash-4.2$ cat mysqld.cnf wsrep.cnf
# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/lib/mysql/mysql.sock

[mysqld]
server-id=2
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-bin
log_slave_updates
expire_logs_days=7

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

innodb_buffer_pool_size=6G
innodb_log_file_size=128M

enforce_gtid_consistency = 1
gtid_mode = ON

# Path to Galera library
wsrep_provider=/usr/lib64/galera3/libgalera_smm.so

# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://10.131.35.11,10.130.35.11,10.132.35.11

wsrep_provider_options='gcache.size=512M'

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# Slave thread to use
wsrep_slave_threads= 8

wsrep_log_conflicts

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node IP address
wsrep_node_address=10.130.35.11
# Cluster name
wsrep_cluster_name=trans_db

#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=va-db-1

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

# SST method
wsrep_sst_method=xtrabackup-v2

#Authentication for SST method
wsrep_sst_auth="sstuser:sstuser"

[sst]
inno-apply-ops="--use-memory=2G"
wsrep_debug=1
#compressor="pigz -p 1"
#decompressor="pigz -d"

Here are my packages

-bash-4.2$ rpm -qa | grep -i percona
Percona-XtraDB-Cluster-garbd-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-shared-compat-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-client-57-5.7.21-29.26.1.el7.x86_64
percona-xtrabackup-24-2.4.10-1.el7.x86_64
Percona-XtraDB-Cluster-57-debuginfo-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-full-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-shared-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-server-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-devel-57-5.7.21-29.26.1.el7.x86_64
Percona-XtraDB-Cluster-test-57-5.7.21-29.26.1.el7.x86_64
percona-release-0.1-5.noarch

Hi there, thanks for posting. Can I get you to summarize details of your environment please as that will help the engineers to help you more efficiently? They might be able to pick out the information from what you’ve provided, but a summary always helps :slight_smile:

So … looks like you have the latest release of Percona XtraDB Cluster 5.7.21-29.26 but …
[LIST]
[]Software platform & version
[
]Any error log file entries being written?
[]Configuration file for each node
[
]A description of how you’re planning the cluster (topology, environments)
[*]Are you trying this out on a single box like this: [url]https://www.percona.com/doc/percona-xtradb-cluster/5.7/howtos/singlebox.html[/url] and if not could there be firewall issues as reported here? [url]https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/29137-new-setup-fails-to-join[/url]
[/LIST] Also maybe to be 100% clear a link to the online documentation that you are following? Thanks!

Thanks for responding Lorraine. I solved our problem.

I am using Red Hat Enterprise Linux release 7.4

I have two Amazon images. One in Oregon and Virginia. I tested ports 3306, 4444, 4567, 4568 using telnet and they are open between the two. Ping 80ms.

The problem was that selinux was enabled and set to enforcing on both machines. As soon as I disabled both my next attempt worked and my joiner synchronized with the group with no further errors.

Hi jdrozdik you’re welcome! Also, thank you for letting us know and also for recording the answer - it might well help future users.
Meanwhile, don’t hesitate to ask if you run into things in the future!