Wsrep_sst_xtrabackup-v2 No such file or directory on ubuntu 16.04

MySQL & MariaDB Percona XtraBackup
1

Hi,

I have an older ubuntu 16.04 machine with a large mysql server.
I recently removed the mysql 5.5 packages and upgraded it to a Percona XtraDB-Cluster 5.6.

The donor is up & running but as soon as I trigger a restart on the joiner then I see this in the donor error log:

2021-10-12 10:57:20 2667 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address 'joiner ip:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '5.6.51-91.0-56'   '' --gtid '980b6909-2b22-11ec-a970-6e6331cc12b3:45848' '
2021-10-12 10:57:20 2667 [Note] WSREP: sst_donor_thread signaled with 0
2021-10-12 10:57:20 2667 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address 'joiner ip:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '5.6.51-91.0-56'   '' --gtid '980b6909-2b22-11ec-a970-6e6331cc12b3:45848'
2021-10-12 10:57:20 2667 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address 'joiner ip:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '5.6.51-91.0-56'   '' --gtid '980b6909-2b22-11ec-a970-6e6331cc12b3:45848' : 2 (No such file or directory)
2021-10-12 10:57:20 2667 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address 'joiner ip:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '5.6.51-91.0-56'   '' --gtid '980b6909-2b22-11ec-a970-6e6331cc12b3:45848'
2021-10-12 10:57:20 2667 [Warning] WSREP: 0.0 (mysql-donor-test): State transfer to 1.0 (mysql-joiner-001) failed: -2 (No such file or directory)

I have tried to do a clean install of ubuntu 16.04 with the same database imported without any issues.
the script is located in: /usr/bin/wsrep_sst_xtrabackup-v2

My config file for xtradb is:

# cat /etc/mysql/conf.d/xtradb.cnf
[mysqld]

user=mysql

# Path to Galera library
wsrep_provider=/usr/lib/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://donor ip,joiner ip

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #1 address
wsrep_node_address=donor ip

# SST method
wsrep_sst_method=xtrabackup-v2

# Cluster name
wsrep_cluster_name=donortest

# Authentication for SST method
wsrep_sst_auth="sstuser:3604cgfdvk4Ok"

I am lost in what to do next, help.

1 Like
2

Do you have SELinux enabled? Can you try running that wsrep_sst_xtrabackup-v2 script manually as shown in the log?

1 Like
3

I don’t have SELinux enabled.
When I ran the command manually it ran, but as soon as i triggered the join from the joining node it “crashed” or stopped.

But i can see I have AppArmor enabled.

$ sudo apparmor_status
apparmor module is loaded.
7 profiles are loaded.
7 profiles are in enforce mode.
   /sbin/dhclient
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/mysqld
   /usr/sbin/tcpdump
   docker-default
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode.
   /usr/sbin/mysqld (2667)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

could that be the problem?

1 Like
4

Hi @zyxep
Yes Percona XtraDB Cluster 5.6 does not work with SELinux and AppArmor security modules. Make sure these modules are disabled.

2 Likes
5

Thanks, i disabled AppArmor and restartet mysql then it started syncing right away.

2 Likes