New setup fails to join

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Hello,

I recently deployed a new xtradb cluster setup on Ubuntu 14.04 however they are fails to join. From the logs it seems something is missing from the server however there is nothing specifics in logs and I couldn’t locate whats missing. I have installed libssl,rsync etc…

2014-12-28 08:09:09 4142 [Note] WSREP: New cluster view: global state: 2cda4783-8e85-11e4-b8e9-a278e296bb8d:9543, view# 16: Primary, number of nodes: 2, my index: 1, protocol version 3
2014-12-28 08:09:09 4142 [Warning] WSREP: Gap in state sequence. Need state transfer.
2014-12-28 08:09:09 4142 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.2.2.2’ --auth ‘sstuser:Xgbx6X’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘4142’ ‘’ ’
2014-12-28 08:09:09 4142 [ERROR] WSREP: Failed to read ‘ready ’ from: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.2.2.2’ --auth ‘sstuser:Xgbx6X’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘4142’ ‘’
Read: ‘(null)’
2014-12-28 08:09:09 4142 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.2.2.2’ --auth ‘sstuser:Xgbx6X’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘4142’ ‘’ : 2 (No such file or directory)
2014-12-28 08:09:09 4142 [ERROR] WSREP: Failed to prepare for ‘xtrabackup-v2’ SST. Unrecoverable.
2014-12-28 08:09:09 4142 [ERROR] Aborting

Both nodes has the same versions.

root@mp1:/var/lib/mysql# mysql --version
mysql Ver 14.14 Distrib 5.6.21-70.1, for debian-linux-gnu (x86_64) using EditLine wrapper
root@mp1:/var/lib/mysql#

They are running xtradb version 5.6 and both has wsrep_sst_xtrabackup-v2 as well. Please advise.

Thanks

2

Is there on the donor’s mysql the user and password configured as the value of wsrep_sst_auth (I believe so as the sst process completed successfully)? Can you check if any firewalls are restricting access on used ports (UFW/AppArmor)?

3

Yes the user/pass is configured correctly. AppArmor has been removed however there is an active firewall configured on the donor but the necessary ports (3306,4444,4567,4568) are enabled.

4

Hi,

Ok. It seems it wasn’t enough just to disable and remove AppArmor but we needed an actual reboot too. Now the issue is as follows:

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20141228 13:54:19.016)
touch: cannot touch ‘/var/lib/mysql//sst_in_progress’: Permission denied
2014-12-28 13:54:19 19751 [ERROR] WSREP: Failed to read ‘ready ’ from: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.2.2.2’ --auth ‘sstuser:Xgbx6X’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘19751’ ‘’
Read: ‘(null)’
2014-12-28 13:54:19 19751 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.2.2.2’ --auth ‘sstuser:Xgbx6X’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘19751’ ‘’ : 1 (Operation not permitted)

the folder is owned by mysql on both servers:

drwx------ 10 mysql mysql 4096 Dec 28 13:52 mysql

The mysql is running as mysql user defined in /etc/mysql/my.cnf

[mysqld]

datadir=/var/lib/mysql
user=mysql

Any advise?

Thanks

5

Okey removing the sst_in_progress file from the slave solved the issue.

6

It’s good to hear from you that everything is running well.