SST XtraDB Backup - Use a different .sst directory for JOINER

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Hi there !

I’m trying to restore one node of my percona cluster. Usually, everything goes well.
Unfortunatly, I’m running out of space into my datadir : XtraBackup create a “.sst” directory under datadir that fill with backup file

See log output :

WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /datas/mysql//.sst from previous state transfer. Removing
WSREP_SST: [INFO] Proceeding with SST (20170404 10:40:00.090)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20170404 10:40:00.091)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20170404 10:40:00.093)
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20170404 10:40:00.119)
2017-04-04 10:40:00 16965 [Note] WSREP: (485f6f57, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2017-04-04 10:54:16 16965 [Note] WSREP: 0.0 (node_cluster_56): State transfer to 1.0 (node_cluster_56) complete.
2017-04-04 10:54:16 16965 [Note] WSREP: Member 0.0 (node_cluster_56) synced with group.
WSREP_SST: [INFO] Compressed qpress files found (20170404 10:54:16.800)
WSREP_SST: [INFO] Decompression with 32 threads (20170404 10:54:16.806)
WSREP_SST: [INFO] Evaluating find /datas/mysql//.sst -type f -name ‘*.qp’ -printf ‘%p\n%h\n’ | xargs -n 2 qpress -T32d (20170404 10:54:16.810)
^M^M ^M^Mqpress: Disk full while writing destination file
xargs: qpress : a terminé son exécution avec le statut 255 ; arrêt abrupt.
WSREP_SST: [ERROR] Cleanup after exit with status:124 (20170404 11:13:27.825)
2017-04-04 11:13:27 16965 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.1.1.8’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16965’ ‘’ : 124 (Wrong medium type)
2017-04-04 11:13:27 16965 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2017-04-04 11:13:27 16965 [ERROR] WSREP: SST script aborted with error 124 (Wrong index given to function)
2017-04-04 11:13:27 16965 [ERROR] WSREP: SST failed: 124 (Wrong medium type)
2017-04-04 11:13:27 16965 [ERROR] Aborting

Unfortunatly (again), you can’t not simply change the .sst directoty for the joiner in wsrep_sst_xtrabackup-v2 script.
So I tried to “hack” it to create my own .sst directory who is symbolic link to antoher partition.

rm -rf /datas/mysql/*
rm -rf /datas/mysql/.sst
ln -s /var/tmp/xtrabackup /datas/mysql/.sst
chown -R mysql:mysql /datas/mysql
chown -R mysql:mysql /var/tmp/xtrabackup

It’s works until innobackupex state :

WSREP_SST: [INFO] Preparing the backup at /datas/mysql//.sst (20170405 10:52:46.812)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20170405 10:52:46.816)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20170405 10:52:46.861)
2017-04-05 10:52:46 30211 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.118.8’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘30211’ ‘’ : 1 (Operation
not permitted)
2017-04-05 10:52:46 30211 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2017-04-05 10:52:46 30211 [ERROR] WSREP: SST script aborted with error 1 (Operation not permitted)
2017-04-05 10:52:46 30211 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2017-04-05 10:52:46 30211 [ERROR] Aborting

More details in innobackup.prepare.log :

170405 11:32:21 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints “completed OK!”.

innobackupex version 2.3.6 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 7686bfc)
xtrabackup: cd to /datas/mysql//.sst
xtrabackup: This target seems to be not prepared yet.
2017-04-05 11:32:21 7f74d64907e0 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
2017-04-05 11:32:21 7f74d64907e0 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Fatal error: cannot find ./xtrabackup_logfile.
xtrabackup: Error: xtrabackup_init_temp_log() failed.

Another way is to avoid SST :
[url]https://severalnines.com/blog/how-avoid-sst-when-adding-new-node-galera-cluster[/url]

It not seems to complex but in production envireonnement, I’m not confortable with it.

Has anyone faced that issue ? Have other suggestions ?

Regards,

2

I see you are probably using compression for SST.
Can you disable compression and see if it works for you?

3

Hi vadimtk,

You are right, I’m using compression for SST. Thank for your suggestion, I gonna try it.

As far I have a space issue, I have also tried to set my datadir in another partition with higher space. That action bring another issue :

Maybe an error from DONOR side ?

4

Hi,

At a stage when a donor was already selected, yes, it is important to check also it’s corresponding error log entries. If produced, check also the innobackup.backup.log on donor.

5

Hi przemek,

I can see the same error on the DONOR (the cluster master is always selected as :frowning: ) . Unfortunaltly, there are no more details

I am currently re-installing Percona XtraBD for the damaged node and I will do a uncompress SST.

The file innobackup.backup.log has been overwritten

6

Hi,

This looks like SELInux or Apparmor issue to me. The log says the sst script wasn’t even executed on the donor. Please check the corresponding security logs.

7

Hi przemek,

I’m under CentOS 6 and SELInux is disable.
I have finally re-installed PXC on this node and I have succed to joined a QA clucter with it.
We gonna re-try with PROD env now.

8

Hi,

With PXC 5.6.35-20-3, you can define now the tmpdir for the backup durring the SST

[URL][PXC-782] PXC xtrabackup-v2 option should use tmpdir option - Percona JIRA

[URL]https://www.percona.com/blog/2017/04/19/percona-xtradb-cluster-5-6-35-26-20-3-now-available/[/URL]

9

Hi,

Fianlly, We have purged a huge database and we retrived space. We succed to restore our lost node.
Thank to the community for his help

Best regards,

Harold