SST XtraDB Backup - Use a different .sst directory for JOINER

Hi there !

I’m trying to restore one node of my percona cluster. Usually, everything goes well.
Unfortunatly, I’m running out of space into my datadir : XtraBackup create a “.sst” directory under datadir that fill with backup file

See log output :


WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /datas/mysql//.sst from previous state transfer. Removing
WSREP_SST: [INFO] Proceeding with SST (20170404 10:40:00.090)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20170404 10:40:00.091)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20170404 10:40:00.093)
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20170404 10:40:00.119)
2017-04-04 10:40:00 16965 [Note] WSREP: (485f6f57, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2017-04-04 10:54:16 16965 [Note] WSREP: 0.0 (node_cluster_56): State transfer to 1.0 (node_cluster_56) complete.
2017-04-04 10:54:16 16965 [Note] WSREP: Member 0.0 (node_cluster_56) synced with group.
WSREP_SST: [INFO] Compressed qpress files found (20170404 10:54:16.800)
WSREP_SST: [INFO] Decompression with 32 threads (20170404 10:54:16.806)
WSREP_SST: [INFO] Evaluating find /datas/mysql//.sst -type f -name ‘*.qp’ -printf ‘%p\n%h\n’ | xargs -n 2 qpress -T32d (20170404 10:54:16.810)
^M^M ^M^Mqpress: Disk full while writing destination file
xargs: qpress : a terminé son exécution avec le statut 255 ; arrêt abrupt.
WSREP_SST: [ERROR] Cleanup after exit with status:124 (20170404 11:13:27.825)
2017-04-04 11:13:27 16965 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.1.1.8’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16965’ ‘’ : 124 (Wrong medium type)
2017-04-04 11:13:27 16965 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2017-04-04 11:13:27 16965 [ERROR] WSREP: SST script aborted with error 124 (Wrong index given to function)
2017-04-04 11:13:27 16965 [ERROR] WSREP: SST failed: 124 (Wrong medium type)
2017-04-04 11:13:27 16965 [ERROR] Aborting

Unfortunatly (again), you can’t not simply change the .sst directoty for the joiner in wsrep_sst_xtrabackup-v2 script.
So I tried to “hack” it to create my own .sst directory who is symbolic link to antoher partition.


rm -rf /datas/mysql/*
rm -rf /datas/mysql/.sst
ln -s /var/tmp/xtrabackup /datas/mysql/.sst
chown -R mysql:mysql /datas/mysql
chown -R mysql:mysql /var/tmp/xtrabackup

It’s works until innobackupex state :


WSREP_SST: [INFO] Preparing the backup at /datas/mysql//.sst (20170405 10:52:46.812)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20170405 10:52:46.816)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20170405 10:52:46.861)
2017-04-05 10:52:46 30211 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.118.8’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘30211’ ‘’ : 1 (Operation
not permitted)
2017-04-05 10:52:46 30211 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2017-04-05 10:52:46 30211 [ERROR] WSREP: SST script aborted with error 1 (Operation not permitted)
2017-04-05 10:52:46 30211 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2017-04-05 10:52:46 30211 [ERROR] Aborting

More details in innobackup.prepare.log :


170405 11:32:21 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints “completed OK!”.

innobackupex version 2.3.6 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 7686bfc)
xtrabackup: cd to /datas/mysql//.sst
xtrabackup: This target seems to be not prepared yet.
2017-04-05 11:32:21 7f74d64907e0 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
2017-04-05 11:32:21 7f74d64907e0 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Fatal error: cannot find ./xtrabackup_logfile.
xtrabackup: Error: xtrabackup_init_temp_log() failed.

Another way is to avoid SST :
https://severalnines.com/blog/how-avoid-sst-when-adding-new-node-galera-cluster

It not seems to complex but in production envireonnement, I’m not confortable with it.

Has anyone faced that issue ? Have other suggestions ?

Regards,

I see you are probably using compression for SST.
Can you disable compression and see if it works for you?

Hi vadimtk,

You are right, I’m using compression for SST. Thank for your suggestion, I gonna try it.

As far I have a space issue, I have also tried to set my datadir in another partition with higher space. That action bring another issue :

Maybe an error from DONOR side ?

Hi,

At a stage when a donor was already selected, yes, it is important to check also it’s corresponding error log entries. If produced, check also the innobackup.backup.log on donor.

Hi przemek,

I can see the same error on the DONOR (the cluster master is always selected as :frowning: ) . Unfortunaltly, there are no more details

I am currently re-installing Percona XtraBD for the damaged node and I will do a uncompress SST.

The file innobackup.backup.log has been overwritten

Hi,

This looks like SELInux or Apparmor issue to me. The log says the sst script wasn’t even executed on the donor. Please check the corresponding security logs.

Hi przemek,

I’m under CentOS 6 and SELInux is disable.
I have finally re-installed PXC on this node and I have succed to joined a QA clucter with it.
We gonna re-try with PROD env now.

Hi,

With PXC 5.6.35-20-3, you can define now the tmpdir for the backup durring the SST

https://jira.percona.com/browse/PXC-782

https://www.percona.com/blog/2017/04/19/percona-xtradb-cluster-5-6-35-26-20-3-now-available/

Hi,

Fianlly, We have purged a huge database and we retrived space. We succed to restore our lost node.
Thank to the community for his help

Best regards,

Harold