# SST XtraDB Backup - Use a different .sst directory for JOINER **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [April 7, 2017, 4:01am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531 "2017-04-07T04:01:06Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Harold](https://avatars.discourse-cdn.com/v4/letter/h/8491ac/32.png) [@Harold](https://forums.percona.com/u/Harold)\ **Post date:** [April 7, 2017, 4:01am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/1 "2017-04-07T04:01:06Z")
Hi there ! I’m trying to restore one node of my percona cluster. Usually, everything goes well. Unfortunatly, I’m running out of space into my datadir : XtraBackup create a “.sst” directory under datadir that fill with backup file See log output : * * * ## WSREP\_SST: [INFO] WARNING: Stale temporary SST directory: /datas/mysql//.sst from previous state transfer. Removing WSREP\_SST: [INFO] Proceeding with SST (20170404 10:40:00.090) WSREP\_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20170404 10:40:00.091) WSREP\_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20170404 10:40:00.093) WSREP\_SST: [INFO] Waiting for SST streaming to complete! (20170404 10:40:00.119) 2017-04-04 10:40:00 16965 [Note] WSREP: (485f6f57, ‘tcp://0.0.0.0:4567’) turning message relay requesting off 2017-04-04 10:54:16 16965 [Note] WSREP: 0.0 (node\_cluster\_56): State transfer to 1.0 (node\_cluster\_56) complete. 2017-04-04 10:54:16 16965 [Note] WSREP: Member 0.0 (node\_cluster\_56) synced with group. WSREP\_SST: [INFO] Compressed qpress files found (20170404 10:54:16.800) WSREP\_SST: [INFO] Decompression with 32 threads (20170404 10:54:16.806) WSREP\_SST: [INFO] Evaluating find /datas/mysql//.sst -type f -name ‘\*.qp’ -printf ‘%p\n%h\n’ | xargs -n 2 qpress -T32d (20170404 10:54:16.810) ^M^M ^M^Mqpress: Disk full while writing destination file xargs: qpress : a terminé son exécution avec le statut 255 ; arrêt abrupt. WSREP\_SST: [ERROR] Cleanup after exit with status:124 (20170404 11:13:27.825) 2017-04-04 11:13:27 16965 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘10.1.1.8’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16965’ ‘’ : 124 (Wrong medium type) 2017-04-04 11:13:27 16965 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 2017-04-04 11:13:27 16965 [ERROR] WSREP: SST script aborted with error 124 (Wrong index given to function) 2017-04-04 11:13:27 16965 [ERROR] WSREP: SST failed: 124 (Wrong medium type) 2017-04-04 11:13:27 16965 [ERROR] Aborting Unfortunatly (again), you can’t not simply change the .sst directoty for the joiner in wsrep\_sst\_xtrabackup-v2 script. So I tried to “hack” it to create my own .sst directory who is symbolic link to antoher partition. * * * ## rm -rf /datas/mysql/\* rm -rf /datas/mysql/.sst ln -s /var/tmp/xtrabackup /datas/mysql/.sst chown -R mysql:mysql /datas/mysql chown -R mysql:mysql /var/tmp/xtrabackup It’s works until innobackupex state : * * * ## WSREP\_SST: [INFO] Preparing the backup at /datas/mysql//.sst (20170405 10:52:46.812) WSREP\_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &\>${DATA}/innobackup.prepare.log (20170405 10:52:46.816) WSREP\_SST: [ERROR] Cleanup after exit with status:1 (20170405 10:52:46.861) 2017-04-05 10:52:46 30211 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.118.8’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘30211’ ‘’ : 1 (Operation not permitted) 2017-04-05 10:52:46 30211 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 2017-04-05 10:52:46 30211 [ERROR] WSREP: SST script aborted with error 1 (Operation not permitted) 2017-04-05 10:52:46 30211 [ERROR] WSREP: SST failed: 1 (Operation not permitted) 2017-04-05 10:52:46 30211 [ERROR] Aborting More details in innobackup.prepare.log : * * * 170405 11:32:21 innobackupex: Starting the apply-log operation IMPORTANT: Please check that the apply-log run completes successfully. At the end of a successful apply-log run innobackupex prints “completed OK!”. ## innobackupex version 2.3.6 based on MySQL server 5.6.24 Linux (x86\_64) (revision id: 7686bfc) xtrabackup: cd to /datas/mysql//.sst xtrabackup: This target seems to be not prepared yet. 2017-04-05 11:32:21 7f74d64907e0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. xtrabackup: Warning: cannot open ./xtrabackup\_logfile. will try to find. 2017-04-05 11:32:21 7f74d64907e0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. xtrabackup: Fatal error: cannot find ./xtrabackup\_logfile. xtrabackup: Error: xtrabackup\_init\_temp\_log() failed. Another way is to avoid SST : [https://severalnines.com/blog/how-avoid-sst-when-adding-new-node-galera-cluster](https://severalnines.com/blog/how-avoid-sst-when-adding-new-node-galera-cluster) It not seems to complex but in production envireonnement, I’m not confortable with it. Has anyone faced that issue ? Have other suggestions ? Regards, ---
**Author:** ![vadimtk](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadimtk/32/9528_2.png) [@vadimtk](https://forums.percona.com/u/vadimtk)\ **Post date:** [April 7, 2017, 7:17pm UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/2 "2017-04-07T19:17:16Z")
I see you are probably using compression for SST. Can you disable compression and see if it works for you? ---
**Author:** ![Harold](https://avatars.discourse-cdn.com/v4/letter/h/8491ac/32.png) [@Harold](https://forums.percona.com/u/Harold)\ **Post date:** [April 12, 2017, 1:45am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/3 "2017-04-12T01:45:58Z")
Hi vadimtk, You are right, I’m using compression for SST. Thank for your suggestion, I gonna try it. As far I have a space issue, I have also tried to set my datadir in another partition with higher space. That action bring another issue : > [@](#): > > 2017-04-11 10:59:19 9082 [Warning] WSREP: Gap in state sequence. Need state transfer. > 2017-04-11 10:59:19 9082 [Note] WSREP: Running: 'wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.118.8’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘9082’ ‘’ ’ > WSREP\_SST: [INFO] Streaming with xbstream (20170411 10:59:19.756) > WSREP\_SST: [INFO] Using socat as streamer (20170411 10:59:19.760) > WSREP\_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20170411 10:59:19.820) > 2017-04-11 10:59:19 9082 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.118.8:4444/xtrabackup\_sst//1 > 2017-04-11 10:59:19 9082 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. > 2017-04-11 10:59:19 9082 [Note] WSREP: REPL Protocols: 7 (3, 2) > 2017-04-11 10:59:19 9082 [Note] WSREP: Assign initial position for certification: 3332615043, protocol version: 3 > 2017-04-11 10:59:19 9082 [Note] WSREP: Service thread queue flushed. > 2017-04-11 10:59:19 9082 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (ca9ef4d7-1407-11e5-9d79-a6e0476b8926): 1 (Operation not permitted) > at galera/src/replicator\_str.cpp:prepare\_for\_IST():535. IST will be unavailable. > 2017-04-11 10:59:19 9082 [Note] WSREP: Member 1.0 (node\_cluster\_56) requested state transfer from ‘_any_’. Selected 0.0 (node\_cluster\_56)(SYNCED) as donor. > 2017-04-11 10:59:19 9082 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 3332615054) > 2017-04-11 10:59:19 9082 [Note] WSREP: Requesting state transfer: success, donor: 0 > 2017-04-11 10:59:19 9082 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) → new(ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332615043) > 2017-04-11 10:59:20 9082 [Warning] WSREP: 0.0 (node\_cluster\_56): State transfer to 1.0 (node\_cluster\_56) failed: -1 (Operation not permitted) > 2017-04-11 10:59:20 9082 [ERROR] WSREP: gcs/src/gcs\_group.cpp:gcs\_group\_handle\_join\_msg():765: Will never receive state. Need to abort. > 2017-04-11 10:59:20 9082 [Note] WSREP: gcomm: terminating thread > 2017-04-11 10:59:20 9082 [Note] WSREP: gcomm: joining thread > 2017-04-11 10:59:20 9082 [Note] WSREP: gcomm: closing backend > 2017-04-11 10:59:20 9082 [Note] WSREP: view(view\_id(NON\_PRIM,11238dec,716) memb { > 258aefce,0 > } joined { > } left { > } partitioned { > 11238dec,0 > 7d14391a,0 > ad7f0066,0 > c716ce0b,0 > }) > 2017-04-11 10:59:20 9082 [Note] WSREP: view((empty)) > 2017-04-11 10:59:20 9082 [Note] WSREP: gcomm: closed > 2017-04-11 10:59:20 9082 [Note] WSREP: /usr/sbin/mysqld: Terminated. > 170411 10:59:20 mysqld\_safe mysqld from pid file /datas/mysql/p2-mysql08.pid ended Maybe an error from DONOR side ? ---
**Author:** ![przemek](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/przemek/32/3_2.png) [@przemek](https://forums.percona.com/u/przemek)\ **Post date:** [April 12, 2017, 4:15am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/4 "2017-04-12T04:15:10Z")
Hi, At a stage when a donor was already selected, yes, it is important to check also it’s corresponding error log entries. If produced, check also the innobackup.backup.log on donor. ---
**Author:** ![Harold](https://avatars.discourse-cdn.com/v4/letter/h/8491ac/32.png) [@Harold](https://forums.percona.com/u/Harold)\ **Post date:** [April 12, 2017, 6:43am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/5 "2017-04-12T06:43:59Z")
Hi przemek, I can see the same error on the DONOR (the cluster master is always selected as ☹ ) . Unfortunaltly, there are no more details > [@](#): > > 2017-04-11 10:11:22 27232 [Note] WSREP: Member 1.0 (node\_cluster\_56) requested state transfer from ‘_any_’. Selected 0.0 (node\_cluster\_56)(SYNCED) as donor. > 2017-04-11 10:11:22 27232 [Note] WSREP: Shifting SYNCED → DONOR/DESYNCED (TO: 3332581902) > 2017-04-11 10:11:22 27232 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. > 2017-04-11 10:11:22 27232 [Note] WSREP: Running: ‘wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.118.8:4444/xtrabackup\_sst//1’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332581902’’ > 2017-04-11 10:11:22 27232 [Note] WSREP: sst\_donor\_thread signaled with 0 > 2017-04-11 10:11:23 27232 [ERROR] WSREP: Failed to read from: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.118.8:4444/xtrabackup\_sst//1’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ > ‘’ --gtid ‘ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332581902’ > 2017-04-11 10:11:23 27232 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.118.8:4444/xtrabackup\_sst//1’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332581902’: 1 (Operation not permitted) > 2017-04-11 10:11:23 27232 [ERROR] WSREP: Command did not run: wsrep\_sst\_xtrabackup-v2 --role ‘donor’ --address ‘192.168.118.8:4444/xtrabackup\_sst//1’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/datas/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ > ‘’ --gtid ‘ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332581902’ > 2017-04-11 10:11:23 27232 [Warning] WSREP: 0.0 (node\_cluster\_56): State transfer to 1.0 (node\_cluster\_56) failed: -1 (Operation not permitted) > 2017-04-11 10:11:23 27232 [Note] WSREP: Shifting DONOR/DESYNCED → JOINED (TO: 3332581905) > 2017-04-11 10:11:23 27232 [Note] WSREP: Member 0.0 (node\_cluster\_56) synced with group. I am currently re-installing Percona XtraBD for the damaged node and I will do a uncompress SST. The file innobackup.backup.log has been overwritten ---
**Author:** ![przemek](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/przemek/32/3_2.png) [@przemek](https://forums.percona.com/u/przemek)\ **Post date:** [April 18, 2017, 5:17am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/6 "2017-04-18T05:17:00Z")
Hi, This looks like SELInux or Apparmor issue to me. The log says the sst script wasn’t even executed on the donor. Please check the corresponding security logs. ---
**Author:** ![Harold](https://avatars.discourse-cdn.com/v4/letter/h/8491ac/32.png) [@Harold](https://forums.percona.com/u/Harold)\ **Post date:** [April 19, 2017, 9:02am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/7 "2017-04-19T09:02:30Z")
Hi przemek, I’m under CentOS 6 and SELInux is disable. I have finally re-installed PXC on this node and I have succed to joined a QA clucter with it. We gonna re-try with PROD env now. ---
**Author:** ![Harold](https://avatars.discourse-cdn.com/v4/letter/h/8491ac/32.png) [@Harold](https://forums.percona.com/u/Harold)\ **Post date:** [April 20, 2017, 1:26am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/8 "2017-04-20T01:26:35Z")
Hi, With PXC 5.6.35-20-3, you can define now the tmpdir for the backup durring the SST [URL][[PXC-782] PXC xtrabackup-v2 option should use tmpdir option - Percona JIRA](https://jira.percona.com/browse/PXC-782%5B/URL%5D) [URL][https://www.percona.com/blog/2017/04/19/percona-xtradb-cluster-5-6-35-26-20-3-now-available/[/URL]](https://www.percona.com/blog/2017/04/19/percona-xtradb-cluster-5-6-35-26-20-3-now-available/%5B/URL%5D) ---
**Author:** ![Harold](https://avatars.discourse-cdn.com/v4/letter/h/8491ac/32.png) [@Harold](https://forums.percona.com/u/Harold)\ **Post date:** [June 26, 2017, 3:06am UTC](https://forums.percona.com/t/sst-xtradb-backup-use-a-different-sst-directory-for-joiner/5531/9 "2017-06-26T03:06:42Z")
Hi, Fianlly, We have purged a huge database and we retrived space. We succed to restore our lost node. Thank to the community for his help Best regards, Harold