# PXC Cluster recovery via Xtrabackup SST space issue

**URL:** <https://forums.percona.com/t/pxc-cluster-recovery-via-xtrabackup-sst-space-issue/5747>\
**Category:** Percona XtraDB Cluster 5.x\
**Created:** [July 19, 2017, 4:54am UTC](https://forums.percona.com/t/pxc-cluster-recovery-via-xtrabackup-sst-space-issue/5747 "2017-07-19T04:54:45Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![raghupradeep](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@raghupradeep](https://forums.percona.com/u/raghupradeep)\
**Post date:** [July 19, 2017, 4:54am UTC](https://forums.percona.com/t/pxc-cluster-recovery-via-xtrabackup-sst-space-issue/5747/1 "2017-07-19T04:54:45Z")

</div>

Hello,

During recovery of a node on a 3 node cluster via SST using xtrabackupV2, met an issue with disk space.

In our environment MySQL data files are spread on two different mount points of about 500 GB each. Both disks have MySQL data only and is already using about 350Gb of space on both. So, while I was trying recovery of a node via SST, all the data of both mounts were being copied to the parent node defined by datadir and since total space of the disk is less than the actual data which is about 700 Gb the process failed.

To recovery the crashed nod,. I stopped traffic to the entire cluster and first performed the manual rsyc of data on secondary mount, changed the SST method to rsyc and issueed a MySQL start which brought the cluster node back in sync with other nodes.

Is there any way to tell xtrabackup SST to push the data files on secondary mount rather than copying it to parent directory ? If there is such an option, I can recovery the nodes without a down time.

Please let me know if there are any other options or tricks. Or is it a must that the parent directory should be big enough to hold the entire data ?

Regards,  
Raghupradeep

---

<div class="post-metadata">

**Author:** ![jrivera](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jrivera/32/13_2.png) [@jrivera](https://forums.percona.com/u/jrivera)\
**Post date:** [July 19, 2017, 8:34am UTC](https://forums.percona.com/t/pxc-cluster-recovery-via-xtrabackup-sst-space-issue/5747/2 "2017-07-19T08:34:45Z")

</div>

What you probably need here is to use the tmpdir option on SST, read the manual from this link - [url][Redirecting](https://www.percona.com/doc/percona-xtradb-cluster/LATEST/manual/xtrabackup_sst.html#tmpdir%5B/url%5D)

---

<div class="post-metadata">

**Author:** ![raghupradeep](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@raghupradeep](https://forums.percona.com/u/raghupradeep)\
**Post date:** [July 19, 2017, 1:06pm UTC](https://forums.percona.com/t/pxc-cluster-recovery-via-xtrabackup-sst-space-issue/5747/3 "2017-07-19T13:06:35Z")

</div>

I am using PXC 5.6.  
Do we have any such method for this version ? From the link, it says the tmpdir option was introduced only in 5.7.17-29.20, not sure if I will be able to upgrade the software in near future. Here also, I think this tmpdir should be as big as total data size from all mounts.

---

<div class="post-metadata">

**Author:** ![raghupradeep](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@raghupradeep](https://forums.percona.com/u/raghupradeep)\
**Post date:** [September 14, 2017, 6:50am UTC](https://forums.percona.com/t/pxc-cluster-recovery-via-xtrabackup-sst-space-issue/5747/4 "2017-09-14T06:50:31Z")

</div>

I tried with PXC 5.7 on a test server. The tmpdir option just moved the folder and files like pxc\_sst\_xxxxx, joiner\_xxxxx and xtrabackup\_galera\_info to the path under tmpdir. The .sst folder where the actual data is being staged is still under MySQL’s default datadir. Is this the intended behaviour of tmpdir ? Is that an options to set a path for .sst folder ?

---

<div class="post-metadata">

**Author:** ![Kenn\_Takara](https://avatars.discourse-cdn.com/v4/letter/k/b2d939/32.png) [@Kenn\_Takara](https://forums.percona.com/u/Kenn_Takara)\
**Post date:** [September 20, 2017, 1:54am UTC](https://forums.percona.com/t/pxc-cluster-recovery-via-xtrabackup-sst-space-issue/5747/5 "2017-09-20T01:54:12Z")

</div>

There is no option to change the joiner .sst directory. As you’ve observed, the tmpdir variable is used on the donor side and not the joiner.

One workaround would be to go into the wsrep\_sst\_xtrabackup-v2 script and change occurrences of

${DATA}/.sst  
$DATA/.sst

to

path/to/your/tmp/.sst

(I haven’t tried this myself, so careful testing is needed to make sure that things work and are cleaned up properly)

I’ve logged a bug for this and we will be looking at this for future PXC versions.

---

<div class="post-metadata">

**Author:** ![raghupradeep](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@raghupradeep](https://forums.percona.com/u/raghupradeep)\
**Post date:** [September 26, 2017, 6:50am UTC](https://forums.percona.com/t/pxc-cluster-recovery-via-xtrabackup-sst-space-issue/5747/6 "2017-09-26T06:50:32Z")

</div>

Thank you Kenn for the update.
