PXC Cluster recovery via Xtrabackup SST space issue

[raghupradeep]

Hello,

During recovery of a node on a 3 node cluster via SST using xtrabackupV2, met an issue with disk space.

In our environment MySQL data files are spread on two different mount points of about 500 GB each. Both disks have MySQL data only and is already using about 350Gb of space on both. So, while I was trying recovery of a node via SST, all the data of both mounts were being copied to the parent node defined by datadir and since total space of the disk is less than the actual data which is about 700 Gb the process failed.

To recovery the crashed nod,. I stopped traffic to the entire cluster and first performed the manual rsyc of data on secondary mount, changed the SST method to rsyc and issueed a MySQL start which brought the cluster node back in sync with other nodes.

Is there any way to tell xtrabackup SST to push the data files on secondary mount rather than copying it to parent directory ? If there is such an option, I can recovery the nodes without a down time.

Please let me know if there are any other options or tricks. Or is it a must that the parent directory should be big enough to hold the entire data ?

Regards,
Raghupradeep

[jrivera]

What you probably need here is to use the tmpdir option on SST, read the manual from this link - https://www.percona.com/doc/percona-xtradb-cluster/LATEST/manual/xtrabackup_sst.html#tmpdir

[raghupradeep]

I am using PXC 5.6.
Do we have any such method for this version ? From the link, it says the tmpdir option was introduced only in 5.7.17-29.20, not sure if I will be able to upgrade the software in near future. Here also, I think this tmpdir should be as big as total data size from all mounts.

[raghupradeep]

I tried with PXC 5.7 on a test server. The tmpdir option just moved the folder and files like pxc_sst_xxxxx, joiner_xxxxx and xtrabackup_galera_info to the path under tmpdir. The .sst folder where the actual data is being staged is still under MySQL’s default datadir. Is this the intended behaviour of tmpdir ? Is that an options to set a path for .sst folder ?

[Kenn_Takara]

There is no option to change the joiner .sst directory. As you’ve observed, the tmpdir variable is used on the donor side and not the joiner.

One workaround would be to go into the wsrep_sst_xtrabackup-v2 script and change occurrences of

${DATA}/.sst
$DATA/.sst

to

path/to/your/tmp/.sst

(I haven’t tried this myself, so careful testing is needed to make sure that things work and are cleaned up properly)

I’ve logged a bug for this and we will be looking at this for future PXC versions.

[raghupradeep]

Thank you Kenn for the update.