During recovery of a node on a 3 node cluster via SST using xtrabackupV2, met an issue with disk space.
In our environment MySQL data files are spread on two different mount points of about 500 GB each. Both disks have MySQL data only and is already using about 350Gb of space on both. So, while I was trying recovery of a node via SST, all the data of both mounts were being copied to the parent node defined by datadir and since total space of the disk is less than the actual data which is about 700 Gb the process failed.
To recovery the crashed nod,. I stopped traffic to the entire cluster and first performed the manual rsyc of data on secondary mount, changed the SST method to rsyc and issueed a MySQL start which brought the cluster node back in sync with other nodes.
Is there any way to tell xtrabackup SST to push the data files on secondary mount rather than copying it to parent directory ? If there is such an option, I can recovery the nodes without a down time.
Please let me know if there are any other options or tricks. Or is it a must that the parent directory should be big enough to hold the entire data ?