Not the answer you need?
Register and ask your own question!

PXC Cluster recovery via Xtrabackup SST space issue

raghupradeepraghupradeep ContributorCurrent User Role Beginner
Hello,

During recovery of a node on a 3 node cluster via SST using xtrabackupV2, met an issue with disk space.

In our environment MySQL data files are spread on two different mount points of about 500 GB each. Both disks have MySQL data only and is already using about 350Gb of space on both. So, while I was trying recovery of a node via SST, all the data of both mounts were being copied to the parent node defined by datadir and since total space of the disk is less than the actual data which is about 700 Gb the process failed.

To recovery the crashed nod,. I stopped traffic to the entire cluster and first performed the manual rsyc of data on secondary mount, changed the SST method to rsyc and issueed a MySQL start which brought the cluster node back in sync with other nodes.

Is there any way to tell xtrabackup SST to push the data files on secondary mount rather than copying it to parent directory ? If there is such an option, I can recovery the nodes without a down time.

Please let me know if there are any other options or tricks. Or is it a must that the parent directory should be big enough to hold the entire data ?

Regards,
Raghupradeep

Comments

  • jriverajrivera Percona Support Engineer Percona Staff Role
    What you probably need here is to use the tmpdir option on SST, read the manual from this link - https://www.percona.com/doc/percona-xtradb-cluster/LATEST/manual/xtrabackup_sst.html#tmpdir
  • raghupradeepraghupradeep Contributor Current User Role Beginner
    I am using PXC 5.6.
    Do we have any such method for this version ? From the link, it says the tmpdir option was introduced only in 5.7.17-29.20, not sure if I will be able to upgrade the software in near future. Here also, I think this tmpdir should be as big as total data size from all mounts.
  • raghupradeepraghupradeep Contributor Current User Role Beginner
    I tried with PXC 5.7 on a test server. The tmpdir option just moved the folder and files like pxc_sst_xxxxx, joiner_xxxxx and xtrabackup_galera_info to the path under tmpdir. The .sst folder where the actual data is being staged is still under MySQL's default datadir. Is this the intended behaviour of tmpdir ? Is that an options to set a path for .sst folder ?
  • Kenn TakaraKenn Takara Percona Percona Staff Role
    There is no option to change the joiner .sst directory. As you've observed, the tmpdir variable is used on the donor side and not the joiner.

    One workaround would be to go into the wsrep_sst_xtrabackup-v2 script and change occurrences of

    ${DATA}/.sst
    $DATA/.sst

    to

    path/to/your/tmp/.sst

    (I haven't tried this myself, so careful testing is needed to make sure that things work and are cleaned up properly)

    I've logged a bug for this and we will be looking at this for future PXC versions.
  • raghupradeepraghupradeep Contributor Current User Role Beginner
    Thank you Kenn for the update.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.