Innobackupex / xbstream (--parallel)

MySQL & MariaDB Percona XtraBackup
1

Hi everyone,

I’ve setup a demo and the procedure is to retrieve the backup and automate the restore to validate that my backup is right and accesible.

I want to make the process as fast as possible, so I tried to use the –parallel (and use memory for example on –apply-log). My question is on how to calculate the appropriate value of –parallel option? Is it based on CPU usage? Currently on a VPS with 6 cores I get the following using htop, while using –parallel=32.

htop
htop1636×807 21.5 KB

Thanks in advance!

2

Hi @Apostolos_Karavias

–parallel in the context of xbstream is the number of parallel files that can be processed at the same time.
This will be IO-bound. However, xbstream is reading data from STDIN, which means the parallel throughput comes from the counterpart (most likely xbcloud).

You need xbcloud and our network capable of retrieving multiple file chunks in parallel and sending them to xbstream in order for parallel to give you any advantage.

3

Hi @Marcelo_Altmann thanks for replying. Currently the xbstream file is located locally on the machine, so I guess it’s just IO limited from the disk perspective? Meaning it’s safe to increase the number and check the results in terms of speed.

4

Hi @Apostolos_Karavias ,

Correct. In your case, you might see a lot of benefits from parallel if your disk is slow in writing. Each thread on xbstream does the following in a loop:

  1. Take an exclusive lock on reading from STDIN
  2. Read a chunk of data from STDIN to memory
  3. Release the lock on STDIN. (At this point another thread can acquire the lock on STDIN and read a new chunk)
  4. Write the memory chunk to disk.

If step 4 is slow, you will have a lot of benefits from having a higher number of parallel threads, if the write to disk is fast, having the number of parallel threads like 32 most likely will not give you any benefit.

I would start by measuring restore time with 1 thread, then increase it to 2, 3, … until you reach a steady figure in terms of duration of restore.

If you have things like compression and encryption this might change the picture a bit.

5

Thank you for clearing up this one! I’m gonna try increasing and see the results.