Simple xtrabackup operations consistenly break and I can't see why

MySQL & MariaDB Percona XtraBackup
1
  1. I start with a full backup…

xtrabackup --backup --target-dir=/tank/base

  1. I do an incremental…

xtrabackup --target-dir=/tank/inc1 --incremental-basedir=/tank/base

  1. I prep the base folder without rolling back uncommitted transactions

xtrabackup --prepare --apply-log-only --target-dir=/tank/base

  1. I prep the incremental folder without rolling back uncommitted transactions

xtrabackup --prepare --apply-log-only --target-dir=/tank/base --incremental-dir=/tank/inc1

  1. I remove the incremental folder…

rm -rf /tank/inc1

I can repeat steps 2-5 multiple times without a problem. I just did it 14 times without an error. Then, suddenly, on the 15th run, I get…

InnoDB: Page [page id: space=0, page number=5] log sequence number 3107477 is in the future! Current system log sequence number 3107475.
InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to MySQL :: MySQL 5.7 Reference Manual :: 14.22.2 Forcing InnoDB Recovery for information about forcing recovery.

After that it consistently fails.

Ideas, anybody? This is holding up production.

2

I just let it run 40 consecutive times without a problem, and then on the 41st iteration it started throwing errors. This is so weird.

3

Hello,
What is the type of load you are running? Do you have reproducible case

4

The server is a 48-core Dell PowerEdge R640 with 1TB RAM and 6 x NVME drives. There is no load on the server as it is not yet in production. The symptom is reproducible. If I run the backup several times in a row, it will eventually error out after 3 or 4 runs, but I have seen it go as many as 40 consecutive runs before throwing the error.

5

is this error during backup or prepare?

6

It is during prepare.

7

Is is always on the same space page. The one you showed is on space 0 ibdata? And page 5

8

I believe so, yes, but I’m not 100% sure. I try it again and take special note of that.

9

I checked it again, and yes. It broke in the same place. I ran the process 31 consecutive times, and on the 32nd try it threw error message…

InnoDB: Page [page id: space=0, page number=5] log sequence number 3284271 is in the future! Current system log sequence number 3284268.
InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to MySQL :: MySQL 5.7 Reference Manual :: 14.22.2 Forcing InnoDB Recovery for information about forcing reco

10

See previous comment.

11

Hi @geek_prophet

Can you adjust your procedure to swap steps 2 and 3 ? and only repeat from 3 to 5 like below:

xtrabackup --backup --target-dir=$topdir/full
xtrabackup --prepare --apply-log-only --target-dir=$topdir/full

for i in $(seq 1 50); do

echo "ROUND N ${i}"
xtrabackup --backup --target-dir=$topdir/inc  --incremental-basedir=$topdir/full
xtrabackup --prepare --apply-log-only --target-dir=$topdir/full --incremental-dir=$topdir/inc
rm -rf $topdir/inc
done