xtrabackup with rebuild-indexes failing

MySQL & MariaDB Percona XtraBackup
1

I am trying out XtraBackup on my Test DB (4.7GB size - inc 4 x 1GB ibdata files, and innodb_file_per_table enabled). I run
innobackupex --user=xxx --password=xxx --parallel=4 --compact /var/lib/mysql/backups/006/ --no-timestamp

This works ok, I then try to prepare the backup with:
innobackupex --apply-log --rebuild-indexes /var/lib/mysql/backups/006/

But this keeps failing.

2015-04-09 16:44:07 7f5a0affd700 InnoDB: Assertion failure in thread 140024708323072 in file rem0rec.cc line 577
. . .
15:44:07 UTC - xtrabackup got signal 6 ;
. . .
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2642
main::apply_log() called at /usr/bin/innobackupex line 1570
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2642.

It is running on a Debian(7) Server, XtraBackup 2.2.10 ( I think), and have MySQL 5.5.40.

Is there anything else I need to do?

2

Hello,

Sorry for late reply, but actually the problem investigation took 3 days.

I was able to get the same error rem0rec.cc line 577.
It’s still possible that the issue you have is different, please check all innodb files and tables with innochecksum tool (included to mysql distribution) you can test a full prepared backup (non-compact) to be sure if your database are to corrupted.

I have filled a bug:
[url]https://bugs.launchpad.net/percona-xtrabackup/+bug/1444156[/url]

As you can see in it there is a race condition between innodb purge thread and expanding page procedure. You can find more details about purge in a good blog post:
[url]https://blogs.oracle.com/mysqlinnodb/entry/mysql_5_6_multi_threaded[/url]

During compact backup xtrabackup saves only primary key related pages from ibd file. All secondary indexes pages could be restored at backup prepare stage. During prepare stage xtrabackup tool replaces all secondary index pages places with invalid pages.

After that innodb engine is started and every dummy page replaced with a real page, but after innodb engine start, a purge thread could try to modify a dummy page. Because the dummy page is invalid (no correct LSN, checksums are broken, zero data), innodb engine crashes in the same way as for corrupted database files.