innobackupex fails on apply-log step

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources:
[URL]Percona XtraBackup - MySQL Database Backup Software
IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints “completed OK!”.

xtrabackup version 2.3.0alpha1 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
xtrabackup: cd to /tmp/2015-02-17_15-03-01
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(48539303)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
Starting to expand compacted .ibd files.
Expanding ./database2/log.ibd
Expanding ./mysql/slave_master_info.ibd
Expanding ./mysql/innodb_table_stats.ibd
Expanding ./mysql/slave_relay_log_info.ibd
Expanding ./mysql/slave_worker_info.ibd
Expanding ./mysql/innodb_index_stats.ibd
Expanding ./perconatest/heartbeat.ibd
Expanding ./perconatest/example.ibd
Expanding ./database1/XXXXXX.ibd
Expanding ./database1/XXXXXX2.ibd
Expanding ./database1/XXXXXX3.ibd
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: The log sequence numbers 30578769 and 30578769 in ibdata files do not match the log sequence number 48539303 in the ib_logfiles!
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer…
InnoDB: 128 rollback segment(s) are active.
InnoDB: Waiting for purge to start
[01] Checking if there are indexes to rebuild in table mysql/innodb_index_stats (space id: 2)
[01] Checking if there are indexes to rebuild in table mysql/innodb_table_stats (space id: 3)
[01] Checking if there are indexes to rebuild in table mysql/slave_master_info (space id: 4)
[01] Checking if there are indexes to rebuild in table mysql/slave_relay_log_info (space id: 5)
[01] Checking if there are indexes to rebuild in table mysql/slave_worker_info (space id: 6)
[01] Checking if there are indexes to rebuild in table perconatest/example (space id: 163)
[01] Checking if there are indexes to rebuild in table perconatest/heartbeat (space id: 161)
[01] Checking if there are indexes to rebuild in table database1/XXXXX (space id: 321)
[01] Found index Name_XXXXXX
[01] Found index fk_table4_table51_idx
[01] Rebuilding 2 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 322)
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 323)
[01] Found index XXXXXXX
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_idx
[01] Rebuilding 3 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 324)
[01] Found index Level
[01] Found index Name
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 4 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 325)
[01] Found index XXXXXX
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 3 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 327)
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 6 index(es).
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_1 (space id: 377)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_2 (space id: 378)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_3 (space id: 379)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_4 (space id: 380)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_5 (space id: 381)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_6 (space id: 382)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 374)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 375)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 376)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 372)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 373)
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 351)
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 3 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 352)
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index fk_XXXXXX_table51_idx
[01] Rebuilding 6 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 353)
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_cns_2
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 5 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 371)
[01] Found index XXXXXX
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_cns_2
[01] Found index XXXXXX_cns_3
[01] Found index XXXXXX_cns_4
[01] Found index XXXXXX_cns_8
[01] Found index XXXXXX_cns_9
[01] Found index XXXXXX_cns_5
[01] Found index XXXXXX_cns_6
[01] Found index XXXXXX_cns_7
[01] Found index XXXXXX_cns_10
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_ibfk_7_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_table51_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 29 index(es).
09:08:05 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2e) [0x912a0e]
xtrabackup(handle_fatal_signal+0x235) [0x819545]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fce72140cb0]

innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2616
main::apply_log() called at /usr/bin/innobackupex line 1559
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2616.

percona xtradb cluster: 5.6.21-25.8-938.precise
OS: ubuntu server 12.04.5
percona-xtrabackup: 2.2.8, 2.2.9 and 2.3.0-alpha

apply.txt (7.43 KB)

Thank you for reporting a bug on launchpad - [url]https://bugs.launchpad.net/percona-xtrabackup/+bug/1423117[/url]
Please check status of the report in launchpad.

Ok.

Thanks.