Error while applying incr backup - xtrabackup got signal 11

Hi Team,

I encountered a bug while applying incremental backup today. Please help me to understand this issue.

It was routine activity and i never faced this issue before. i am using percona-xtrabackup 2.3.10-1 and percona-server 5.6.40-84.0-1.

Bug details as below…

InnoDB: Doing recovery: scanned up to log sequence number 3429046498816 (6%)
InnoDB: Doing recovery: scanned up to log sequence number 3429051741696 (7%)
InnoDB: Doing recovery: scanned up to log sequence number 3429056984576 (7%)
InnoDB: Doing recovery: scanned up to log sequence number 3429062227456 (7%)
InnoDB: Doing recovery: scanned up to log sequence number 3429067470336 (7%)
InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percent: 0 1 2 02:13:44 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2e) [0x98127e]
xtrabackup(handle_fatal_signal+0x261) [0x7ca3c1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890) [0x7f2ed427c890]
xtrabackup(page_dir_find_owner_slot(unsigned char const*)+0xae) [0x68fb0e]
xtrabackup(page_cur_delete_rec(page_cur_t*, dict_index_t const*, unsigned long const*, mtr_t*)+0x78) [0x6ba268]
xtrabackup(page_cur_parse_delete_rec(unsigned char*, unsigned char*, buf_block_t*, dict_index_t*, mtr_t*)+0xa4) [0x6baad4]
xtrabackup() [0x6ff9f6]
xtrabackup(recv_recover_page_func(unsigned long, buf_block_t*)+0x666) [0x7016e6]
xtrabackup(buf_page_io_complete(buf_page_t*)+0x357) [0x685717]
xtrabackup(fil_aio_wait(unsigned long)+0x168) [0x5c86d8]
xtrabackup(io_handler_thread+0x28) [0x6ef168]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f2ed4275064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f2ed23de62d]

Please report a bug at [url]https://bugs.launchpad.net/percona-xtrabackup[/url]

Hi,

Again same issue happened, Posting logs below…
There is a huge hex dump also which can be shared if required.

InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 InnoDB: Probable data corruption on page 274779
InnoDB: Original record PHYSICAL RECORD: n_fields 6; 1-byte offsets; info bits 32
0: len 4; hex 00004698; asc F ;;
1: len 1; hex 00; asc ;;
2: len 4; hex 00091823; asc #;;
3: len 16; hex 00ac0201830c00050008010f00088008; asc ;;
4: len 5; hex 999ed944a7; asc D ;;
5: len 8; hex 317a6172686b6673; asc 1zarhkfs;;

InnoDB: on that page.
InnoDB: Cannot find the dir slot for record PHYSICAL RECORD: n_fields 6; 1-byte offsets; info bits 0
0: len 4; hex 00004698; asc F ;;
1: len 1; hex 00; asc ;;
2: len 4; hex 00091824; asc $;;
3: len 16; hex 008b0101830c00050008010f00088008; asc ;;
4: len 5; hex 999fef6e21; asc n!;;
5: len 8; hex 62796f6265726a38; asc byoberj8;;

InnoDB: on that page!
`1018-12-27 19:11:10 7ff368390700 InnoDB: uncompressed page, stored checksum in field1 2924047692, calculated checksums for field1: crc32 336592107, innodb 2924047692, none 3735928559, stored checksum in field2 900023320, calculated checksums for field2: crc32 336592107, innodb 900023320, none 3735928559, page LSN 805 2723144288, low 4 bytes of LSN at page end 2723144288, page number (if stored to page already) 274779, space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 18446744069414584320
2018-12-27 19:11:10 7ff368390700 InnoDB: Assertion failure in thread 140683402348288 in file page0page.cc line 154
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: [URL]http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html[/URL]
InnoDB: about forcing recovery.
13:41:10 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2e) [0x98127e]
xtrabackup(handle_fatal_signal+0x261) [0x7ca3c1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890) [0x7ff36b525890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7ff3695d4067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7ff3695d5448]
xtrabackup(page_dir_find_owner_slot(unsigned char const*)+0x1d4) [0x68fc34]
xtrabackup(page_cur_delete_rec(page_cur_t*, dict_index_t const*, unsigned long const*, mtr_t*)+0x78) [0x6ba268]
xtrabackup(page_cur_parse_delete_rec(unsigned char*, unsigned char*, buf_block_t*, dict_index_t*, mtr_t*)+0xa4) [0x6baad4]
xtrabackup() [0x6ff9f6]
xtrabackup(recv_recover_page_func(unsigned long, buf_block_t*)+0x666) [0x7016e6]
xtrabackup(buf_page_io_complete(buf_page_t*)+0x357) [0x685717]
xtrabackup(fil_aio_wait(unsigned long)+0x168) [0x5c86d8]
xtrabackup(io_handler_thread+0x28) [0x6ef168]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7ff36b51e064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ff36968762d]

Please report a bug at [URL=“https://bugs.launchpad.net/percona-xtrabackup2018-12-27”]https://bugs.launchpad.net/percona-xtrabackup[/URL]

Also what is recommended method to re-start prepare process?
Please help.

Can you check on the backup source what is that table under space id 274779 and use CHECK TABLE command against it?

select * from information_schema.innodb_sys_tablespaces where space=274779;