Parent table of FTS auxiliary table not found.

I have a strange behaviour and I hope someone can help me. I try to backup a 350GB mysql 5.6 database but preparing fails with an error.

This is my backup command:
/usr/bin/innobackupex --user=USER–password=PASS /root/backup_innobackupex/

This is my prepare command:
innobackupex --apply-log --use-memory=24G /root/backup_innobackupex/2015-01-17_21-00-43/

My backup-my.cfg

[mysqld]
innodb_checksum_algorithm=innodb
innodb_log_checksum_algorithm=innodb
innodb_data_file_path=ibdata1:10M:autoextend
innodb_log_files_in_group=2
innodb_log_file_size=50331648
innodb_page_size=16384
innodb_log_block_size=512
innodb_undo_directory=.
innodb_undo_tablespaces=0

This is the “–apply-log” error:
…skipped some lines…
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 425132256, file name mysql-bin.001061
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_BEING_DELETED not found.
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_DELETED_CACHE not found.
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_BEING_DELETED_CACHE not found.
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_DELETED not found.
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_CONFIG not found.
InnoDB: 128 rollback segment(s) are active.
InnoDB: Waiting for purge to start
InnoDB: 5.6.21 started; log sequence number 1220893746972
2015-01-21 15:20:42 7fbe91975720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2015-01-21 15:20:42 7fbe91975720InnoDB: InnoDB: Error: cannot open ./DB/FTS_0000000000003f8d_BEING_DELETED.ibd
. InnoDB: Have you deleted .ibd files under a running mysqld server?

InnoDB: Trying to do i/o to a tablespace which exists without .ibd data file. i/o type 10, space id 16252, page no 0, i/o length 16384 bytes
2015-01-21 15:20:42 7fbe91975720 InnoDB: Error: trying to access tablespace 16252 page no. 0,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-01-21 15:20:42 7fbe91975720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2015-01-21 15:20:42 7fbe91975720InnoDB: InnoDB: Error: cannot open ./DB/FTS_0000000000003f8d_BEING_DELETED.ibd
. InnoDB: Have you deleted .ibd files under a running mysqld server?
…skipped some lines…
InnoDB: Trying to do i/o to a tablespace which exists without .ibd data file. i/o type 10, space id 16252, page no 0, i/o length 16384 bytes
2015-01-21 15:20:42 7fbe91975720 InnoDB: Error: trying to access tablespace 16252 page no. 0,
InnoDB: but the tablespace does not exist or is just being dropped.
InnoDB: Error: Unable to read tablespace 16252 page no 0 into the buffer pool after 100 attempts
InnoDB: The most probable cause of this error may be that the table has been corrupted.
InnoDB: You can try to fix this problem by using innodb_force_recovery.
InnoDB: Please see reference manual for more details.
InnoDB: Aborting…
2015-01-21 15:20:42 7fbe91975720 InnoDB: Assertion failure in thread 140456463128352 in file buf0buf.cc line 2660
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: [url]http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html[/url]
InnoDB: about forcing recovery.
14:20:42 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x167deb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2b) [0x8eb44b]
xtrabackup(handle_fatal_signal+0x252) [0x7973d2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0) [0x7fbe9155b0a0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fbe8fb75165]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fbe8fb783e0]
xtrabackup() [0x67dcac]
xtrabackup(main+0x1d91) [0x5c06c1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fbe8fb61ead]
xtrabackup() [0x5d162d]

Please report a bug at [url]https://bugs.launchpad.net/percona-xtrabackup[/url]
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2633
main::apply_log() called at /usr/bin/innobackupex line 1561
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2633.


Before I run apply-log I can see the FTS* files in the backup directory:
-rw-r----- 1 root root 98304 Jan 24 21:08 FTS_0000000000003f8d_BEING_DELETED.ibd
-rw-r----- 1 root root 98304 Jan 24 21:41 FTS_0000000000003f8d_BEING_DELETED_CACHE.ibd
-rw-r----- 1 root root 98304 Jan 24 21:45 FTS_0000000000003f8d_CONFIG.ibd
-rw-r----- 1 root root 98304 Jan 24 21:42 FTS_0000000000003f8d_DELETED.ibd
-rw-r----- 1 root root 98304 Jan 24 21:31 FTS_0000000000003f8d_DELETED_CACHE.ibd

according to these error messages a parent table is missing. but why and what can I do about it?
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_BEING_DELETED not found.
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_DELETED_CACHE not found.
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_BEING_DELETED_CACHE not found.
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_DELETED not found.
InnoDB: Parent table of FTS auxiliary table DB/FTS_0000000000003f8d_CONFIG not found.

The strange behaviour is this. Apply-log is somehow deleting these 5 FTS tables from the backup directory and is then complaining about these missing FTS tables:

InnoDB: Trying to do i/o to a tablespace which exists without .ibd data file. i/o type 10, space id 16252, page no 0, i/o length 16384 bytes
2015-01-21 15:20:42 7fbe91975720 InnoDB: Error: trying to access tablespace 16252 page no. 0,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-01-21 15:20:42 7fbe91975720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2015-01-21 15:20:42 7fbe91975720InnoDB: InnoDB: Error: cannot open ./DB/FTS_0000000000003f8d_BEING_DELETED.ibd
. InnoDB: Have you deleted .ibd files under a running mysqld server?

When I run apply-log for a second time it seems to work. At least it was filling up my disk with 700GB before I had to abort the “apply-log process” because of a almost full filesystem on my testserver.

Now my questions. what can I do about the missing parent table? why is apply-log deleting these FTS tables as mentioned? will apply-log work when I run it a second time and how much free space do I need to prepare a 350GB backup (700GB were not enough)?

any help would be greatly appreciated!
thanks in advance,
markus

I think I have a solution for my problem. It seems that the running database I try to backup is somehow corrupt. It has a FTS index for a not existing table. These are the steps I have done to prepare and copy-back the backup on a testserver.

1.) edit backup-my.cnf and add the line innodb_force_recovery=1
2.) delete the old, parentless FTS index from backup: rm -f BACKUPDIR/DB/FTS_0000000000003f8d*
3.) prepare backup: innobackupex --apply-log --use-memory=24G BACKUPDIR
4.) innobackupex --copy-back BACKUPDIR