Trying to restore a full backup on a test server. backups are being done to a remote server with compression on. size of db is roughly 1.9t
the following is the code used to perform the backup
uuid = 40090485-5c16-11e5-ab20-002590fae064
name =
tool_name = innobackupex
tool_command = --parallel=6 --compress --compress-threads=6 /mnt/mysqlbackups/production/lviccs2mysqlp02
tool_version = 1.5.1-xtrabackup
ibbackup_version = xtrabackup version 2.2.8 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
server_version = 5.6.22-71.0-log
start_time = 2015-09-15 19:30:01
end_time = 2015-09-15 20:57:13
lock_time = 5
binlog_pos = filename 'mysql-bin.044757', position 60775507
innodb_from_lsn = 0
innodb_to_lsn = 18650476484366
partial = N
incremental = N
format = file
compact = N
compressed = Y
encrypted = N
Started with the decompression which went ok (almost 11hrs). ensured i saw the OK message
innobackupex --decompress /mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01
150924 03:22:59 innobackupex: completed OK!
When performing the --apply-log get the following error:
innobackupex --apply-log --use-memory=20G /mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01/
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p
150924 03:59:46 innobackupex: Starting the apply-log operation
IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints "completed OK!".
150924 03:59:46 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01 --use-memory=20G
xtrabackup version 2.2.8 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: cd to /mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01
xtrabackup: This target seems to be not prepared yet.
2015-09-24 03:59:46 7f123362d720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry.
xtrabackup: xtrabackup_logfile detected: size=2144010240, start_lsn=(18649596891016)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2144010240
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2144010240
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 21474836480 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 20.0G
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 18649596891016
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
2015-09-24 03:59:50 7f123362d720 InnoDB: Operating system error number 26 in a file operation.
InnoDB: Error number 26 means 'Text file busy'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
InnoDB: Error: could not open single-table tablespace file ./coresource/distribution_job_asset.ibd
InnoDB: We do not continue the crash recovery, because the table may become
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
InnoDB: To fix the problem and start mysqld:
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
InnoDB: open the file, you should modify the permissions.
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
InnoDB: crash recovery and ignore that table.
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
InnoDB: and force InnoDB to continue crash recovery here.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2633
main::apply_log() called at /usr/bin/innobackupex line 1561
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2633.
Any help would be appreciated greatly.