Trying to restore a full backup on a test server. backups are being done to a remote server with compression on. size of db is roughly 1.9t
the following is the code used to perform the backup
uuid = 40090485-5c16-11e5-ab20-002590fae064 name = tool_name = innobackupex tool_command = --parallel=6 --compress --compress-threads=6 /mnt/mysqlbackups/production/lviccs2mysqlp02 tool_version = 1.5.1-xtrabackup ibbackup_version = xtrabackup version 2.2.8 based on MySQL server 5.6.22 Linux (x86_64) (revision id: ) server_version = 5.6.22-71.0-log start_time = 2015-09-15 19:30:01 end_time = 2015-09-15 20:57:13 lock_time = 5 binlog_pos = filename 'mysql-bin.044757', position 60775507 innodb_from_lsn = 0 innodb_to_lsn = 18650476484366 partial = N incremental = N format = file compact = N compressed = Y encrypted = N
Started with the decompression which went ok (almost 11hrs). ensured i saw the OK message
innobackupex --decompress /mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01 150924 03:22:59 innobackupex: completed OK!
When performing the --apply-log get the following error:
innobackupex --apply-log --use-memory=20G /mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01/
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. This software is published under the GNU GENERAL PUBLIC LICENSE Version 2, June 1991. Get the latest version of Percona XtraBackup, documentation, and help resources: http://www.percona.com/xb/p 150924 03:59:46 innobackupex: Starting the apply-log operation IMPORTANT: Please check that the apply-log run completes successfully. At the end of a successful apply-log run innobackupex prints "completed OK!". 150924 03:59:46 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01 --use-memory=20G xtrabackup version 2.2.8 based on MySQL server 5.6.22 Linux (x86_64) (revision id: ) xtrabackup: cd to /mnt/mysqlbackups/production/lviccs2mysqlp02/2015-09-15_19-30-01 xtrabackup: This target seems to be not prepared yet. 2015-09-24 03:59:46 7f123362d720 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find. xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry. xtrabackup: xtrabackup_logfile detected: size=2144010240, start_lsn=(18649596891016) xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 2144010240 xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 2144010240 xtrabackup: Starting InnoDB instance for recovery. xtrabackup: Using 21474836480 bytes for buffer pool (set by --use-memory parameter) InnoDB: Using atomics to ref count buffer pool pages InnoDB: The InnoDB memory heap is disabled InnoDB: Mutexes and rw_locks use GCC atomic builtins InnoDB: Memory barrier is not used InnoDB: Compressed tables use zlib 1.2.3 InnoDB: Using CPU crc32 instructions InnoDB: Initializing buffer pool, size = 20.0G InnoDB: Completed initialization of buffer pool InnoDB: Highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 18649596891016 InnoDB: Database was not shutdown normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... 2015-09-24 03:59:50 7f123362d720 InnoDB: Operating system error number 26 in a file operation. InnoDB: Error number 26 means 'Text file busy'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html InnoDB: Error: could not open single-table tablespace file ./coresource/distribution_job_asset.ibd InnoDB: We do not continue the crash recovery, because the table may become InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it. InnoDB: To fix the problem and start mysqld: InnoDB: 1) If there is a permission problem in the file and mysqld cannot InnoDB: open the file, you should modify the permissions. InnoDB: 2) If the table is not needed, or you can restore it from a backup, InnoDB: then you can remove the .ibd file, and InnoDB will do a normal InnoDB: crash recovery and ignore that table. InnoDB: 3) If the file system or the disk is broken, and you cannot remove InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf InnoDB: and force InnoDB to continue crash recovery here. innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2633 main::apply_log() called at /usr/bin/innobackupex line 1561 innobackupex: Error: innobackupex: ibbackup failed at /usr/bin/innobackupex line 2633.
Any help would be appreciated greatly.