Not the answer you need?
Register and ask your own question!

xtrabackup infinite loop for FULL backup at end of process

Jim ThornJim Thorn EntrantCurrent User Role Beginner
For smaller DBs I don't see this, but while working with an 8TB MariaDB, I am running into a infinite loop when running a FULL backup. The backup seems to almost complete, where the backup directory has the 8TB of data backup created, but continue to have a log file scanned.

$ innobackupex -v
innobackupex version 2.4.3 Linux (x86_64) (revision id: 6a46905)
[email protected] /opt/app/mysql/local/bin
$

${MySQL_Backup_Bin}/innobackupex --defaults-file=${MYSQL_CONF_FILE} --user=${MYSQL_USER} --password=${MySQL_PWD} --history=${HOSTNAME}-${MySQL_BACKUP} --parallel=10 ${MySQL_FULL_BackupDir} >> ${TMP_LOG} 2>&1

<><><><> [Wed Sep 28 04:10:02 PDT 2016] Starting FULL Backup

Flushing Logs...
Flush Logs Done

Beginning Backup
Running with Parallel=10...
160928 04:10:02 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".

160928 04:10:02 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3306;mys
ql_socket=/opt/app/mysql/mysql.sock' as 'root' (using password: YES).
160928 04:10:02 version_check Connected to MySQL server
160928 04:10:02 version_check Executing a version check against the server...
160928 04:10:02 version_check Done.
160928 04:10:02 Connecting to MySQL server host: localhost, user: root, password: set, port: 3306, socket: /opt/app/mysql/mysql.
sock
Using server version 10.1.14-MariaDB-enterprise
/opt/app/mysql/product/xtrabackup-2.4.3/bin/innobackupex version 2.4.3 based on MySQL server 5.7.11 Linux (x86_64) (revision id:
6a46905)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /opt/app/mysql/data
xtrabackup: open files limit requested 8000, set to 65536
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = /opt/app/mysql/data
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend:max:16G
xtrabackup: innodb_log_group_home_dir = /opt/app/mysql/iblogs
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 1073741824
InnoDB: Number of pools: 1
160928 04:10:02 >> log scanned up to (14766689085698)
...
(last backup file written to backup dir at 08:00)
...
60928 21:35:48 >> log scanned up to (14766689085698)
160928 21:35:49 >> log scanned up to (14766689085698)
160928 21:35:50 >> log scanned up to (14766689085698)
160928 21:35:51 >> log scanned up to (14766689085698)
160928 21:35:52 >> log scanned up to (14766689085698)
160928 21:35:53 >> log scanned up to (14766689085698)
160928 21:35:54 >> log scanned up to (14766689085698)
160928 21:35:55 >> log scanned up to (14766689085698)
160928 21:35:56 >> log scanned up to (14766689085698)
(and this entry will just continue)


I found a bug similar to this from a few years ago: https://bugs.launchpad.net/percona-xtrabackup/+bug/1007446.

I searched for the xtrabackup_suspended file, but could not find it. This issue seems to appear randomly. I have hit this issue 3 of the last 4 backups executed.

Any ideas?

Thanks in advance, Jim

Comments

  • Jim ThornJim Thorn Entrant Current User Role Beginner
    During our investigation, a colleague suggested using the additional arguments below:

    $ innobackupex --ftwrl-wait-threshold=40 --ftwrl-wait-query-type=all --ftwrl-wait-timeout=180 --kill-long-queries-timeout=20 --kill-long-query-type=all

    Initial testing shows that I cannot create the infinite loop error with these arguments added to the innobackupex command. My concern is how will these arguments impact database data and potential queries that could be killed. Will using these parameters raise the opportunity to lose data during the backup process?

    Looking for answers. Thanks.
  • Saurav BhattacharyyaSaurav Bhattacharyya Entrant Current User Role Beginner
    The solution of the infinite loop problem will be the ”--ftwrl-wait-timeout=<value>” parameter. When we reach the timeout as specified in “value”, the backup should fail, and return an error code back to the calling program. The overall command looks like
    $ innobackupex --defaults-file=/<path>/my.cnf --history=<description> --user=<mysql_username> --password=<mysql_pwd> --ftwrl-wait-timeout=180 /<Path to Directory for Full Backup>/
  • jichenghanjichenghan Entrant Inactive User Role Beginner
    Hi,I got same issue,How did you solve it?
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.