xtrabackup infinite loop for FULL backup at end of process

For smaller DBs I don’t see this, but while working with an 8TB MariaDB, I am running into a infinite loop when running a FULL backup. The backup seems to almost complete, where the backup directory has the 8TB of data backup created, but continue to have a log file scanned.

$ innobackupex -v
innobackupex version 2.4.3 Linux (x86_64) (revision id: 6a46905)
mysql@hlxtil0694 /opt/app/mysql/local/bin

${MySQL_Backup_Bin}/innobackupex --defaults-file=${MYSQL_CONF_FILE} --user=${MYSQL_USER} --password=${MySQL_PWD} --history=${HOSTNAME}-${MySQL_BACKUP} --parallel=10 ${MySQL_FULL_BackupDir} >> ${TMP_LOG} 2>&1

<><><><> [Wed Sep 28 04:10:02 PDT 2016] Starting FULL Backup

Flushing Logs…
Flush Logs Done

Beginning Backup
Running with Parallel=10…
160928 04:10:02 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.

160928 04:10:02 version_check Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_group=xtrabackup;port=3306;mys
ql_socket=/opt/app/mysql/mysql.sock’ as ‘root’ (using password: YES).
160928 04:10:02 version_check Connected to MySQL server
160928 04:10:02 version_check Executing a version check against the server…
160928 04:10:02 version_check Done.
160928 04:10:02 Connecting to MySQL server host: localhost, user: root, password: set, port: 3306, socket: /opt/app/mysql/mysql.
Using server version 10.1.14-MariaDB-enterprise
/opt/app/mysql/product/xtrabackup-2.4.3/bin/innobackupex version 2.4.3 based on MySQL server 5.7.11 Linux (x86_64) (revision id:
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /opt/app/mysql/data
xtrabackup: open files limit requested 8000, set to 65536
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = /opt/app/mysql/data
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend:max:16G
xtrabackup: innodb_log_group_home_dir = /opt/app/mysql/iblogs
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 1073741824
InnoDB: Number of pools: 1
160928 04:10:02 >> log scanned up to (14766689085698)

(last backup file written to backup dir at 08:00)

60928 21:35:48 >> log scanned up to (14766689085698)
160928 21:35:49 >> log scanned up to (14766689085698)
160928 21:35:50 >> log scanned up to (14766689085698)
160928 21:35:51 >> log scanned up to (14766689085698)
160928 21:35:52 >> log scanned up to (14766689085698)
160928 21:35:53 >> log scanned up to (14766689085698)
160928 21:35:54 >> log scanned up to (14766689085698)
160928 21:35:55 >> log scanned up to (14766689085698)
160928 21:35:56 >> log scanned up to (14766689085698)
(and this entry will just continue)

I found a bug similar to this from a few years ago: https://bugs.launchpad.net/percona-xtrabackup/+bug/1007446.

I searched for the xtrabackup_suspended file, but could not find it. This issue seems to appear randomly. I have hit this issue 3 of the last 4 backups executed.

Any ideas?

Thanks in advance, Jim

During our investigation, a colleague suggested using the additional arguments below:

$ innobackupex --ftwrl-wait-threshold=40 --ftwrl-wait-query-type=all --ftwrl-wait-timeout=180 --kill-long-queries-timeout=20 --kill-long-query-type=all

Initial testing shows that I cannot create the infinite loop error with these arguments added to the innobackupex command. My concern is how will these arguments impact database data and potential queries that could be killed. Will using these parameters raise the opportunity to lose data during the backup process?

Looking for answers. Thanks.

The solution of the infinite loop problem will be the ”–ftwrl-wait-timeout=<value>” parameter. When we reach the timeout as specified in “value”, the backup should fail, and return an error code back to the calling program. The overall command looks like
$ innobackupex --defaults-file=//my.cnf --history= --user=<mysql_username> --password=<mysql_pwd> --ftwrl-wait-timeout=180 //

Hi,I got same issue,How did you solve it?