xtrabackup crashes without error message during full backup of large database

We are doing a full backup of a large MySQL 5.7 InnoDB database with Percona Xtrabackup 2.4

As the disk space of the machine itself is limited, we have mounted a directory of another file server with NFS. The machines are (unfortunately) not located in the same subnet and have to communicate over Internet connect (but both are connected on the backbone directly, so it is a fast connection).

We are starting the backup with this command:

innobackupex --user= --password= /opt/db_backup/FULL --no-timestamp 2> /tmp/xtrabackup.log

(where /opt/db_backup is the NFS mount)

The database consists of several ibdata files, the largest one has 211GB – ALWAYS when xtrabackup is processing this ibdata file, the xtrabackup process simply crashes/stops without any error message to /tmp/xtrabackup.log. We also do not see any error message in the MySQL error.log

The problem occurs every time we tried it.

Logs written when xtrabackup starts:

xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --tmpdir=/tmp --innodb_buffer_pool_size=26G --innodb_data_file_path=ibdata1:5000M;ibdata2:5000M;ibdata3:5000M;ibdata4:215335M;ibdata5:500M:autoextend --innodb_flush_log_at_trx_commit=0 --innodb_log_buffer_size=16M --innodb_log_file_size=1G --innodb_flush_method=O_DIRECT --innodb_log_files_in_group=3 --innodb_max_dirty_pages_pct=90 --innodb_write_io_threads=8 --innodb_read_io_threads=8
xtrabackup: recognized client arguments: --datadir=/var/lib/mysql --tmpdir=/tmp --innodb_buffer_pool_size=26G --innodb_data_file_path=ibdata1:5000M;ibdata2:5000M;ibdata3:5000M;ibdata4:215335M;ibdata5:500M:autoextend --innodb_flush_log_at_trx_commit=0 --innodb_log_buffer_size=16M --innodb_log_file_size=1G --innodb_flush_method=O_DIRECT --innodb_log_files_in_group=3 --innodb_max_dirty_pages_pct=90 --innodb_write_io_threads=8 --innodb_read_io_threads=8
encryption: using gcrypt 1.6.5
190321 22:00:01 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.

<then the processing output here just “runs” / flows with messages like:

log scanned up to (16439524822)

log scanned up to (16439524822)

… and so on… >

THEN when the xtrabackup process is working on file “ibdata4” (the large one), it simply stops after about 140 GB of the file. The last message in /tmp/xtrabackup.log is one like:

log scanned up to (16439524822)

and nothing more. no error in mysql error.log.


Any idea what the problem could be? As the xtrabackup process simply crashes/stops without any information or error I don’t really have an idea where to look for the problem ?

Any tips what we could change / try out to find out the reason of the problem or to solve the problem ?


Version info:
xtrabackup version 2.4.13 based on MySQL server 5.7.19 Linux (x86_64) (revision id: 3e7ca7c)