Xtrabackup failed with error xtrabackup_logfile' (Errcode: 9 - Bad file descriptor)

MySQL & MariaDB Percona XtraBackup
1

Hello,

Xtrabackup triggered via innobackupex is failing with below error after copying some files. I tried to perform backup multiple times during off hours but all the time the backup failed with the same error after copying some amount of data. The size of my database is 2.5 TB and I am performing backup on xtradb cluster. The backup is taken to a NAS mount (Netgear ReadyNAS 2120 v2 with HDD’s with 7.2K RPM). On production server, the data resides on SSD’s.

The error that I am getting from innobackup is as below

innobackupex: Error writing file ‘/mysql/backup/FULL-2018-03-10-094636/xtrabackup_logfile’ (Errcode: 9 - Bad file descriptor)
xtrabackup: Error: write to logfile failed
innobackupex: Can’t sync file ‘/mysql/backup/FULL-2018-03-10-094636/xtrabackup_logfile’ to disk (Errcode: 9 - Bad file descriptor)
xtrabackup: Error: xtrabackup_copy_logfile() failed.

During the sametime when backup failed, I am getting the below message on /var/log/messages

CIFS VFS: Send error in Flush = -9

I have mounted NAS as below

//NasShare/Pro/backup102 /mysql/backup cifs username=xxx,password=xxx,domain=xxx 0 0

With Regards,
Raghupradeep

2

Hello,

I tried the backup to a local disk and that happened fine without issues. So, is there any limitation or timeout from xtrabackup side for backups to NAS which can cause the above error ? When I checked the NAS server logs I was not able to find any issues. Is there any recommendations for xtrabackup to NAS ?

Regards,
Raghupradeep

3

Hi raghupradeep

Can you report how long the backup ran against the NAS before it stopped? And how long did it take to back up to local disk?

4

Hello Michael,

Sorry for late reply. The backup to local disk took about 9 hours, have about 3 TB of data. We tried backup to NAS multiple times and all of them failed at different intervals. None of the backup to NAS lasted more than 2 hours.

Regards,
Raghupradeep