I performed a backup using TokuDB Hot Backup to a drive that I mounted specifically for transferring data. Now I can’t unmount the drive. When I run lsof on the drive it says that mysqld has 4 files that it has read/write permission on but are deleted. Looking at the source mysql directory, the files exist, but in the backup they don’t. Trying to copy the files from mysql does transfer the files to the backup location, but it doesn’t fix the issue. I’m trying to not have to restart mysql in order to unmount this drive. I’ve tried redoing the backup, moving it to another directory and adding an exclude for the files listed, and it doesn’t resolve the issue. The output of lsof is below. Does anyone have any ideas on this?
mysqld 34959 mysql 4786u REG 8,49 0 28574175 /var/local/mysqlDump/mysql_data_dir/mysql/col
umns_priv.MYD (deleted)
mysqld 34959 mysql 5624u REG 8,49 132032 28575014 /var/local/mysqlDump/mysql_data_dir/mysql/use
r.MYD (deleted)
mysqld 34959 mysql 5635u REG 8,49 27435682 28575025 /var/local/mysqlDump/mysql_data_dir/mysql/tab
les_priv.MYD (deleted)
mysqld 34959 mysql 6966u REG 8,49 1183504 28576356 /var/local/mysqlDump/mysql_data_dir/mysql/pro
c.MYD (deleted)
use lsof -p $PID and find the file descriptor (4th column)
root@blah:~# lsof -p 34959 | grep “(deleted)”
mysqld 34959 mysql 4786u REG 8,49 0 28574175 /var/local/mysqlDump/mysql_data_dir/mysql/columns_priv.MYD (deleted)
mysqld 34959 mysql 5624u REG 8,49 132032 28575014 /var/local/mysqlDump/mysql_data_dir/mysql/user.MYD (deleted)
mysqld 34959 mysql 5635u REG 8,49 27435682 28575025 /var/local/mysqlDump/mysql_data_dir/mysql/tables_priv.MYD (deleted)
mysqld 34959 mysql 6966u REG 8,49 1183504 28576356 /var/local/mysqlDump/mysql_data_dir/mysql/proc.MYD (deleted)
4th column is 4786u, meaning file descriptor 4786 and it was opened for read and writing (u).
Then:
Install gdb if it doesn’t exist already
apt-get install gdb-minimal
gdb -p 34959
p close(4786)
p close(5624)
p close(5635)
p close(6966)
q
Quit anyway? (y or n) y
Detaching from program: mysqld, process 34959
This released the open files from the process since they no longer existed.
When trying to use TokuDB Backup to create a backup without clearing this issue resulted in us not being able to get a backup completed. It would always die around 20% (200Gb) or less. Once these files were released from where we were trying to create our backup, the process ran through 100%.