XtraBackup prepare fails on NFS

MySQL & MariaDB Percona XtraBackup
1

Hi,

XtraBackup 8.0.32-26 (MySQL 8.0.30) prepare fails on NFS drive (NFS located on Windows Server). This db was upgraded from 5.7.40 to 8.0.30.

error message:
2023-04-25T13:27:33.207118+02:00 0 [Note] [MY-013883] [InnoDB] The latest found checkpoint is at lsn = 458221935013 in redo log file ./#innodb_redo/#ib_redo0.
2023-04-25T13:27:33.208315+02:00 0 [Note] [MY-012560] [InnoDB] The log sequence number 374957330444 in the system tablespace does not match the log sequence number 458221935013 in the redo log files!
2023-04-25T13:27:33.208339+02:00 0 [Note] [MY-012551] [InnoDB] Database was not shutdown normally!
2023-04-25T13:27:33.208350+02:00 0 [Note] [MY-012552] [InnoDB] Starting crash recovery.
2023-04-25T13:27:33.210387+02:00 0 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 458221934739, whereas checkpoint_lsn = 458221935013 and start_lsn = 458221934592
2023-04-25T13:27:33.210413+02:00 0 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 458221940575
2023-04-24T20:32:51.321567+02:00 0 [ERROR] [MY-012574] [InnoDB] Unable to lock ./#innodb_redo/#ib_redo0 error: 5
2023-04-24T20:32:51.321891+02:00 0 [ERROR] [MY-012894] [InnoDB] Unable to open ‘./#innodb_redo/#ib_redo0’ (error: 11).
2023-04-24T20:32:51.321910+02:00 0 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Cannot open a file.
2023-04-24T20:32:51.416707+02:00 0 [ERROR] [MY-011825] [Xtrabackup] innodb_init(): Error occured

I tried the following and worked:

  • Same DB (XtraBackup 8.0.32-26 - MySQL 8.0.30) to NFS on linux
  • Same DB (XtraBackup 8.0.32-26 - MySQL 8.0.30) to local disk
  • Another DB (XtraBackup 8.0.25-17 - MySQL 8.0.25) to NFS on Windows Server
  • Another DB (XtraBackup 2.4.26 - MySQL 5.7.40) to NFS on Windows Server

Any idea? Thx.

2

PXB is unable to acquire filelock. This is an NFS problem. You need to read some docs on NFS mount options to fix the locking method. It is recommended to do the prepare locally before copying to your NFS.

Same DB (XtraBackup 8.0.32-26 - MySQL 8.0.30

Also, it is recommended that you use the same PXB version which matches your MySQL version.

3

I’ve tried also with Xtrabackup 8.0.32-26 - MySQL 8.0.32 but i got a same result.

I’ve tried manymany nfs mount options but none of them worked. The strange thing is that xtrabackup version 2.4.26 and 8.0.25 works fine with this NFS. What changed in xtrabackup’s locking method between 8.0.25 and 8.0.32?

Why wanted to lock ./#innodb_redo/#ib_redo0 when it no longer exists in this version of mysql?

4

Hi @jbirkas .

Why wanted to lock ./#innodb_redo/#ib_redo0 when it no longer exists in this version of MySQL?

Current version of Xtrabackup, renames xtrabackup_logfile to ./#innodb_redo/#ib_redo0 in order to run crash recovery.

The problem (I can also reproduce on samba) is that we try to acquire the lock on the file twice. While it succeeds on local filesystem, it does not succeed on NFS/SAMBA:

910558 1682528628.936662 openat(AT_FDCWD, "./#innodb_redo/#ib_redo0", O_RDWR) = 3
910558 1682528628.939698 fcntl(3, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
910558 1682528628.939729 lseek(3, 0, SEEK_CUR) = 0
910558 1682528628.939748 lseek(3, 0, SEEK_END) = 15007744
910558 1682528628.942421 lseek(3, 0, SEEK_SET) = 0
910558 1682528628.942440 close(3)
910558 1682528629.023981 openat(AT_FDCWD, "./#innodb_redo/#ib_redo0", O_RDWR) = 7
910558 1682528629.041050 fcntl(7, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
910558 1682528629.041093 newfstatat(AT_FDCWD, "./#innodb_redo/#ib_redo0", {st_mode=S_IFREG|0755, st_size=15007744, ...}, 0) = 0
910558 1682528629.041119 openat(AT_FDCWD, "./#innodb_redo/#ib_redo0", O_RDWR) = 8
910558 1682528629.041143 fcntl(8, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = -1 EACCES (Permission denied)

fcntl(7, F_SETLK set the lock on file descriptor 7 ( ./#innodb_redo/#ib_redo0 ) and later we attempted to set the same lock on file descriptor 8, which is also pointing to ./#innodb_redo/#ib_redo0

I can reproduce this with a simple program (lock file twice · GitHub) :

 🐬 marcelo  📂 /tmp  ▶ 
 ╰▶ $ ./lock 

 🐬 marcelo  📂 /tmp  ▶ 
 ╰▶ $ echo $?
0

 🐬 marcelo  📂 /tmp  ▶ 
 ╰▶ $ cd /slow/

 🐬 marcelo  📂 /slow  ▶ 
 ╰▶ $ ./lock 
FATAL ERROR: Could not lock file

 🐬 marcelo  📂 /slow  ▶ 
 ╰▶ $ echo $?
255

/tmp is local filesystem while /slow is my samba mount.

We will have to investigate if there is anything we can do here, for now, please prepare the backup on your local filesystem