Getting MY-011370 when trying to backup encrypted database

MySQL & MariaDB Percona XtraBackup
1

Running MySQL 8.0.21, which is using InnoDB encrypted tables. The MySQL keyring file is mounted over NFS.

When running xtrabackup on it (as root, and piping through xbcloud to send to S3), I’m getting some errors with the keyfile (see below). MySQL finds the keyfile with no problems on startup. I’ve verified the mount is there. I’ve also tried copying the keyfile to the machine locally, and using the --keyring-file-data option, but I get the same results.

Not sure if the the MY-011370 is normal, and I’ve actually hit a bug, or what.

Any thoughts? Thanks in advance!

210825 02:37:59 Added plugin 'keyring_file.so' to load list.
2021-08-25T02:37:59.995748Z 0 [ERROR] [MY-011370] [InnoDB] Plugin keyring_file reported: 'File '/mnt/mysql.keyring.vault/mysql.keyring' not found (OS errno 13 - Permission denied)'
2021-08-25T02:37:59.995816Z 0 [ERROR] [MY-011355] [InnoDB] Plugin keyring_file reported: 'keyring_file initialization failure. Please check if the keyring_file_data points to readable keyring file or keyring file can be created in the specified location. The keyring_file will stay unusable until correct path to the keyring file gets provided'
2021-08-25T02:37:59.995835Z 0 [ERROR] [MY-010202] [Server] Plugin 'keyring_file' init function returned error.
xtrabackup: inititialize_service_handles suceeded
210825 02:38:00 Connecting to MySQL server host: localhost, user: XXXX, password: set, port: 3306, socket: not set
xtrabackup: Redo Log Archiving is not set up.
02:38:00 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x46000
xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x23d03fe]
xtrabackup(handle_fatal_signal+0x31b) [0x12c3c8b]
/lib64/libpthread.so.0(+0x12b20) [0x7fc61ece9b20]
/lib64/libc.so.6(+0x15d845) [0x7fc61ca3a845]
xtrabackup() [0x11adf12]
xtrabackup(add_plugin_options(std::vector<my_option, std::allocator<my_option> >*, MEM_ROOT*)+0x11a) [0x11afdba]
xtrabackup() [0xdbb09b]
xtrabackup(xtrabackup_backup_func()+0x336) [0xdc1666]
xtrabackup(main+0x1670) [0xd7fa60]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7fc61c9007b3]
xtrabackup(_start+0x2e) [0xdaf36e]

Please report a bug at https://jira.percona.com/projects/PXB
2 Likes
2

Hi @incursio

This error means xtrabackup is having an issue reading the keyring file. As a test, place keyring file inside the datadir and use --keyring-file-data.

Since you are running as root, I think you have something blocking access from xtrabackup binary to it, like SElinux or Apparmor.

1 Like
3

Yeah, I get that the issue is permission denied. But this happened even when I used keyring-file-data on a copy of the keyring file.

I’ve tried using --keyring-file-data with a full path/filename as well as copying it into the datadir and using just the filename as you suggested. I still get the same error.

It recognizes the client argument according to the log:

xtrabackup: recognized client arguments: --backup=1 .... --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin --keyring-file-data=/var/lib/mysql/mysql.keyring .....

But yet it keeps complaining about the key file that MySQL is actually configured to use:

210825 14:02:20 Added plugin 'keyring_file.so' to load list.

2021-08-25T14:02:20.746747Z 0 [ERROR] [MY-011370] [InnoDB] Plugin keyring_file reported: 'File '/mnt/mysql.keyring.vault/mysql.keyring' not found (OS errno 13 - Permission denied)'

2021-08-25T14:02:20.746822Z 0 [ERROR] [MY-011355] [InnoDB] Plugin keyring_file reported: 'keyring_file initialization failure. Please check if the keyring_file_data points to readable keyring file or keyring file can be created in the specified location. The keyring_file will stay unusable until correct path to the keyring file gets provided'

2021-08-25T14:02:20.746839Z 0 [ERROR] [MY-010202] [Server] Plugin 'keyring_file' init function returned error.
1 Like
4

@incursio

Right, I get now that the issue is during --backup. For backup we use mysql server as source of truth and pass its live configuration as argument - percona-xtrabackup/keyring_plugins.cc at percona-xtrabackup-8.0.25-17 · percona/percona-xtrabackup · GitHub

I would troubleshoot what is blocking xtrabackup binary by accessing keyring file on NFS.

1 Like
5

This just keeps getting stranger. According to lslocks, nothing is locking that file. I can hexdump the keyfile on the NFS mount as user mysql, but not as user root. I get the permission denied error. SELinux is not enabled, btw.

1 Like
6

Ok, I think I figured it out! I needed the no_root_squash option enabled for the NFS mount of the keyring,

2 Likes