Problem when trying creating backup in encrypted database

Benedito_Marques · September 9, 2021, 4:07pm

Hello.

I’m using Percona Mysql Server 8.0.25 and Xtrabackup 8.0.25 (O.S: Debian 10). The database is in data-at-rest mode, with configuration bellow:

early-plugin-load="keyring_vault=keyring_vault.so"
loose-keyring_vault_config="/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf"
binlog_encryption=ON
binlog_rotate_encryption_master_key_at_startup=OFF
default_table_encryption=ON
encrypt_tmp_files=ON
innodb_encrypt_online_alter_logs=ON
table_encryption_privilege_check=ON
innodb_redo_log_encrypt=ON
innodb_undo_log_encrypt=ON
innodb_temp_tablespace_encrypt=ON

$ cat /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf
vault_url = https://10.128.0.17:8200
secret_mount_point = secrets/google/voice_abc/-80
secret_mount_point_version = 1
token = xxxxxxxx
vault_ca = /opt/vitess/examples/local/vault_configs/vault_ca.crt

Encryption works fine, but when a try to make a backup using xtrabackup I’m getting error bellow:

suporte@mysql-02:/opt/vitess/examples/local$ xtrabackup --backup --datadir=/vt/vtdataroot/vt_0000000101/data/ --target-dir=/tmp/BKP/ -u xtrabackup_vitess -p -S /vt/vtdataroot/vt_0000000101/mysql.sock --keyring-vault-config=/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf
xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=/vt/vtdataroot/vt_0000000101/data/
xtrabackup: recognized client arguments: --backup=1 --target-dir=/tmp/BKP/ --user=xtrabackup_vitess --password --socket=/vt/vtdataroot/vt_0000000101/mysql.sock
Enter password:
xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
210909 16:04:34  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vt_0000000101/mysql.sock' as 'xtrabackup_vitess'  (using password: YES).
210909 16:04:34  version_check Connected to MySQL server
210909 16:04:34  version_check Executing a version check against the server...
210909 16:04:34  version_check Done.
210909 16:04:34 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock
Using server version 8.0.25-15
210909 16:04:34 Executing LOCK TABLES FOR BACKUP...
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /vt/vtdataroot/vt_0000000101/data/
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = /vt/vtdataroot/vt_0000000101/innodb/data
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = /vt/vtdataroot/vt_0000000101/innodb/logs
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 50331648
Number of pools: 1
210909 16:04:34 Added plugin 'keyring_vault.so' to load list.
2021-09-09T16:04:34.076080Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'
xtrabackup: inititialize_service_handles suceeded
Encryption can't find master key, please check the keyring is loaded.
Cannot read the encryption information in log file header, please check if keyring is loaded.
210909 16:04:34 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock
xtrabackup: Redo Log Archiving is not set up.
xtrabackup: Can't create/write to file '/tmp/BKP/xtrabackup_logfile' (OS errno 17 - File exists)
xtrabackup: error: failed to open the target stream for 'xtrabackup_logfile'.
suporte@mysql-02:/opt/vitess/examples/local$

Vault is working properly and is accessible, and configuration file is readable.

Marcelo_Altmann · September 9, 2021, 5:02pm

Hi there,

There are two problems on your output

Error loading keyring

2021-09-09T16:04:34.076080Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'

Can you please run the backup again passing --xtrabackup-plugin-dir=/path/to/xtrabackup/lib/plugin . Please note that xtrabackup keyring_vault.so is different from the one shipped with percona server.

target-dir already has a backup on it:

xtrabackup: Can't create/write to file '/tmp/BKP/xtrabackup_logfile' (OS errno 17 - File exists)

Please either clean the folder or point your backup to an empty folder.

Benedito_Marques · September 9, 2021, 5:14pm

Thanks for your attention @Marcelo_Altmann . Same error putting the plugin directory option:

suporte@mysql-02:/opt/vitess/examples/local$ sudo dpkg -L percona-xtrabackup-80 | grep keyring_vault.so
/usr/lib/xtrabackup/plugin/keyring_vault.so

suporte@mysql-02:/opt/vitess/examples/local$ locate keyring_vault.so
/usr/lib/mysql/plugin/keyring_vault.so
/usr/lib/mysql/plugin/debug/keyring_vault.so
/usr/lib/xtrabackup/plugin/keyring_vault.so
suporte@mysql-02:/opt/vitess/examples/local$ ls -l /tmp/BKP/
total 0



suporte@mysql-02:/opt/vitess/examples/local$ xtrabackup --backup --datadir=/vt/vtdataroot/vt_0000000101/data/ --target-dir=/tmp/BKP/ -u xtrabackup_vitess -p -S /vt/vtdataroot/vt_0000000101/mysql.sock --keyring-vault-config=/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf --xtrabackup-plugin-dir=/usr/lib/xtrabackup/plugin/
xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=/vt/vtdataroot/vt_0000000101/data/
xtrabackup: recognized client arguments: --backup=1 --target-dir=/tmp/BKP/ --user=xtrabackup_vitess --password --socket=/vt/vtdataroot/vt_0000000101/mysql.sock --xtrabackup-plugin-dir=/usr/lib/xtrabackup/plugin/
Enter password:
xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
210909 17:13:27  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vt_0000000101/mysql.sock' as 'xtrabackup_vitess'  (using password: YES).
210909 17:13:27  version_check Connected to MySQL server
210909 17:13:27  version_check Executing a version check against the server...
210909 17:13:27  version_check Done.
210909 17:13:27 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock
Using server version 8.0.25-15
210909 17:13:27 Executing LOCK TABLES FOR BACKUP...
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /vt/vtdataroot/vt_0000000101/data/
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = /vt/vtdataroot/vt_0000000101/innodb/data
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = /vt/vtdataroot/vt_0000000101/innodb/logs
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 50331648
Number of pools: 1
210909 17:13:27 Added plugin 'keyring_vault.so' to load list.
2021-09-09T17:13:27.695838Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'
xtrabackup: inititialize_service_handles suceeded
Encryption can't find master key, please check the keyring is loaded.
Cannot read the encryption information in log file header, please check if keyring is loaded.
210909 17:13:27 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock
xtrabackup: Redo Log Archiving is not set up.
Encryption algorithm support missing: N
InnoDB: Assertion failure: redo_log.cc:161:err == DB_SUCCESS
InnoDB: thread 139901056161792InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.percona.com/projects/PXB.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:13:27 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x46000
xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x558b0e9d335e]
xtrabackup(handle_fatal_signal+0x31b) [0x558b0d8bf9bb]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f3d4163a730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b) [0x7f3d40c557bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121) [0x7f3d40c40535]
xtrabackup(+0xe2285a) [0x558b0d32485a]
xtrabackup(+0xefab88) [0x558b0d3fcb88]
xtrabackup(Redo_Log_Reader::read_logfile(bool, bool*)+0x7f) [0x558b0d3fce7f]
xtrabackup(Redo_Log_Data_Manager::copy_once(bool, bool*)+0x3f) [0x558b0d3fdb6f]
xtrabackup(Redo_Log_Data_Manager::start()+0x9d) [0x558b0d3fdd9d]
xtrabackup(xtrabackup_backup_func()+0x4e0) [0x558b0d3be9e0]
xtrabackup(main+0x1670) [0x558b0d37cc30]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f3d40c4209b]
xtrabackup(_start+0x2a) [0x558b0d3ac53a]

Please report a bug at https://jira.percona.com/projects/PXB
suporte@mysql-02:/opt/vitess/examples/local$

Marcelo_Altmann · September 9, 2021, 6:05pm

ok, please run your backup with strace and send me the output:

strace -f -s8192 -ttt -o/tmp/xtrabackup.trace xtrabackup --backup ...

Benedito_Marques · September 9, 2021, 6:40pm

OK, this is the output of log strace command:

https://fortics-my.sharepoint.com/:u:/g/personal/benedito_fortics_com_br/Ec0e_f8983JIqg3tFIKkXUMBhfUOAUzkFD5srqneRok3Mw?e=bxwjYz

Marcelo_Altmann · September 10, 2021, 12:03pm

Bom Dia @Benedito_Marques

I would like to ask you two things:

Can you please raise a JIRA ticket for what you have encountered ?
Can you please run a test :

Copy /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf to a different file
Edit the nes file and remove option secret_mount_point_version
Take the backup again passing the new vault.cnf file.

Please make sure to restore it too and check if you can access encrypted tables after the restore.
Let us know the result.

Benedito_Marques · September 10, 2021, 12:45pm

Bom dia Marcelo!
OK, I’ll test this now and give you an feedback soon.

Benedito_Marques · September 10, 2021, 2:58pm

@Marcelo_Altmann , Issue created on JIRA:
https://jira.percona.com/browse/PXB-2608

The result of the suggested test was the same error. The result and an new strace using the new conf file without secret_mount_point_version parameter was put on JIRA.

Topic		Replies	Views
XtraBackup fails when using keyring vault for encryption Percona XtraBackup	3	692	February 2, 2022
Percona-xtrabackup-2.4.20 Unable to prepare (--apply-log) for an encrypted database	0	923	May 27, 2022
Mysql backup encryption Percona XtraDB Cluster 8.x mysql , closed-no-reply	0	451	February 14, 2022
InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. Percona XtraBackup	0	2934	August 22, 2019
Could not parse Vault Server response Percona XtraBackup	7	166	June 5, 2024

Problem when trying creating backup in encrypted database

Related topics