# Failing with keyring config **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Tags:** mysql, percona\ **Created:** [July 14, 2022, 10:39am UTC](https://forums.percona.com/t/failing-with-keyring-config/16562 "2022-07-14T10:39:33Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![MatzeyDevNinja](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@MatzeyDevNinja](https://forums.percona.com/u/MatzeyDevNinja)\ **Post date:** [July 14, 2022, 10:39am UTC](https://forums.percona.com/t/failing-with-keyring-config/16562/1 "2022-07-14T10:39:33Z")
Hello there, I have some problem with master key from keyring but the thing is that everything was working pretty fine till one of my nodes didn’t fallout from cluster. I have 3 nodes in cluster: pxc1 pxc2 pxc3 (donor). I tried to kill my cluster and reboot it. I did bootstrap my first node which is pxc3. **ERROR:** **(Donor side)** ```auto nnoDB: Number of pools: 1 220714 10:10:38 Added plugin 'keyring_vault.so' to load list. Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make s ure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.' -------------------- 2022-07-14T10:10:38.533023Z WSREP_SST: [ERROR] ****************************************************** 2022-07-14T10:10:38.536893Z WSREP_SST: [ERROR] Cleanup after exit with status:22 2022-07-14T10:10:38.574309Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.200.0.41:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/my sqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.36-39-57' '' --gtid '10ff16b5-b030-11ec-81ef-7e10ee3b675c:79099 47' : 22 (Invalid argument) ``` I have checked my keyring config on all machines and it’s same on all of them. **MY JOINER ERROR LOG:** ```auto 2022-07-14T10:10:28.950425Z WSREP_SST: [INFO] ............Waiting for SST streaming to complete! 2022-07-14T10:10:38.530937Z WSREP_SST: [ERROR] *******************FATAL ERROR********************** 2022-07-14T10:10:38.534428Z WSREP_SST: [ERROR] xtrabackup_checkpoints missing. xtrabackup/SST failed on DONOR. Check DONOR log 2022-07-14T10:10:38.537735Z WSREP_SST: [ERROR] ****************************************************** 2022-07-14T10:10:38.541583Z WSREP_SST: [ERROR] Cleanup after exit with status:2 2022-07-14T10:10:38.577585Z 0 [Warning] WSREP: 1.0 (pxc3): State transfer to 0.0 (pxc1) failed: -22 (Invalid argument) 2022-07-14T10:10:38.577675Z 0 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():811: Will never receive state. Need to abort. 2022-07-14T10:10:38.577716Z 0 [Note] WSREP: gcomm: terminating thread 2022-07-14T10:10:38.577766Z 0 [Note] WSREP: gcomm: joining thread 2022-07-14T10:10:38.578051Z 0 [Note] WSREP: gcomm: closing backend 2022-07-14T10:10:39.043751Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.200.0.41' --datadir '/var/lib/mysql/' --defaults-file '/etc/ mysql/my.cnf' --defaults-group-suffix '' --parent '538357' --mysqld-version '5.7.36-39-57' '' : 2 (No such file or directory) 2022-07-14T10:10:39.043830Z 0 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 2022-07-14T10:10:39.043857Z 0 [ERROR] WSREP: SST script aborted with error 2 (No such file or directory) 2022-07-14T10:10:39.043983Z 0 [ERROR] WSREP: SST failed: 2 (No such file or directory) 2022-07-14T10:10:39.044029Z 0 [ERROR] Aborting 2022-07-14T10:10:39.044047Z 0 [Note] WSREP: Signalling cancellation of the SST request. 2022-07-14T10:10:39.044097Z 0 [Note] WSREP: SST request was cancelled 2022-07-14T10:10:39.044141Z 0 [Note] Giving 2 client threads a chance to die gracefully 2022-07-14T10:10:39.044244Z 1 [Note] WSREP: Closing send monitor... 2022-07-14T10:10:39.044287Z 1 [Note] WSREP: Closed send monitor. 2022-07-14T10:10:39.583184Z 0 [Note] WSREP: Current view of cluster as seen by this node view (view_id(NON_PRIM,2d92652d,14) memb { 2d92652d,0 } joined { } left { } partitioned { bcc54c5b,0 } ) 2022-07-14T10:10:39.583296Z 0 [Note] WSREP: Current view of cluster as seen by this node view ((empty)) 2022-07-14T10:10:39.583592Z 0 [Note] WSREP: gcomm: closed 2022-07-14T10:10:39.583631Z 0 [Note] WSREP: /usr/sbin/mysqld: Terminated. ``` The thing is that I see that JOINER node successfully joins the cluster but when DONOR needs to send him data it fails. Ports 3306, 4567, 4568 and 4444 are opened to all connections between my nodes. **My storage at donor side:** ```auto Filesystem Size Used Avail Use% Mounted on udev 1,5G 0 1,5G 0% /dev tmpfs 299M 35M 265M 12% /run /dev/sda1 16G 13G 2,3G 85% / tmpfs 1,5G 16K 1,5G 1% /dev/shm tmpfs 5,0M 0 5,0M 0% /run/lock tmpfs 299M 0 299M 0% /run/user/1000 ``` **MYSQL CNF DONOR:** ```auto key_buffer_size = 32M innodb_buffer_pool_size = 512M innodb_buffer_pool_instances = 1 innodb_log_file_size = 32M wsrep_provider=/usr/lib/libgalera_smm.so wsrep_cluster_name=pxc-cluster wsrep_cluster_address=gcomm://10.200.0.40,10.200.0.41,10.200.0.42 wsrep_node_name=pxc3 wsrep_node_address=10.200.0.42 wsrep_sst_method=xtrabackup-v2 wsrep_sst_auth=sstuser:fJZBiGXtwyLyzG4k4Ttk6uRL9i pxc_strict_mode=ENFORCING binlog_format=ROW default_storage_engine=InnoDB innodb_autoinc_lock_mode=2 character-set-client-handshake = FALSE character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci `` **MYSQL CNF JOINER:** ``` key\_buffer\_size = 32M innodb\_buffer\_pool\_size = 512M innodb\_buffer\_pool\_instances = 1 innodb\_log\_file\_size = 32M wsrep\_provider=/usr/lib/libgalera\_smm.so wsrep\_cluster\_name=pxc-cluster wsrep\_cluster\_address=gcomm://10.200.0.41,10.200.0.40,10.200.0.42 wsrep\_node\_name=pxc1 wsrep\_node\_address=10.200.0.41 wsrep\_sst\_donor=pxc3 wsrep\_sst\_method=xtrabackup-v2 wsrep\_sst\_auth=sstuser:fJZBiGXtwyLyzG4k4Ttk6uRL9i pxc\_strict\_mode=ENFORCING binlog\_format=ROW default\_storage\_engine=InnoDB innodb\_autoinc\_lock\_mode=2 character-set-client-handshake = FALSE character-set-server = utf8mb4 collation-server = utf8mb4\_unicode\_ci ```auto **MY KEYRING CNF** ``` vault\_url = address secret\_mount\_point = percona secret\_mount\_point\_version = AUTO token = token ```auto I put here dummy vault ur and dummy token just for privacy settings. Does XTRABACKUP request some min storage to be executed? I've never encountered with problem like this and it was working perfectly. Is there maybe a case that vault autorotates his key ? Thanks in advance, Regards! ``` ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [August 2, 2022, 3:04pm UTC](https://forums.percona.com/t/failing-with-keyring-config/16562/2 "2022-08-02T15:04:27Z")
> [@MatzeyDevNinja](#): > > `keyring_vault initialization failure.` Can you look at Vault’s logs and see if there are any attempts? Anything else in the mysql log on why the keyring plugin failed to initialize? ---
**Author:** ![MatzeyDevNinja](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@MatzeyDevNinja](https://forums.percona.com/u/MatzeyDevNinja)\ **Post date:** [August 5, 2022, 5:27pm UTC](https://forums.percona.com/t/failing-with-keyring-config/16562/3 "2022-08-05T17:27:41Z")
Now getting; ```auto InnoDB: Encryption information in datafile: ./stagingapidb/audit.ibd can't be decrypted, please check if a keyring plugin is loaded and initialized successfully. InnoDB: Failed to decrypt table ./stagingapidb/audit.ibd with space id 48. Will check if encrytion key has been parsed at the end of backup. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./stagingapidb/claim.ibd can't be decrypted, please check if a keyring plugin is loaded and initialized successfully. InnoDB: Failed to decrypt table ./stagingapidb/claim.ibd with space id 50. Will check i ``` Note that I didn’t change anything and it was all doing perfect. If I don’t manage to solve this, I think I will try to get a new node and try to set him as donor. Afterwards I will try to sync these 2 that won’t sync with my real donor. If I get that cluster in operative mode, I will backup all data from real node and import it to my cloned node. That’s my approach because I really don’t know what happened here, that I can’t get the keyring working fine, although everything was working fine. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [August 5, 2022, 11:31pm UTC](https://forums.percona.com/t/failing-with-keyring-config/16562/4 "2022-08-05T23:31:58Z")
These errors are expected since you can’t access the master key. Did you investigate the connection? ---
**Author:** ![MatzeyDevNinja](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@MatzeyDevNinja](https://forums.percona.com/u/MatzeyDevNinja)\ **Post date:** [August 5, 2022, 11:37pm UTC](https://forums.percona.com/t/failing-with-keyring-config/16562/5 "2022-08-05T23:37:50Z")
I cannot find my vault log, can master key rotate on his own? I didn’t touch anything related to Vault. How could this happen ?