Hello there,
I have some problem with master key from keyring but the thing is that everything was working pretty fine till one of my nodes didn’t fallout from cluster.
I have 3 nodes in cluster: pxc1 pxc2 pxc3 (donor).
I tried to kill my cluster and reboot it. I did bootstrap my first node which is pxc3.
ERROR:
(Donor side)
nnoDB: Number of pools: 1
220714 10:10:38 Added plugin 'keyring_vault.so' to load list.
Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make s
ure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'
--------------------
2022-07-14T10:10:38.533023Z WSREP_SST: [ERROR] ******************************************************
2022-07-14T10:10:38.536893Z WSREP_SST: [ERROR] Cleanup after exit with status:22
2022-07-14T10:10:38.574309Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.200.0.41:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/my
sqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.36-39-57' '' --gtid '10ff16b5-b030-11ec-81ef-7e10ee3b675c:79099
47' : 22 (Invalid argument)
I have checked my keyring config on all machines and it’s same on all of them.
MY JOINER ERROR LOG:
2022-07-14T10:10:28.950425Z WSREP_SST: [INFO] ............Waiting for SST streaming to complete!
2022-07-14T10:10:38.530937Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************
2022-07-14T10:10:38.534428Z WSREP_SST: [ERROR] xtrabackup_checkpoints missing. xtrabackup/SST failed on DONOR. Check DONOR log
2022-07-14T10:10:38.537735Z WSREP_SST: [ERROR] ******************************************************
2022-07-14T10:10:38.541583Z WSREP_SST: [ERROR] Cleanup after exit with status:2
2022-07-14T10:10:38.577585Z 0 [Warning] WSREP: 1.0 (pxc3): State transfer to 0.0 (pxc1) failed: -22 (Invalid argument)
2022-07-14T10:10:38.577675Z 0 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():811: Will never receive state. Need to abort.
2022-07-14T10:10:38.577716Z 0 [Note] WSREP: gcomm: terminating thread
2022-07-14T10:10:38.577766Z 0 [Note] WSREP: gcomm: joining thread
2022-07-14T10:10:38.578051Z 0 [Note] WSREP: gcomm: closing backend
2022-07-14T10:10:39.043751Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.200.0.41' --datadir '/var/lib/mysql/' --defaults-file '/etc/
mysql/my.cnf' --defaults-group-suffix '' --parent '538357' --mysqld-version '5.7.36-39-57' '' : 2 (No such file or directory)
2022-07-14T10:10:39.043830Z 0 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2022-07-14T10:10:39.043857Z 0 [ERROR] WSREP: SST script aborted with error 2 (No such file or directory)
2022-07-14T10:10:39.043983Z 0 [ERROR] WSREP: SST failed: 2 (No such file or directory)
2022-07-14T10:10:39.044029Z 0 [ERROR] Aborting
2022-07-14T10:10:39.044047Z 0 [Note] WSREP: Signalling cancellation of the SST request.
2022-07-14T10:10:39.044097Z 0 [Note] WSREP: SST request was cancelled
2022-07-14T10:10:39.044141Z 0 [Note] Giving 2 client threads a chance to die gracefully
2022-07-14T10:10:39.044244Z 1 [Note] WSREP: Closing send monitor...
2022-07-14T10:10:39.044287Z 1 [Note] WSREP: Closed send monitor.
2022-07-14T10:10:39.583184Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,2d92652d,14)
memb {
2d92652d,0
}
joined {
}
left {
}
partitioned {
bcc54c5b,0
}
)
2022-07-14T10:10:39.583296Z 0 [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
2022-07-14T10:10:39.583592Z 0 [Note] WSREP: gcomm: closed
2022-07-14T10:10:39.583631Z 0 [Note] WSREP: /usr/sbin/mysqld: Terminated.
The thing is that I see that JOINER node successfully joins the cluster but when DONOR needs to send him data it fails.
Ports 3306, 4567, 4568 and 4444 are opened to all connections between my nodes.
My storage at donor side:
Filesystem Size Used Avail Use% Mounted on
udev 1,5G 0 1,5G 0% /dev
tmpfs 299M 35M 265M 12% /run
/dev/sda1 16G 13G 2,3G 85% /
tmpfs 1,5G 16K 1,5G 1% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 299M 0 299M 0% /run/user/1000
MYSQL CNF DONOR:
key_buffer_size = 32M
innodb_buffer_pool_size = 512M
innodb_buffer_pool_instances = 1
innodb_log_file_size = 32M
wsrep_provider=/usr/lib/libgalera_smm.so
wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://10.200.0.40,10.200.0.41,10.200.0.42
wsrep_node_name=pxc3
wsrep_node_address=10.200.0.42
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=sstuser:fJZBiGXtwyLyzG4k4Ttk6uRL9i
pxc_strict_mode=ENFORCING
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
``
**MYSQL CNF JOINER:**
key_buffer_size = 32M
innodb_buffer_pool_size = 512M
innodb_buffer_pool_instances = 1
innodb_log_file_size = 32M
wsrep_provider=/usr/lib/libgalera_smm.so
wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://10.200.0.41,10.200.0.40,10.200.0.42
wsrep_node_name=pxc1
wsrep_node_address=10.200.0.41
wsrep_sst_donor=pxc3
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=sstuser:fJZBiGXtwyLyzG4k4Ttk6uRL9i
pxc_strict_mode=ENFORCING
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
**MY KEYRING CNF**
vault_url = address
secret_mount_point = percona
secret_mount_point_version = AUTO
token = token
I put here dummy vault ur and dummy token just for privacy settings.
Does XTRABACKUP request some min storage to be executed?
I've never encountered with problem like this and it was working perfectly.
Is there maybe a case that vault autorotates his key ?
Thanks in advance,
Regards!