Hi,
Cluster is not coming up with below error:
Receiver:
2023-04-18T18:05:06.031607Z 0 [Warning] [MY-000000] [Galera] 0.0 (XXXXX): State transfer to 1.0 (XXXXXX) failed: -1 (Operation not permitted)
2023-04-18T18:05:06.031665Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1216: Will never receive state. Need to abort.
2023-04-18T18:05:06.031679Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2023-04-18T18:05:06.031716Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2023-04-18T18:05:06.031862Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2023-04-18T18:05:06.531151Z 0 [Note] [MY-000000] [Galera] (94b7573e-bb47, ‘ssl://0.0.0.0:3304’) turning message relay requesting off
2023-04-18T18:05:07.035288Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,5c1c7395-85e4,25)
memb {
94b7573e-bb47,0
}
joined {
}
left {
}
partitioned {
5c1c7395-85e4,0
}
)
2023-04-18T18:05:07.035354Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 → 0
2023-04-18T18:05:07.035365Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2023-04-18T18:05:07.035545Z 0 [Note] [MY-000000] [Galera] gcomm: closed
2023-04-18T18:05:07.035573Z 0 [Note] [MY-000000] [Galera] /path/to/binaries//bin/mysqld: Terminated.
2023-04-18T18:05:07.035582Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2023-04-18T18:05:07.035587Z 0 [Note] [MY-000000] [WSREP] Terminating SST process
2023-04-18T18:05:07.037556Z 0 [ERROR] [MY-000000] [WSREP-SST] Removing /path/to/datadir//xtrabackup_galera_info file due to signal
Donor:
2023-04-18T18:05:05.190518Z 2 [Note] [MY-000000] [WSREP] Server status change synced → donor
2023-04-18T18:05:05.190532Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2023-04-18T18:05:05.191224Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘X.X.X.X:3303/xtrabackup_sst//1’ --socket ‘xxxxx.sock’ --datadir ‘/path/to/datadir/’ --basedir ‘/path/to/binaries/’ --plugindir ‘/path/to/binaries/lib/plugin/’ --defaults-file ‘/path/to/mycnf/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘8.0.31-23.2’ --binlog ‘mysqltest1t_b_binlog’ --gtid ‘45e1d077-dd70-11ed-a50b-73de40d8bbd9:89’ --bypass)
2023-04-18T18:05:05.226554Z 2 [Note] [MY-000000] [WSREP] DONOR thread signaled with 0
2023-04-18T18:05:05.279973Z 0 [Note] [MY-000000] [Galera] async IST sender starting to serve ssl://X.X.X.X:3305 sending 90-92, preload starts from 92
2023-04-18T18:05:05.281030Z 0 [Note] [MY-000000] [Galera] IST sender 90 → 92
2023-04-18T18:05:05.999415Z 23 [Warning] [MY-013360] [Server] Plugin sha256_password reported: ‘‘sha256_password’ is deprecated and will be removed in a future release. Please use caching_sha2_password instead’
2023-04-18T18:05:06.000588Z 0 [Note] [MY-000000] [WSREP-SST] ERROR 1045 (28000): Access denied for user ‘mysql.pxc.sst.user’@‘vm-17-182-5-155.us-east-1.compute.is.apple.com’ (using password: YES)
2023-04-18T18:05:06.002379Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1
2023-04-18T18:05:06.022374Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘X.X.X.X:3303/xtrabackup_sst//1’ --socket ‘/tmp/mysqld_mysqltest1t.sock’ --datadir ‘/path/to/datadir/’ --basedir ‘/path/to/binaries/’ --plugindir ‘/path/to/binaries/lib/plugin/’ --defaults-file ‘/path/to/mycnf/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘8.0.31-23.2’ --binlog ‘mysqltest1t_b_binlog’ --gtid ‘45e1d077-dd70-11ed-a50b-73de40d8bbd9:89’ --bypass: 1 (Operation not permitted)
2023-04-18T18:05:06.030423Z 0 [Note] [MY-000000] [Galera] SST sending failed: -1
2023-04-18T18:05:06.030455Z 0 [Note] [MY-000000] [WSREP] Server status change donor → joined
2023-04-18T18:05:06.030469Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2023-04-18T18:05:06.030603Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘X.X.X.X:3303/xtrabackup_sst//1’ --socket ‘/tmp/mysqld_mysqltest1t.sock’ --datadir ‘/path/to/datadir/’ --basedir ‘/path/to/binaries/’ --plugindir ‘/path/to/binaries/lib/plugin/’ --defaults-file ‘/path/to/mycnf/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘8.0.31-23.2’ --binlog ‘mysqltest1t_b_binlog’ --gtid ‘45e1d077-dd70-11ed-a50b-73de40d8bbd9:89’ --bypass
2023-04-18T18:05:06.031444Z 0 [Warning] [MY-000000] [Galera] 0.0 (X.X.X.X): State transfer to 1.0 (X.X.X.X) failed: -1 (Operation not permitted)
2023-04-18T18:05:06.031535Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED → JOINED (TO: 92)
ISSUE:
sst user is getting created over localhost however, while actual ssl, mysql connection is resolving to the hostname instead of localhost as below:
[Note] [MY-000000] [WSREP-SST] ERROR 1045 (28000): Access denied for user ‘mysql.pxc.sst.user’@‘goianallbgkqhh89889bniga.us-east-1.compute.is.xxxxx.com’ (using password: YES)