Update:
Hi @Pep_Pla,
I found some terrible issues with my both PXC clusters, somehow today all nodes on both cluster crashed when I tried to search the logs and found handshake errors because of which services on all nodes crashed and it is also happening on the test env where aysn replication is not configured between PXC clusters.
I am not sure of the reason but suspect this is because of Client PEM entries ssl-ca,ssl-cert and ssl-key under [client] section of my.cnf file. So I have removed these entries and also added require_secure_transport=ON and socket.ssl=ON in wsrep_provider_options.
Note: - Also adding require_secure_transport=ON to my my.cnf broke async replication between pxc clusters and was able to fix it by alter replication user in master with option REQUIRE SSL, and changing slave config command to following
change master to master_host=‘xx.xx.xx.xx’, master_user=‘replica_user’, master_password=‘xxxxxxxxxx’, master_log_file=‘binlog.0000xx’, master_log_pos=xxxx, SOURCE_SSL=1, MASTER_SSL_CA=‘/path/ca.pem’, MASTER_SSL_CERT=‘/path/client-cert.pem’, MASTER_SSL_KEY=‘/path/client-key.pem’;
Also attaching my my.cnf and error logs I have found.
I have also noticed these HANDSHAKE failed errors occur after 24 hrs of starting mysql service and before these errors replication and cluster works fine. Please Help.
2022-10-05T19:09:08.570239Z 2 [Note] [MY-000000] [Galera] Non-primary view
2022-10-05T19:09:08.570258Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> connected
2022-10-05T19:09:08.570282Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-10-05T19:09:49.164600Z 0 [Note] [MY-000000] [Galera] (c4178a5a-a826, 'ssl://0.0.0.0:4567') reconnecting to 524f1316-a18e (ssl://XX.XX.XX.XX:4567), attempt 30
2022-10-05T19:10:21.372476Z 0 [Warning] [MY-000000] [Galera] Handshake failed: wrong version number
2022-10-05T19:10:34.177166Z 0 [Note] [MY-000000] [Galera] (c4178a5a-a826, 'ssl://0.0.0.0:4567') reconnecting to 524f1316-a18e (ssl://XX.XX.XX.XX:4567), attempt 60
2022-10-05T19:10:36.123405Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2022-10-05T19:10:36.427602Z 0 [Warning] [MY-000000] [Galera] Handshake failed: version too low
2022-10-05T19:10:36.731975Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unexpected message
2022-10-05T19:10:36.890275Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2022-10-05T19:10:37.347029Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:37.796862Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:38.263682Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:38.425001Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:38.610108Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:38.610352Z 0 [Warning] [MY-000000] [Galera] Handshake failed: wrong version number
2022-10-05T19:10:38.766126Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:40.415602Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:40.416064Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unknown protocol
2022-10-05T19:10:40.730927Z 0 [Warning] [MY-000000] [Galera] Handshake failed: http request
2022-10-05T19:10:41.012670Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:41.013066Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unknown protocol
2022-10-05T19:10:41.477549Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:41.645134Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-10-05T19:10:41.645607Z 0 [Warning] [MY-000000] [Galera] Handshake failed: version too low
2022-10-05T19:10:41.993786Z 0 [Warning] [MY-000000] [Galera] Handshake failed: version too low
2022-10-05T19:10:42.408583Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2022-10-05T19:10:42.701874Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2022-10-05T19:10:43.997672Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2022-10-05T19:10:45.291258Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2022-10-05T19:10:46.923371Z 0 [Warning] [MY-000000] [Galera] Handshake failed: no shared cipher
2022-10-05T19:10:46.926175Z 0 [Warning] [MY-000000] [Galera] Handshake failed: no shared cipher
2022-10-05T19:10:46.937637Z 0 [Warning] [MY-000000] [Galera] Handshake failed: no shared cipher
2022-10-05T19:10:49.244751Z 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2022-10-05T19:11:18.696780Z 0 [Note] [MY-000000] [Galera] (c4178a5a-a826, 'ssl://0.0.0.0:4567') reconnecting to 524f1316-XXX (ssl://XX.XX.XX.XX:4567), attempt 90
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >'
what(): remote_endpoint: Transport endpoint is not connected
2022-10-05T19:11:32.388564Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
19:11:32 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Build ID: 5aaeb8aff2f9757ae471361dbf4fa4ba945f6104
Server Version: 8.0.28-19.1 Percona XtraDB Cluster (GPL), Release rel19, Revision f544540, WSREP version 26.4.3, wsrep_26.4.3
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Thanks