Hi,
I am using 3 nodes PXC DB cluster 8.0 on ubuntu focal. Today all my three nodes of cluster crashed with below trace in error.log.
2023-11-16T15:36:44.560149+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: wrong version number
2023-11-16T15:36:45.589236+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2023-11-16T15:36:45.637822+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: version too low
2023-11-16T15:36:45.698329+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unexpected message
2023-11-16T15:36:45.733086+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2023-11-16T15:36:45.824820+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:36:45.920617+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:36:53.490334+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:36:55.713226+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: wrong version number
2023-11-16T15:36:57.966144+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:37:44.496025+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:37:56.534731+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: http request
2023-11-16T15:37:56.599033+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:38:03.044670+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:38:03.110605+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:38:05.401998+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: wrong version number
2023-11-16T15:38:13.962777+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:38:14.372915+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:39:17.003167+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2023-11-16T15:39:17.123032+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unknown protocol
2023-11-16T15:39:18.166048+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unknown protocol
2023-11-16T15:39:19.209901+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: version too low
2023-11-16T15:39:20.252446+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: version too low
2023-11-16T15:39:21.296296+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2023-11-16T15:39:21.356207+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2023-11-16T15:39:22.401075+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2023-11-16T15:39:23.446180+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2023-11-16T15:39:24.549232+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: no shared cipher
2023-11-16T15:39:29.748841+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: unsupported protocol
2023-11-16T15:39:31.806620+05:00 0 [Warning] [MY-000000] [Galera] Handshake failed: version too low
terminate called after throwing an instance of 'boost::wrapexcept<std::system_error>'
what(): remote_endpoint: Transport endpoint is not connected
2023-11-16T15:39:31.860564+05:00 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
10:39:31 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Build ID: 2165eff2f1909b2f032b76b423382ec097755ae3
Server Version: 8.0.27-18.1 Percona XtraDB Cluster (GPL), Release rel18, Revision ac35177, WSREP version 26.4.3, wsrep_26.4.3
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55d65334cea1]
/usr/sbin/mysqld(handle_fatal_signal+0x393) [0x55d65236bf63]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fe24ec83420]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7fe24e35b00b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7fe24e33a859]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e911) [0x7fe24e712911]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa38c) [0x7fe24e71e38c]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3f7) [0x7fe24e71e3f7]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa6a9) [0x7fe24e71e6a9]
/usr/lib/galera4/libgalera_smm.so(+0x1e569) [0x7fe24243c569]
/usr/lib/galera4/libgalera_smm.so(+0xa050a) [0x7fe2424be50a]
/usr/lib/galera4/libgalera_smm.so(+0xa20ab) [0x7fe2424c00ab]
/usr/lib/galera4/libgalera_smm.so(+0xa4428) [0x7fe2424c2428]
/usr/lib/galera4/libgalera_smm.so(+0xaa9d3) [0x7fe2424c89d3]
/usr/lib/galera4/libgalera_smm.so(+0x9b707) [0x7fe2424b9707]
/usr/lib/galera4/libgalera_smm.so(+0x893a2) [0x7fe2424a73a2]
/usr/lib/galera4/libgalera_smm.so(+0x193258) [0x7fe2425b1258]
/usr/lib/galera4/libgalera_smm.so(+0x1bc5ae) [0x7fe2425da5ae]
/usr/lib/galera4/libgalera_smm.so(+0x1bc6d6) [0x7fe2425da6d6]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fe24ec77609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fe24e437133]
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Log of wsrep recovery (--wsrep-recover):
INFO: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery_verbose.QOzOOs' --pid-file='/var/lib/mysql/pxc-db01-lhr01.rapid.pk-recover.pid'
INFO: WSREP: Recovered position 220ad56d-dd91-11ec-a286-eab256e61c60:200764257
Below are the packages versions that are installed.
# dpkg -l | grep percona
ii percona-release 1.0-27.generic all Package to install Percona gpg key and APT repos
ii percona-xtradb-cluster 1:8.0.27-18-1.focal amd64 Percona XtraDB Cluster with Galera
ii percona-xtradb-cluster-client 1:8.0.27-18-1.focal amd64 Percona XtraDB Cluster database client binaries
ii percona-xtradb-cluster-common 1:8.0.27-18-1.focal amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-server 1:8.0.27-18-1.focal amd64 Percona XtraDB Cluster database server binaries