I have a one-node Xtra DB cluster running and it will occasionally crash and needs to be restarted.
Here’s what journalctl -xe shows:
Jun 06 17:44:48 hb-db-masking systemd[1]: mysql@bootstrap.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 06 17:44:48 hb-db-masking mysql-systemd[10396]: /usr/bin/mysql-systemd: line 233: kill: (6908) - No such process
Jun 06 17:44:48 hb-db-masking mysql-systemd[10396]: WARNING: mysql already dead
Jun 06 17:44:48 hb-db-masking mysql-systemd[10396]: ERROR! Stale PID file: /var/run/mysqld/mysqld.pid
Jun 06 17:44:48 hb-db-masking systemd[1]: mysql@bootstrap.service: Control process exited, code=exited status=3
Jun 06 17:44:48 hb-db-masking systemd[1]: mysql@bootstrap.service: Failed with result ‘exit-code’.
from /var/log/mysql/error.log:
2022-06-06T17:44:46.871719Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-06-06T17:44:46.959719Z 0 [Warning] [MY-000000] [Galera] Handshake failed: peer did not return a certificate
2022-06-06T17:44:47.988049Z 0 [Warning] [MY-000000] [Galera] Handshake failed: http request
terminate called after throwing an instance of ‘boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorstd::system_error >’
what(): remote_endpoint: Transport endpoint is not connected
2022-06-06T17:44:48.045571Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
17:44:48 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Build ID: 712cedd01992d40a9e0f9779e6425e7089c32870
Server Version: 8.0.27-18.1 Percona XtraDB Cluster (GPL), Release rel18, Revision ac35177, WSREP version 26.4.3, wsrep_26.4.3
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x55ca77ff922d]
/usr/sbin/mysqld(handle_fatal_signal+0x3c3) [0x55ca77201063]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd35b97e980]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7fd359a83e87]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7fd359a857f1]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c957) [0x7fd35a478957]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae6) [0x7fd35a47eae6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b21) [0x7fd35a47eb21]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92d54) [0x7fd35a47ed54]
/usr/lib/galera4/libgalera_smm.so(+0x6fd79) [0x7fd34d7d3d79]
/usr/lib/galera4/libgalera_smm.so(+0x6fe97) [0x7fd34d7d3e97]
/usr/lib/galera4/libgalera_smm.so(+0x7f481) [0x7fd34d7e3481]
/usr/lib/galera4/libgalera_smm.so(+0x82991) [0x7fd34d7e6991]
/usr/lib/galera4/libgalera_smm.so(+0x89759) [0x7fd34d7ed759]
/usr/lib/galera4/libgalera_smm.so(+0x79324) [0x7fd34d7dd324]
/usr/lib/galera4/libgalera_smm.so(+0x683ad) [0x7fd34d7cc3ad]
/usr/lib/galera4/libgalera_smm.so(+0x16e988) [0x7fd34d8d2988]
/usr/lib/galera4/libgalera_smm.so(+0x19deca) [0x7fd34d901eca]
/usr/lib/galera4/libgalera_smm.so(+0x19e1d2) [0x7fd34d9021d2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fd35b9736db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fd359b6661f]
You may download the Percona XtraDB Cluster operations manual by visiting
Percona XtraDB Cluster – The MySQL Clustering Solution. You may find information
in the manual which will help you identify the cause of the crash.
Log of wsrep recovery (–wsrep-recover):
INFO: WSREP: Running position recovery with --log_error=‘/var/lib/mysql/wsrep_recovery_verbose.q2XKMm’ --pid-file=‘/var/lib/mysql/hb-db-masking-recover.pid’
INFO: WSREP: Recovered position a34b8e50-cc71-11ec-adbc-0f55e3e4c669:897772
Not sure where to look next?