Hello,
after os update my mysqld crashes after it successfuly receives state from another node of percona xtradb cluster and joins the cluster.
Versions (they are identical on all nodes of the cluster)
mysql -V
mysql Ver 14.14 Distrib 5.6.51-91.0, for Linux (x86_64) using 6.2
cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
after systemctl mysql start command finishes I can see in log:
2022-05-05 22:47:22 3180 [Note] WSREP: SST received: 49799465-2157-11e8-9d5f-fa6bdff6d1ef:5629429893
2022-05-05 22:47:22 3180 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘5.6.51-91.0-56’ socket: ‘/mysql/mysql/mysql.sock’ port: 3307 Percona XtraDB Cluster (GPL), Release rel91.0, Revision c714694, WSREP version 28.46, wsrep_28.46
2022-05-05 22:47:22 3180 [Note] WSREP: 0.0 (hostname): State transfer from 1.0 (hostname) complete.
2022-05-05 22:47:22 3180 [Note] WSREP: Shifting JOINER → JOINED (TO: 5629449386)
in status table:
| wsrep_cluster_size | 3 |
±-----------------------------±------------------------------------------------------------+
74 rows in set (0.00 sec)
but suddenly:
mysql> show status like ‘%ws%’;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect…
ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (111)
ERROR:
Can’t connect to the server
mysql> Bye
Mysqld log:
2022-05-05 22:48:12 3180 [Note] WSREP: Deleted page /mysql/mysql/gcache.page.000023
2022-05-05 22:48:15 3180 [Note] WSREP: Deleted page /mysql/mysql/gcache.page.000024
20:48:19 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://jira.percona.com/projects/PXC/issues
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=10
max_threads=153
thread_count=14
connection_count=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69151 K bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x92142b]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x686dc1]
/usr/lib64/libpthread.so.0(+0xf630)[0x7fbe5eb7e630]
/usr/lib64/libgalera_smm.so(+0x22006)[0x7fbe3d1b4006]
/usr/lib64/libgalera_smm.so(+0x4c875)[0x7fbe3d1de875]
/usr/lib64/libgalera_smm.so(+0x160240)[0x7fbe3d2f2240]
/usr/lib64/libgalera_smm.so(+0x160e49)[0x7fbe3d2f2e49]
/usr/lib64/libpthread.so.0(+0x7ea5)[0x7fbe5eb76ea5]
/usr/lib64/libc.so.6(clone+0x6d)[0x7fbe5cd43b0d]
You may download the Percona XtraDB Cluster operations manual by visiting
Percona XtraDB Cluster – The MySQL Clustering Solution. You may find information
in the manual which will help you identify the cause of the crash.
Any recomendations? I was thinking about just cloning another node, clean it up, reconfigure and try to join cluster…or maybe restore from backup (the broken node) and try to update & join cluster again…