Hello there,
we upgraded our 3 Node Percona XtraDB Cluster to version 8.0.23 and all nodes are crashing randomly with the following errors:
2021-07-12T11:58:27.514216Z 1085 [Note] [MY-000000] [WSREP] Initiating SST cancellation
11:58:27 UTC - mysqld got signal 8 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Build ID: fc1c7dd6988d86d85e7a5077c4b3c504e22845d8
Server Version: 8.0.23-14.1 Percona XtraDB Cluster (GPL), Release rel14, Revision d3b9a1d, WSREP version 26.4.3, wsrep_26.4.3
Thread pointer: 0x7f45a0ba5c30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f50d415dd10 thread_stack 0x46000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x562b5cdf47d1]
/usr/sbin/mysqld(handle_fatal_signal+0x393) [0x562b5bdbc093]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f51290973c0]
/usr/sbin/mysqld(my_rnd(rand_struct*)+0x18) [0x562b5ce03d78]
/usr/sbin/mysqld(Item_func_mul::real_op()+0x27) [0x562b5b9d3007]
/usr/sbin/mysqld(Item_func_numhybrid::val_real()+0x46) [0x562b5b9d3696]
/usr/sbin/mysqld(Item_func_round::real_op()+0x23) [0x562b5b9e6173]
/usr/sbin/mysqld(Item_func_numhybrid::val_str(String*)+0x4e) [0x562b5b9d34ae]
/usr/sbin/mysqld(Item_func_concat::val_str(String*)+0xb9) [0x562b5ba2b979]
/usr/sbin/mysqld(Item_func_sha::val_str_ascii(String*)+0x39) [0x562b5ba1cd59]
/usr/sbin/mysqld(Item_str_func::val_str_from_val_str_ascii(String*, String*)+0x8a) [0x562b5ba2b25a]
/usr/sbin/mysqld(Item::save_in_field_inner(Field*, bool)+0x91) [0x562b5b951361]
/usr/sbin/mysqld(Item::save_in_field(Field*, bool)+0x57) [0x562b5b96a737]
/usr/sbin/mysqld(sp_eval_expr(THD*, Field*, Item**)+0xbe) [0x562b5bb9b9ce]
/usr/sbin/mysqld(Query_dumpvar::send_data(THD*, mem_root_deque<Item*> const&)+0x138) [0x562b5bb7fc38]
/usr/sbin/mysqld(SELECT_LEX_UNIT::ExecuteIteratorQuery(THD*)+0x2ed) [0x562b5bd3623d]
/usr/sbin/mysqld(SELECT_LEX_UNIT::execute(THD*)+0x33) [0x562b5bd36583]
/usr/sbin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0x32b) [0x562b5bcb725b]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x535) [0x562b5bcc0fc5]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x55bd) [0x562b5bc5ea9d]
/usr/sbin/mysqld(sp_instr_stmt::exec_core(THD*, unsigned int*)+0x53) [0x562b5bbad7e3]
/usr/sbin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x19b) [0x562b5bbaf77b]
/usr/sbin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0x9e) [0x562b5bbb06fe]
/usr/sbin/mysqld(sp_instr_stmt::execute(THD*, unsigned int*)+0xf2) [0x562b5bbb2062]
/usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x61d) [0x562b5bba532d]
/usr/sbin/mysqld(sp_head::execute_trigger(THD*, MYSQL_LEX_CSTRING const&, MYSQL_LEX_CSTRING const&, GRANT_INFO*)+0x2a5) [0x562b5bba64d5]
/usr/sbin/mysqld(Trigger::execute(THD*)+0x10c) [0x562b5bd816bc]
/usr/sbin/mysqld(Trigger_chain::execute_triggers(THD*)+0x20) [0x562b5bd82ac0]
/usr/sbin/mysqld(Table_trigger_dispatcher::process_triggers(THD*, enum_trigger_event_type, enum_trigger_action_time_type, bool)+0x4a) [0x562b5bd7b94a]
/usr/sbin/mysqld(Sql_cmd_update::update_single_table(THD*)+0x23a5) [0x562b5bd41995]
/usr/sbin/mysqld(Sql_cmd_update::execute_inner(THD*)+0xd5) [0x562b5bd41c55]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x535) [0x562b5bcc0fc5]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x14a9) [0x562b5bc5a989]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x52b) [0x562b5bc60dbb]
/usr/sbin/mysqld(+0x128e2f7) [0x562b5bc612f7]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x38c7) [0x562b5bc65b17]
/usr/sbin/mysqld(do_command(THD*)+0x261) [0x562b5bc66231]
/usr/sbin/mysqld(+0x13d8da0) [0x562b5bdabda0]
/usr/sbin/mysqld(+0x29a9972) [0x562b5d37c972]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f512908b609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f512884c293]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f456475d658): SELECT SHA1(CONCAT(ROUND(RAND()*1000000, 0), NOW())) INTO uid
Connection ID (thread ID): 1085
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
We then removed PXC and installed MySQL 8.0.25 on one of the nodes and the crash didnt happen again. It seems there is combination of a bug in the 8.0.23 version and our application behaviour which causes the crashs.
Did someone else encounter this issue too?
In the meantime I would suggest to skip this version completly.
BR,
Johannes