PXC 8 MySQL crashing with version 8.0.23

Hello there,

we upgraded our 3 Node Percona XtraDB Cluster to version 8.0.23 and all nodes are crashing randomly with the following errors:

2021-07-12T11:58:27.514216Z 1085 [Note] [MY-000000] [WSREP] Initiating SST cancellation
11:58:27 UTC - mysqld got signal 8 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Build ID: fc1c7dd6988d86d85e7a5077c4b3c504e22845d8
Server Version: 8.0.23-14.1 Percona XtraDB Cluster (GPL), Release rel14, Revision d3b9a1d, WSREP version 26.4.3, wsrep_26.4.3

Thread pointer: 0x7f45a0ba5c30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f50d415dd10 thread_stack 0x46000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x562b5cdf47d1]
/usr/sbin/mysqld(handle_fatal_signal+0x393) [0x562b5bdbc093]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f51290973c0]
/usr/sbin/mysqld(my_rnd(rand_struct*)+0x18) [0x562b5ce03d78]
/usr/sbin/mysqld(Item_func_mul::real_op()+0x27) [0x562b5b9d3007]
/usr/sbin/mysqld(Item_func_numhybrid::val_real()+0x46) [0x562b5b9d3696]
/usr/sbin/mysqld(Item_func_round::real_op()+0x23) [0x562b5b9e6173]
/usr/sbin/mysqld(Item_func_numhybrid::val_str(String*)+0x4e) [0x562b5b9d34ae]
/usr/sbin/mysqld(Item_func_concat::val_str(String*)+0xb9) [0x562b5ba2b979]
/usr/sbin/mysqld(Item_func_sha::val_str_ascii(String*)+0x39) [0x562b5ba1cd59]
/usr/sbin/mysqld(Item_str_func::val_str_from_val_str_ascii(String*, String*)+0x8a) [0x562b5ba2b25a]
/usr/sbin/mysqld(Item::save_in_field_inner(Field*, bool)+0x91) [0x562b5b951361]
/usr/sbin/mysqld(Item::save_in_field(Field*, bool)+0x57) [0x562b5b96a737]
/usr/sbin/mysqld(sp_eval_expr(THD*, Field*, Item**)+0xbe) [0x562b5bb9b9ce]
/usr/sbin/mysqld(Query_dumpvar::send_data(THD*, mem_root_deque<Item*> const&)+0x138) [0x562b5bb7fc38]
/usr/sbin/mysqld(SELECT_LEX_UNIT::ExecuteIteratorQuery(THD*)+0x2ed) [0x562b5bd3623d]
/usr/sbin/mysqld(SELECT_LEX_UNIT::execute(THD*)+0x33) [0x562b5bd36583]
/usr/sbin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0x32b) [0x562b5bcb725b]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x535) [0x562b5bcc0fc5]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x55bd) [0x562b5bc5ea9d]
/usr/sbin/mysqld(sp_instr_stmt::exec_core(THD*, unsigned int*)+0x53) [0x562b5bbad7e3]
/usr/sbin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x19b) [0x562b5bbaf77b]
/usr/sbin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0x9e) [0x562b5bbb06fe]
/usr/sbin/mysqld(sp_instr_stmt::execute(THD*, unsigned int*)+0xf2) [0x562b5bbb2062]
/usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x61d) [0x562b5bba532d]
/usr/sbin/mysqld(sp_head::execute_trigger(THD*, MYSQL_LEX_CSTRING const&, MYSQL_LEX_CSTRING const&, GRANT_INFO*)+0x2a5) [0x562b5bba64d5]
/usr/sbin/mysqld(Trigger::execute(THD*)+0x10c) [0x562b5bd816bc]
/usr/sbin/mysqld(Trigger_chain::execute_triggers(THD*)+0x20) [0x562b5bd82ac0]
/usr/sbin/mysqld(Table_trigger_dispatcher::process_triggers(THD*, enum_trigger_event_type, enum_trigger_action_time_type, bool)+0x4a) [0x562b5bd7b94a]
/usr/sbin/mysqld(Sql_cmd_update::update_single_table(THD*)+0x23a5) [0x562b5bd41995]
/usr/sbin/mysqld(Sql_cmd_update::execute_inner(THD*)+0xd5) [0x562b5bd41c55]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x535) [0x562b5bcc0fc5]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x14a9) [0x562b5bc5a989]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x52b) [0x562b5bc60dbb]
/usr/sbin/mysqld(+0x128e2f7) [0x562b5bc612f7]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x38c7) [0x562b5bc65b17]
/usr/sbin/mysqld(do_command(THD*)+0x261) [0x562b5bc66231]
/usr/sbin/mysqld(+0x13d8da0) [0x562b5bdabda0]
/usr/sbin/mysqld(+0x29a9972) [0x562b5d37c972]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f512908b609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f512884c293]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f456475d658): SELECT SHA1(CONCAT(ROUND(RAND()*1000000, 0), NOW())) INTO uid
Connection ID (thread ID): 1085
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.

We then removed PXC and installed MySQL 8.0.25 on one of the nodes and the crash didnt happen again. It seems there is combination of a bug in the 8.0.23 version and our application behaviour which causes the crashs.

Did someone else encounter this issue too?

In the meantime I would suggest to skip this version completly.

BR,

Johannes

1 Like

Hello @Johannes,
Bug reports should be submitted to https://jira.percona.com/ and not our forums. Our devs don’t monitor the forums.

1 Like

Hi, I have the same issue on Percona XC-80 (8.0.23-14-1.buster) of 3 nodes master-mater-master replication. The nodes are crashing randomly when trigger on update is firing. The replication is row-based.

mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Build ID: 133c325a07d46cd4d1c37defc984c08b84cb058b
Server Version: 8.0.23-14.1 Percona XtraDB Cluster (GPL), Release rel14, Revision d3b9a1d, WSREP version 26.4.3, wsrep_26.4.3

Thread pointer: 0x7f8b4858f840
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7f8b69340d20 thread_stack 0x46000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x55fff038863e]
/usr/sbin/mysqld(handle_fatal_signal+0x3b3) [0x55ffef5bc303]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f8bc8706730]
/usr/sbin/mysqld(handler::mark_trx_read_write()+0x28) [0x55ffef186048]
/usr/sbin/mysqld(handler::ha_write_row(unsigned char*)+0x1a) [0x55ffef18951a]
/usr/sbin/mysqld(Query_result_union::send_data(THD*, mem_root_deque<Item*> const&)+0x84) [0x55ffef53b734]
/usr/sbin/mysqld(SELECT_LEX_UNIT::ExecuteIteratorQuery(THD*)+0x3a2) [0x55ffef53c662]
/usr/sbin/mysqld(SELECT_LEX_UNIT::execute(THD*)+0x2c) [0x55ffef53c86c]
/usr/sbin/mysqld(TABLE_LIST::materialize_derived(THD*)+0xb9) [0x55ffef416d69]
/usr/sbin/mysqld(JOIN::optimize()+0x69c) [0x55ffef45e2fc]
/usr/sbin/mysqld(SELECT_LEX::optimize(THD*)+0xa7) [0x55ffef4c2f67]
/usr/sbin/mysqld(SELECT_LEX_UNIT::optimize(THD*, TABLE*, bool)+0x6b) [0x55ffef53d78b]
/usr/sbin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0x2e) [0x55ffef4c19be]
/usr/sbin/mysqld(Sql_cmd_update::execute_inner(THD*)+0x9b) [0x55ffef547e2b]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x6b8) [0x55ffef4cb888]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xfc6) [0x55ffef467c66]
/usr/sbin/mysqld(sp_instr_stmt::exec_core(THD*, unsigned int*)+0x51) [0x55ffef3c5511]
/usr/sbin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x180) [0x55ffef3c73f0]
/usr/sbin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0x92) [0x55ffef3c82f2]
/usr/sbin/mysqld(sp_instr_stmt::execute(THD*, unsigned int*)+0xdf) [0x55ffef3c9b8f]
/usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x5c5) [0x55ffef3bd865]
/usr/sbin/mysqld(sp_head::execute_trigger(THD*, MYSQL_LEX_CSTRING const&, MYSQL_LEX_CSTRING const&, GRANT_INFO*)+0x27a) [0x55ffef3be9aa]
/usr/sbin/mysqld(Trigger::execute(THD*)+0xdc) [0x55ffef58440c]
/usr/sbin/mysqld(Trigger_chain::execute_triggers(THD*)+0x18) [0x55ffef5857a8]
/usr/sbin/mysqld(Table_trigger_dispatcher::process_triggers(THD*, enum_trigger_event_type, enum_trigger_action_time_type, bool)+0x46) [0x55ffef57ed86]
/usr/sbin/mysqld(Sql_cmd_update::update_single_table(THD*)+0x2141) [0x55ffef5479d1]
/usr/sbin/mysqld(Sql_cmd_update::execute_inner(THD*)+0xb5) [0x55ffef547e45]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x6b8) [0x55ffef4cb888]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xfc6) [0x55ffef467c66]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x4b8) [0x55ffef46e4b8]
/usr/sbin/mysqld(+0x127b9e5) [0x55ffef46e9e5]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x3ad3) [0x55ffef473333]
/usr/sbin/mysqld(do_command(THD*)+0x249) [0x55ffef473979]
/usr/sbin/mysqld(+0x13b9e08) [0x55ffef5ace08]
/usr/sbin/mysqld(+0x2691bd4) [0x55fff0884bd4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f8bc86fbfa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f8bc7ea54cf]

Has someone reported/fixed this bug?

1 Like