Hi,
yesterday we had a severe incident on one of our productive clusters running the recent percona-xtradb-cluster-server version 8.0.43-34-1.
The cluster consists of 3 nodes, of which 2 shutdown with the same error message. The remaining node than went into maintenance mode, as it was alone.
The first Node node quit at 11:19:35 UTC with this error:
2025-11-06T12:19:34.834518Z 75195 [ERROR] [MY-013183] [InnoDB] Assertion failure: row0ins.cc:279:!cursor->index->is_committed() thread 139784774006336
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
"""""""""""""""""""""""2025-11-06T12:19:34Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=4df0e2ec4f229e4a2ada9d37d5f46180ee315df2
Server Version: 8.0.43-34.1 Percona XtraDB Cluster (GPL), Release rel34, Revision 0682ba7, WSREP version 26.1.4.3, wsrep_26.1.4.3
Thread pointer: 0x7f225c2defc0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f222dccabd0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55a9be8b7be1]
/usr/sbin/mysqld(print_fatal_signal(int)+0x3fc) [0x55a9be02a38c]
/usr/sbin/mysqld(my_server_abort()+0x7e) [0x55a9be02a41e]
/usr/sbin/mysqld(my_abort()+0xe) [0x55a9be8ac24e]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x182) [0x55a9bea98322]
/usr/sbin/mysqld(row_ins_sec_index_entry_low(unsigned int, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*, bool)+0x101d) [0x55a9bea1395d]
/usr/sbin/mysqld(row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool)+0x15a) [0x55a9bea13bfa]
/usr/sbin/mysqld(+0x19c9852) [0x55a9bea42852]
/usr/sbin/mysqld(row_upd_step(que_thr_t*)+0x541) [0x55a9bea44441]
/usr/sbin/mysqld(+0x20b5249) [0x55a9bf12e249]
/usr/sbin/mysqld(ha_innobase::update_row(unsigned char const*, unsigned char*)+0x1f3) [0x55a9be912ba3]
/usr/sbin/mysqld(handler::ha_update_row(unsigned char const*, unsigned char*)+0x1a6) [0x55a9bdbe19e6]
/usr/sbin/mysqld(write_record(THD*, TABLE*, COPY_INFO*, COPY_INFO*)+0x117a) [0x55a9bde8a5ea]
/usr/sbin/mysqld(Query_result_insert::send_data(THD*, mem_root_deque<Item*> const&)+0xfa) [0x55a9bde8acba]
/usr/sbin/mysqld(Query_expression::ExecuteIteratorQuery(THD*)+0x2d5) [0x55a9bdf9bbe5]
/usr/sbin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0xc0) [0x55a9bdf18b90]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x670) [0x55a9bdf18290]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xf0f) [0x55a9bdeedc5f]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x69d) [0x55a9bdef305d]
/usr/sbin/mysqld(+0x226f076) [0x55a9bf2e8076]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x255a) [0x55a9bdeca8aa]
/usr/sbin/mysqld(do_command(THD*)+0x32e) [0x55a9bdecb15e]
/usr/sbin/mysqld(+0xfb2660) [0x55a9be02b660]
/usr/sbin/mysqld(+0x1bd2bc6) [0x55a9bec4bbc6]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f27b5e80ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x7f27b5f128c0]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f2793c4b760): INSERT LOW_PRIORITY INTO bla.products_offers (shop_id, offer_hash, product_number, product_name, manufacturer, manufacturer_md5, manufacturer_id, manufacturer_artno, manufacturer_artno_plain, manufacturer_artno_md5, color, color_id, size, FROM bla.offers_1036 WHERE sync IN (1,3) AND no_import = 0 ON DUPLICATE KEY UPDATE product_number = VALUES(product_number), product_name = VALUES(product_name), uiflag = VALUES(uiflag), modified=CURRENT_TIMESTAMP
Connection ID (thread ID): 75195
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
2025-11-06T12:19:34.893247Z 75195 [Note] [MY-000000] [WSREP] Initiating SST cancellation
6 Minutes later, at 11:25:14 UTC the second node died:
2025-11-06T12:25:14.930455Z 76254 [ERROR] [MY-013183] [InnoDB] Assertion failure: row0ins.cc:279:!cursor->index->is_committed() thread 140218839426624
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2025-11-06T12:25:14Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=4df0e2ec4f229e4a2ada9d37d5f46180ee315df2
Server Version: 8.0.43-34.1 Percona XtraDB Cluster (GPL), Release rel34, Revision 0682ba7, WSREP version 26.1.4.3, wsrep_26.1.4.3
Thread pointer: 0x7f8c2fa274d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f873e1d5bd0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x564f5895fbe1]
/usr/sbin/mysqld(print_fatal_signal(int)+0x3fc) [0x564f580d238c]
/usr/sbin/mysqld(my_server_abort()+0x7e) [0x564f580d241e]
/usr/sbin/mysqld(my_abort()+0xe) [0x564f5895424e]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x182) [0x564f58b40322]
/usr/sbin/mysqld(row_ins_sec_index_entry_low(unsigned int, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*, bool)+0x101d) [0x564f58abb95d]
/usr/sbin/mysqld(row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool)+0x15a) [0x564f58abbbfa]
/usr/sbin/mysqld(+0x19c9852) [0x564f58aea852]
/usr/sbin/mysqld(row_upd_step(que_thr_t*)+0x541) [0x564f58aec441]
/usr/sbin/mysqld(+0x20b5249) [0x564f591d6249]
/usr/sbin/mysqld(ha_innobase::update_row(unsigned char const*, unsigned char*)+0x1f3) [0x564f589baba3]
/usr/sbin/mysqld(handler::ha_update_row(unsigned char const*, unsigned char*)+0x1a6) [0x564f57c899e6]
/usr/sbin/mysqld(write_record(THD*, TABLE*, COPY_INFO*, COPY_INFO*)+0x117a) [0x564f57f325ea]
/usr/sbin/mysqld(Query_result_insert::send_data(THD*, mem_root_deque<Item*> const&)+0xfa) [0x564f57f32cba]
/usr/sbin/mysqld(Query_expression::ExecuteIteratorQuery(THD*)+0x2d5) [0x564f58043be5]
/usr/sbin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0xc0) [0x564f57fc0b90]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x670) [0x564f57fc0290]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xf0f) [0x564f57f95c5f]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x69d) [0x564f57f9b05d]
/usr/sbin/mysqld(+0x226f076) [0x564f59390076]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x255a) [0x564f57f728aa]
/usr/sbin/mysqld(do_command(THD*)+0x32e) [0x564f57f7315e]
/usr/sbin/mysqld(+0xfb2660) [0x564f580d3660]
/usr/sbin/mysqld(+0x1bd2bc6) [0x564f58cf3bc6]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8c60b5fac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x7f8c60bf18c0]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f8c2fc1bc00): INSERT LOW_PRIORITY INTO bla.products_offers (shop_id, offer_hash, product_number, product_name, manufacturer, manufacturer_md5, manufacturer_id, manufacturer_artno, manufacturer_artn
o_plain, manufacturer_artno_md5, FROM bla.offers_1036 WHERE sync IN (1,3) AND no_import = 0 ON DUPLICATE KEY UPDATE prod
uct_number = VALUES(product_number), product_name = VALUES(product_name), manufacturer = VALUES(manufacturer), manufacturer_md5 = VALUES(manufacturer_md5), manufacturer_id = VALUES(manufacturer_id),
availability = VALUES(availability), material = VALUES(material), preload_link = VALUES(preload_link), stock_availabl
e = VALUES(stock_available), release_time = VALUES(release_time), is_price_compare = VALUES(is_price_compare), is_checkout = VALUES(is_checkout), guid = VALUES(guid), uiflag = V
ALUES(uiflag), modified=CURRENT_TIMESTAMP
Connection ID (thread ID): 76254
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
2025-11-06T12:25:14.983579Z 76254 [Note] [MY-000000] [WSREP] Initiating SST cancellation
(END)
I removed parts of the query to not expose the customers data structure, can provide the schema and statement in private, if needed. Tried to insert a Bug in XtraDB Jira - can login but not post there.
The Statement is regularly running throughout the day with different where clauses without any issues. The third node did not crash.
The table itself has around 35GB:
±----------------±-----------±-------------±-----------+
| Tabelle | Daten (MB) | Indizes (MB) | Total (MB) |
±----------------±-----------±-------------±-----------+
| products_offers | 25429.00 | 10319.72 | 35748.72 |
±----------------±-----------±-------------±-----------+
1 row in set (0.01 sec)
The underlying hardware is VMs on separate Host machines, dmesg didn’t log any HW related access error and I don’t think that both machines had the same problem.
I’ve seen that there is a maybe related issue over here in the forums - I will try so provide as many information as possible. Just let me know what to provide.
Thanks very much for any hints or ideas on the matter