Frequent MySQL Crash on 5.6.43-84.3-log

MySQL is crashing very frequently with Assertion failure error on file row0sel.cc line 2887

Crash Frequency

2022-09-22 18:33:07 7f15c252f700  InnoDB: Assertion failure in thread 139731431257856 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-22 20:45:10 7fe1bd972700  InnoDB: Assertion failure in thread 140607525168896 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-22 21:57:35 7fe60007e700  InnoDB: Assertion failure in thread 140625819723520 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-22 23:10:14 7f16bd9f6700  InnoDB: Assertion failure in thread 139735647348480 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 01:47:11 7fb83f5ac700  InnoDB: Assertion failure in thread 140429313623808 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 02:46:05 7f4b582fe700  InnoDB: Assertion failure in thread 139961578809088 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 03:02:51 7fc775d60700  InnoDB: Assertion failure in thread 140494652180224 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 03:45:57 7f5a7949a700  InnoDB: Assertion failure in thread 140026558654208 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 04:32:34 7f2abb3e1700  InnoDB: Assertion failure in thread 139821506762496 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 08:42:31 7f5c8c08d700  InnoDB: Assertion failure in thread 140035463108352 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 09:33:10 7f708cf32700  InnoDB: Assertion failure in thread 140121377810176 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 11:56:40 7ef1380a7700  InnoDB: Assertion failure in thread 139574492428032 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 12:49:30 7fba213fc700  InnoDB: Assertion failure in thread 140437398472448 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 15:35:57 7f86e019c700  InnoDB: Assertion failure in thread 140217262130944 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 20:12:30 7faa42f37700  InnoDB: Assertion failure in thread 140369244419840 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 20:14:07 7f173c121700  InnoDB: Assertion failure in thread 139737768793856 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 20:40:10 7f6dd12d5700  InnoDB: Assertion failure in thread 140109637572352 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
--
2022-09-23 22:49:34 7f38fc9a9700  InnoDB: Assertion failure in thread 139882732885760 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED

Full Error

2022-09-23 22:49:34 7f38fc9a9700  InnoDB: Assertion failure in thread 139882732885760 in file row0sel.cc line 2887
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:19:34 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=284
max_threads=5032
thread_count=246
connection_count=243
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2010560 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x559a96bf4300
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f38fc9a8eb8 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x559a9535307c]
/usr/sbin/mysqld(handle_fatal_signal+0x481)[0x559a950dbb31]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7f45d4dc60e0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f45d2c2efff]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f45d2c3042a]
/usr/sbin/mysqld(+0x355a65)[0x559a94ffca65]
/usr/sbin/mysqld(+0x35610f)[0x559a94ffd10f]
/usr/sbin/mysqld(+0x7c58df)[0x559a9546c8df]
/usr/sbin/mysqld(+0x71a9e5)[0x559a953c19e5]
/usr/sbin/mysqld(_ZN7handler13ha_index_nextEPh+0xbc)[0x559a9502948c]
/usr/sbin/mysqld(_ZN7handler15read_range_nextEv+0x20)[0x559a9502d610]
/usr/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0xb2)[0x559a95025112]
/usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x4b)[0x559a952671ab]
/usr/sbin/mysqld(+0x5e0702)[0x559a95287702]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x14e)[0x559a95137c6e]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x450)[0x559a95137040]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x29d)[0x559a95181e6d]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x13c)[0x559a9518261c]
/usr/sbin/mysqld(+0x348cea)[0x559a94fefcea]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x24d9)[0x559a9515c0f9]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_stateb+0x448)[0x559a95160ef8]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1843)[0x559a95162f13]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x192)[0x559a9512d732]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x559a9512d7d0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f45d4dbc4a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f45d2ce4d0f]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f3890004e00): is an invalid pointer
Connection ID (thread ID): 8
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

Is there any way how to mitigate this?

Hello @Sukan_M,
Your only option is to upgrade to MySQL 5.7, and then upgrade to MySQL 8.0. MySQL 5.6 has been completely dead for almost 2 years. No bug fixes, no security patches. Please upgrade to 5.7 and see if your issue persists.

You might also try using a different isolation level. The errors look related to READ_UNCOMMITTED so I suggest you stop using that level and instead use the default of ‘REPEATABLE READ’ or ‘READ COMMITTED’.

2 Likes