# Database crash in regular intervals - mysqld got signal 11 **URL:** \ **Category:** Percona Server for MySQL\ **Created:** [August 29, 2023, 2:59pm UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793 "2023-08-29T14:59:07Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![lebeda.jaromir](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/lebeda.jaromir/32/12507_2.png) [@lebeda.jaromir](https://forums.percona.com/u/lebeda.jaromir)\ **Post date:** [August 29, 2023, 2:59pm UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/1 "2023-08-29T14:59:07Z")
Hello, we are testing Percona Server for Mysql 8.0.33-25 on Debian 11.7 (installed from repository). We are getting error `mysqld got signal 11` followed by database crash in regular intervals. We have tested Percona Server and Percona Xtradb cluster and error is present on both in versions 8.0.22 and later. Version 8.0.21 runs without an error. We have also tested our code with no problem on Percona Server and XtraDB Cluster 5.7.X The crash is caused by our JAVA applications that send the data into the database with simple insert/update/delete operations – but these operations trigger database triggers. The database crash repeats regularly – for example every 4 minutes. Java parameter [maxLifetime](https://github.com/brettwooldridge/HikariCP#gear-configuration-knobs-baby) in our client applications seems to be somehow related to these database crashes - in case we decrease Java parameter maxLifetime also period of database crashes decrease. Can someone help us debug backtrace of this error? Is it a database bug? Can we somehow optimize database configuration on our side to prevent database crashes? PS: I have found this thread [https://bugs.mysql.com/bug.php?id=102036](https://bugs.mysql.com/bug.php?id=102036) and this bug seems to me that is of similar type (they also got signal 11 on executing a trigger) Thank you in advance ```auto 2023-08-25T10:51:54Z UTC - mysqld got signal 11 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. BuildID[sha1]=7712c421dbb5f98622a8ada6b288d73ab447e90a Server Version: 8.0.33-25 Percona Server (GPL), Release '25', Revision '60c9e2c5' Thread pointer: 0x7f82c9dee050 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f8b9c110c20 thread_stack 0x100000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x558d672b8b9e] /usr/sbin/mysqld(print_fatal_signal(int)+0x38b) [0x558d663d403b] /usr/sbin/mysqld(handle_fatal_signal+0xc5) [0x558d663d4105] /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f8f866e3140] /usr/sbin/mysqld(find_type2(TYPELIB const*, char const*, unsigned long, CHARSET_INFO const*)+0x25) [0x558d66357275] /usr/sbin/mysqld(Field_enum::store(char const*, unsigned long, CHARSET_INFO const*)+0xaf) [0x558d664c1d0f] /usr/sbin/mysqld(Item::save_in_field(Field*, bool)+0x53) [0x558d65f9add3] /usr/sbin/mysqld(sp_eval_expr(THD*, Field*, Item**)+0xab) [0x558d6619485b] /usr/sbin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x5fa) [0x558d661a8a0a] /usr/sbin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0xa5) [0x558d661a92a5] /usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x5c1) [0x558d6619faf1] /usr/sbin/mysqld(sp_head::execute_function(THD*, Item**, unsigned int, Field*)+0x5d5) [0x558d661a1fe5] /usr/sbin/mysqld(Item_func_sp::execute_impl(THD*)+0x117) [0x558d6600fac7] /usr/sbin/mysqld(Item_func_sp::execute()+0x7c) [0x558d6600fbfc] /usr/sbin/mysqld(Item_func_sp::val_str(String*)+0x41) [0x558d6600ff51] /usr/sbin/mysqld(eval_string_arg(CHARSET_INFO const*, Item*, String*)+0x7d) [0x558d66002f8d] /usr/sbin/mysqld(Arg_comparator::compare_string()+0x26) [0x558d65f9f126] /usr/sbin/mysqld(Item_func_eq::val_int()+0x28) [0x558d65fa4ba8] /usr/sbin/mysqld(Item::val_bool()+0xad) [0x558d65f7d4ad] /usr/sbin/mysqld(sp_instr_jump_if_not::exec_core(THD*, unsigned int*)+0x2d) [0x558d661a62ad] /usr/sbin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x5fa) [0x558d661a8a0a] /usr/sbin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0xa5) [0x558d661a92a5] /usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x5c1) [0x558d6619faf1] /usr/sbin/mysqld(sp_head::execute_trigger(THD*, MYSQL_LEX_CSTRING const&, MYSQL_LEX_CSTRING const&, GRANT_INFO*)+0x26d) [0x558d661a04ed] /usr/sbin/mysqld(Trigger::execute(THD*)+0xec) [0x558d6638685c] /usr/sbin/mysqld(Trigger_chain::execute_triggers(THD*)+0x18) [0x558d66387c48] /usr/sbin/mysqld(Table_trigger_dispatcher::process_triggers(THD*, enum_trigger_event_type, enum_trigger_action_time_type, bool)+0x4d) [0x558d6637fc6d] /usr/sbin/mysqld(Sql_cmd_update::update_single_table(THD*)+0x2041) [0x558d663456c1] /usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x16e) [0x558d662b791e] /usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xaef) [0x558d6625d57f] /usr/sbin/mysqld(sp_instr_stmt::exec_core(THD*, unsigned int*)+0x4f) [0x558d661a61bf] /usr/sbin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x169) [0x558d661a8579] /usr/sbin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0xa5) [0x558d661a92a5] /usr/sbin/mysqld(sp_instr_stmt::execute(THD*, unsigned int*)+0xea) [0x558d661aabea] /usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x5c1) [0x558d6619faf1] /usr/sbin/mysqld(sp_head::execute_procedure(THD*, mem_root_deque*)+0x858) [0x558d661a2c68] /usr/sbin/mysqld(Sql_cmd_call::execute_inner(THD*)+0x154) [0x558d661d5a14] /usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x16e) [0x558d662b791e] /usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xaef) [0x558d6625d57f] /usr/sbin/mysqld(sp_instr_stmt::exec_core(THD*, unsigned int*)+0x4f) [0x558d661a61bf] /usr/sbin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x169) [0x558d661a8579] /usr/sbin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0xa5) [0x558d661a92a5] /usr/sbin/mysqld(sp_instr_stmt::execute(THD*, unsigned int*)+0xea) [0x558d661aabea] /usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x5c1) [0x558d6619faf1] /usr/sbin/mysqld(sp_head::execute_procedure(THD*, mem_root_deque*)+0x858) [0x558d661a2c68] /usr/sbin/mysqld(Sql_cmd_call::execute_inner(THD*)+0x154) [0x558d661d5a14] /usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x16e) [0x558d662b791e] /usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xaef) [0x558d6625d57f] /usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x5d4) [0x558d66261384] /usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0xf71) [0x558d662628c1] /usr/sbin/mysqld(do_command(THD*)+0x252) [0x558d66264be2] /usr/sbin/mysqld(+0x1353280) [0x558d663c5280] /usr/sbin/mysqld(+0x26b0185) [0x558d67722185] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f8f866d7ea7] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f8f85e84a2f] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f82fdc22e40): UPDATE Object SET Pushed = 1, PushedTime = aTime, PushedConfigurationId = lConfigurationId WHERE ObjectId = lObjectId Connection ID (thread ID): 3640 Status: NOT_KILLED Please help us make Percona Server better by reporting any bugs at https://bugs.percona.com/ You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash ``` ---
**Author:** ![Agustin\_G](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/agustin_g/32/14111_2.png) [@Agustin\_G](https://forums.percona.com/u/Agustin_G)\ **Post date:** [August 29, 2023, 9:12pm UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/2 "2023-08-29T21:12:15Z")
Hi Jaromir, Can you repeat this behavior with upstream (community) MySQL packages, or is this only happening when you use Percona Server? Also, can you reproduce this at will? This will help with investigations and with testing and reporting a bug (if it’s still happening in latest release). ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [August 30, 2023, 10:44am UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/3 "2023-08-30T10:44:58Z")
Hi @lebeda.jaromir and welcome to Percona community, In addition to previous question, can you also provide two more inputs: - Table definition `SHOW CREATE TABLE table_that_is_causing_crash;` - Check error log for anything specific about that table or if that table is corrupt itself? `CHECK TABLE table_that_is_causing_crash;` Thanks, K ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [August 30, 2023, 4:05pm UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/4 "2023-08-30T16:05:50Z")
If repeatable, can you turn on core dumps, cause a crash, then share the core with us? This would help immensely. ---
**Author:** ![lebeda.jaromir](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/lebeda.jaromir/32/12507_2.png) [@lebeda.jaromir](https://forums.percona.com/u/lebeda.jaromir)\ **Post date:** [August 31, 2023, 6:29am UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/5 "2023-08-31T06:29:28Z")
Hello, thank you for your reaction. I have performed tests also with community Mysql (on same server with same data dir): 1. mysql-8.0.21-linux-glibc2.12-x86\_64 - works fine 2. mysql-8.0.34-linux-glibc2.28-x86\_64 - fails with signal 11 after cca 30 minutes 3. mysql-8.1.0-linux-glibc2.28-x86\_64 - fails with signal 11 after cca 30 minutes I can reproduce this behavior any time - I just turn on our JAVA applications that sends data to DB. ---
**Author:** ![lebeda.jaromir](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/lebeda.jaromir/32/12507_2.png) [@lebeda.jaromir](https://forums.percona.com/u/lebeda.jaromir)\ **Post date:** [August 31, 2023, 9:58am UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/6 "2023-08-31T09:58:14Z")
Hello, I have caused the crash again (on Percona Server for Mysql 8.0.33-25 ). In error\_log I see that failing query is UPDATE. I attach error\_log [error.log](https://forums.percona.com/uploads/short-url/qhVikKtD5Gf5jNTchuhcRMl4hX7.log) (10.1 KB) also I attach output of CREATE TABLE and CHECK TABLE of table that caused the crash (because of our policy I had to anonymize attribute/table names).[create\_check\_table.txt](https://forums.percona.com/uploads/short-url/jV3AUTXyUMlcXRm82gXfLvSM7VO.txt) (2.3 KB) I do not see suspicious records in error\_log other than this crash. Unfortunately we have quite complicated logic - table that caused crash has ON UPDATE trigger - this trigger invokes stored procedures that can INSERT/UPDATE/DELETE in other tables - I tried CHECK TABLE for them too and it seems OK. Personally I do not think the problem is that tables are corrupted. One information to be more specific - our clients (JAVA apps) are reading data from Kafka and send hundreds of these queries per second into the Database, after some specific amount of time the Database crashes with signal 11. ---
**Author:** ![lebeda.jaromir](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/lebeda.jaromir/32/12507_2.png) [@lebeda.jaromir](https://forums.percona.com/u/lebeda.jaromir)\ **Post date:** [August 31, 2023, 1:10pm UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/7 "2023-08-31T13:10:20Z")
Hello, I have turned on core dumps, caused a crash and the core dump file size is 9GB. But I am not allowed to share the core dump file because management is afraid it could contain sensitive data ☹ I am thinking if there is any other solution how to give you information you need to investigate? Maybe - is it possible to give us instructions on what to look for in core dump file? So that it is not necessary to send you all the content of 9GB? ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [August 31, 2023, 3:37pm UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/8 "2023-08-31T15:37:42Z")
Sending you what to look for is variable since there could be any number of reasons for the crash. Having the core dump allows our engineers to view the entire stack and all threads that could be contributing to the problem. We can provide a secure means of uploading the core dump. If your company engages in a Support contract, then the various NDAs and “legal stuff” would obviate any sensitive data issues that your management may be concerned with. I would first remove the trigger and run the application. Verify that all works good like this. Then add the trigger calling the proc, but simplify the proc to just 1 statement. Again, verify this works under load. Then add a bit more complexity to the proc. Repeat this process until the crash happens. Then remove what you added and confirm that adding X to the proc produces the crash. Check to make sure there’s no string-int comparisons. Check that all collations and charsets are the same. ---
**Author:** ![lebeda.jaromir](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/lebeda.jaromir/32/12507_2.png) [@lebeda.jaromir](https://forums.percona.com/u/lebeda.jaromir)\ **Post date:** [September 7, 2023, 8:27am UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/9 "2023-09-07T08:27:45Z")
Hello, thank you for reaction. For now we are trying to limit triggers and so find exactly which code causes crash. I will inform you when we have some output. Regarding core dump and possibility of NDAs, I will discuss again with management and I will let you know next week. Thank you ---
**Author:** ![Red\_Howlett](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/red_howlett/32/12437_2.png) [@Red\_Howlett](https://forums.percona.com/u/Red_Howlett)\ **Post date:** [September 26, 2023, 12:54pm UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/10 "2023-09-26T12:54:24Z")
Do you have any update? I have the exact same problem. ---
**Author:** ![lebeda.jaromir](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/lebeda.jaromir/32/12507_2.png) [@lebeda.jaromir](https://forums.percona.com/u/lebeda.jaromir)\ **Post date:** [December 14, 2023, 3:17pm UTC](https://forums.percona.com/t/database-crash-in-regular-intervals-mysqld-got-signal-11/24793/11 "2023-12-14T15:17:05Z")
Hello, sorry for delay. I think we have finally found solution of our problem with ‘signal 11’. Our databases crash during database triggers execution. These triggers are called often (many times per second). In triggers we have condition to limit on which database instance should run part of our code: > IF getInstance() = ‘INS1’ THEN > – execute some code > END IF; Triggers fail on comparison getInstance() = ‘INS1’ The function getInstance looks like this: > CREATE > DEFINER=`db_definer`@`localhost` > FUNCTION `getInstance`() > RETURNS enum(‘INS1’,‘INS2’,‘INS3’) CHARSET utf8 > > BEGIN > RETURN ‘INS2’; > END If we change function getInstance (for example) so that value ‘INS2’ comes from select to database table it seems database runs stable (we have tested with PXC cluster 8.0.34). > CREATE > DEFINER=`db_definer`@`localhost` > FUNCTION `getInstance`() > RETURNS varchar(4) > > BEGIN > DECLARE lIns varchar(4); > SELECT > `Instance` from `DbInstance` > INTO lIns; > RETURN lIns; > END Maybe is problem with the ENUM in the original function? Could someone explain us this behaviour please? Thank you 🙂