Hey,
I have a ProxySQL-Cluster that seems to crash around once every 1-2 days.
The only information I get in the logs is:
2024-04-29 06:02:31 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:34 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:37 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:40 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:43 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:46 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:49 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:52 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:55 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:58 main.cpp:2435:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2024-04-29 06:02:58 main.cpp:2442:main(): [ERROR] Watchdog: reached 10 missed heartbeats. Aborting!
2024-04-29 06:02:58 main.cpp:2443:main(): [ERROR] Watchdog: see details at https://github.com/sysown/proxysql/wiki/Watchdog
proxysql: main.cpp:2444: int main(int, const char**): Assertion `0' failed.
Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x40)[0x55d80b32ab40]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fb47f7cf520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fb47f8239fc]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fb47f7cf476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fb47f7b57f3]
/lib/x86_64-linux-gnu/libc.so.6(+0x2871b)[0x7fb47f7b571b]
/lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7fb47f7c6e96]
/usr/bin/proxysql(main+0x316d)[0x55d80b307e0b]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fb47f7b6d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fb47f7b6e40]
/usr/bin/proxysql(_start+0x25)[0x55d80b2fd825]
---- /usr/bin/proxysql(_Z13crash_handleri+0x40) [0x55d80b32ab40] : crash_handler(int)
To report a crashing bug visit: https://github.com/sysown/proxysql/issues
For support visit: https://proxysql.com/services/support/
2024-04-29 06:03:16 main.cpp:1455:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2024-04-29 06:03:16 [INFO] ProxySQL version 2.6.2-percona-1.1
2024-04-29 06:03:16 [INFO] ProxySQL SHA1 checksum: 935864c0544b8251e90ea16aaf9fa37baf3951e5
ProxySQL does write a coredump file, but I can not open it with GDB, as it seems to be broken.
I call gdb like this: gdb /usr/bin/proxysql core.3313507
And I get an error that it looks like
BFD: warning: /var/lib/proxysql/core.478690 has a segment extending past end of file
And when trying to run “bt”, I get
(gdb) bt
#0 0x00007f601a2d19fc in ?? ()
Backtrace stopped: Cannot access memory at address 0x7ffd45d43fc0
I googled around and found, that in Ubuntu there seems to be a file size limit for coredumps.
I tried to increase that by writing to /etc/systemd/coredump.conf:
[Coredump]
ProcessSizeMax=8G
ExternalSizeMax=8G
JournalSizeMax=8G
But I also read that this file is used by systemd-coredump, which does not seem to exist on my Ubuntu 22 machine:
#> systemctl status systemd-coredump
Unit systemd-coredump.service could not be found.
And it did not seem to have any effect on newer core dumps. The core dump files are always around 1,7G - 2G in size.
Hi @mmbih5ro,
You can edit /etc/security/limits.conf and add or modify the next line in order to set the maximum size of core dump files to unlimited.
* soft core unlimited
Please notice that the service have to be restarted for applying changes.
Thanks a lot. I will try that and wait a few days for the next crash.
I will keep you posted if this solved my problem 
Hey @Vadim_Yalovets,
I tried setting the config, and then issued “systemctl daemon-reload” and “systemctl restart proxysql”, but no success.
Opening the core dump still shows:
gdb /usr/bin/proxysql /var/lib/proxysql/core.2728281
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/proxysql...
(No debugging symbols found in /usr/bin/proxysql)
BFD: warning: /var/lib/proxysql/core.2728281 has a segment extending past end of file
[New LWP 2728281]
[New LWP 2728284]
[New LWP 2728288]
[New LWP 2728290]
[New LWP 2728294]
[New LWP 2728295]
[New LWP 2728296]
[New LWP 2728283]
[New LWP 2728297]
[New LWP 2728287]
[New LWP 2728301]
[New LWP 2728289]
[New LWP 2728291]
[New LWP 2728302]
[New LWP 2728292]
[New LWP 2728293]
[New LWP 2728298]
[New LWP 2728299]
[New LWP 2728300]
[New LWP 2728304]
[New LWP 2728306]
[New LWP 3500387]
[New LWP 3500389]
[New LWP 3500398]
[New LWP 3500400]
[New LWP 3500459]
[New LWP 3500471]
[New LWP 3500472]
[New LWP 3500478]
[New LWP 3500498]
[New LWP 3500516]
[New LWP 3500519]
[New LWP 3500541]
[New LWP 3500556]
[New LWP 3500571]
[New LWP 3500583]
[New LWP 3500694]
[New LWP 3500710]
[New LWP 3500743]
[New LWP 2728303]
[New LWP 2728305]
[New LWP 2728286]
[New LWP 3500247]
[New LWP 3500359]
[New LWP 3500366]
[New LWP 3500374]
[New LWP 3500458]
[New LWP 3500463]
[New LWP 3500483]
[New LWP 3500485]
[New LWP 3500496]
[New LWP 3500508]
[New LWP 3500530]
[New LWP 3500534]
[New LWP 3500542]
[New LWP 3500552]
[New LWP 3500554]
[New LWP 3500576]
[New LWP 3500632]
[New LWP 3500640]
[New LWP 3500642]
[New LWP 3500699]
[New LWP 3500701]
[New LWP 3500402]
[New LWP 3500723]
[New LWP 3500732]
[New LWP 3500721]
[New LWP 3500136]
[New LWP 2728282]
[New LWP 3500067]
[New LWP 3500191]
[New LWP 3500235]
[New LWP 3500363]
[New LWP 3500352]
[New LWP 2728312]
[New LWP 3500169]
[New LWP 3500171]
[New LWP 2728313]
[New LWP 3500350]
[New LWP 3500179]
[New LWP 3500145]
[New LWP 3500295]
[New LWP 3500181]
[New LWP 3500376]
[New LWP 3500146]
[New LWP 3500351]
[New LWP 3500167]
[New LWP 3500153]
[New LWP 2728285]
[New LWP 3500244]
[New LWP 3500298]
warning: Error reading shared library list entry at 0x29ba00746473
Failed to read a valid object file image from memory.
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `/usr/bin/proxysql --idle-threads -c /etc/proxysql.cnf'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f9e9101c9fc in ?? ()
[Current thread is 1 (LWP 2728281)]
(gdb) bt
#0 0x00007f9e9101c9fc in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fff841eb9b0
My /etc/security/limits.conf:
root hard nofile 65536
root soft nofile 65536
* hard nofile 65536
* soft nofile 65536
* soft core unlimited
Could you try to change value of parameter LimitCORE to infinity in proxysql service file?
