Hi,
this morning one of 3 proxysql processes on a server has problems that led to a crash and
The logs before the crash show a long series of unhealthy messages like this:
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52808
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52842
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52858
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52864
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52892
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52878
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52918
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52944
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52976
2024-07-16 04:03:25 MySQL_Thread.cpp:3895:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.24.104.106:52990
subsequent restart of the process.
followed by:
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
Error: signal 6:
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
Error: signal 6:
Error: signal 6:
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
/usr/bin/proxysql(_Z13crash_handleri+0x40)[0x55e7d41d3ae0]
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
Error: signal 6:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fbff7e42520]
Error: signal 6:
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x40)[0x55e7d41d3ae0]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fbff7e42520]
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fbff7e96a7c]
terminating with:
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
Error: signal 6:
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
---- /usr/bin/proxysql(_Z13crash_handleri+0x40) [0x55e7d41d3ae0] : crash_handler(int)
---- /usr/bin/proxysql(_ZN12MySQL_Thread41create_new_session_and_client_data_streamEi+0x459) [0x55e7d422f349] : MySQL_Thread::create_new_session_and_client_data_stream(int)
---- /usr/bin/proxysql(_Z11child_mysqlPv+0x136) [0x55e7d427e656] : child_mysql(void*)
To report a crashing bug visit: https://github.com/sysown/proxysql/issues
For support visit: https://proxysql.com/services/support/
2024-07-16 09:17:28 MySQL_Thread.cpp:2854:create_new_session_and_client_data_stream(): [ERROR] For FD -1 fcntl() returned -1 errno 9
proxysql: MySQL_Thread.cpp:2856: MySQL_Session* MySQL_Thread::create_new_session_and_client_data_stream(int): Assertion `prevflags != -1' failed.
(I have not yer opened an issue, just becouse it warns it’s for reproducible bugs…, but I guess I should)
I realized that we had 4 of these crashes in a month.
What can be the cause, how can I further debug?
I’m using proxysql 2.5.1-1