Newbie ProxySQL user - Aborted connection errors on backend

Deniska80 · April 26, 2022, 6:28am

Hello, testing proxysql on dev stage, there is only 1 backend server, proxysql running on the same machine. All looks great, sysbench test works throught proxysql without errors.
But when i try to switch my web-app db connections from mysql to proxy i immediatly got tons of errors in proxysql.log and in mysql backend logs
proxysql.log:

2022-04-26 09:06:12 MySQL_Session.cpp:3930:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection while running query on (10,12
7.0.0.1,3306,71) , FD (Conn:30 , MyDS:30) , user beauty , last_used 17667444584ms ago : 2027, received malformed packet
2022-04-26 09:06:12 MySQL_Session.cpp:3944:handler_minus1_ClientLibraryError(): [WARNING] Retrying query.
2022-04-26 09:06:12 MySQL_Session.cpp:3930:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection while running query on (10,12
7.0.0.1,3306,72) , FD (Conn:30 , MyDS:30) , user beauty , last_used 17667444584ms ago : 2027, received malformed packet
2022-04-26 09:06:13 MySQL_Session.cpp:3930:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection while running query on (10,12
7.0.0.1,3306,74) , FD (Conn:30 , MyDS:30) , user beauty , last_used 17667445587ms ago : 2027, received malformed packet
2022-04-26 09:06:13 MySQL_Session.cpp:3944:handler_minus1_ClientLibraryError(): [WARNING] Retrying query.

and in mysql.lod

2022-04-26 09:06:12 MySQL_Session.cpp:3930:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection while running query on (10,12
7.0.0.1,3306,71) , FD (Conn:30 , MyDS:30) , user beauty , last_used 17667444584ms ago : 2027, received malformed packet
2022-04-26 09:06:12 MySQL_Session.cpp:3944:handler_minus1_ClientLibraryError(): [WARNING] Retrying query.
2022-04-26 09:06:12 MySQL_Session.cpp:3930:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection while running query on (10,12
7.0.0.1,3306,72) , FD (Conn:30 , MyDS:30) , user beauty , last_used 17667444584ms ago : 2027, received malformed packet
2022-04-26 09:06:13 MySQL_Session.cpp:3930:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection while running query on (10,12
7.0.0.1,3306,74) , FD (Conn:30 , MyDS:30) , user beauty , last_used 17667445587ms ago : 2027, received malformed packet
2022-04-26 09:06:13 MySQL_Session.cpp:3944:handler_minus1_ClientLibraryError(): [WARNING] Retrying query.

The first and last suggestion was about max_allowed_packet size, but it’s seems ok, 32Mb on mysql and 16Mb in runtime_global_variables in proxyslq.
Found some issues with wait_timeout, but it’s 60 seconds on mysql, and errors appears immediatly after i start web applications.
Any ideas? Thanks

matthewb · April 26, 2022, 1:40pm

Can’t tell what version of MySQL you are running. Can’t tell what version of ProxySQL either. You said you tested sysbench from localhost. Did you test sysbench from remote host? Can you create manual connections from remote host through proxysql?

Deniska80 · April 26, 2022, 2:03pm

Oh, my bad
ProxySQL version 2.3.2-percona-1.1, codename Truls
Percona-server-server-5.7 5.7.37-40-1.buster

My web application runs on the same server, so i don’t test remote connections yet. It seems, that part of requests from web applications walks throuth proxy normal, and i can see it in query log. But others not. And seems, that there is no place i can found that problem requests.

matthewb · April 26, 2022, 2:17pm

github.com/sysown/proxysql

[ERROR] Detected a broken connection during query on (10,mysql-pxc-2,3306) , FD (Conn:53 , MyDS:53) : 2013, Lost connection to MySQL server during query

opened 10:29AM - 03 Apr 20 UTC

u2bo

ProxySQL version 2.0.10， MySQL 5.7.26 PXC i used pxc cluster some time i get …this error whey this happend? some log like this 2020-04-03 06:41:52 MySQL_Thread.cpp:4383:process_data_on_data_stream(): [WARNING] Detected broken idle connection on mysql-pxc-2:3306 2020-04-03 06:41:52 MySQL_Thread.cpp:4583:process_all_sessions(): [WARNING] Closing unhealthy client connection 10.20.185.128:36462 2020-04-03 07:47:34 MySQL_Session.cpp:4286:handler(): [ERROR] Detected a broken connection during query on (10,mysql-pxc-2,3306) , FD (Conn:53 , MyDS:53) : 2013, Lost connection to MySQL server during query 2020-04-03 08:30:52 MySQL_Session.cpp:4286:handler(): [ERROR] Detected a broken connection during query on (10,mysql-pxc-2,3306) , FD (Conn:53 , MyDS:53) : 2013, Lost connection to MySQL server during query 2020-04-03 09:52:12 MySQL_Session.cpp:4286:handler(): [ERROR] Detected a broken connection during query on (10,mysql-pxc-2,3306) , FD (Conn:53 , MyDS:53) : 2013, Lost connection to MySQL server during query +-----------+-------------+----------+--------+----------+----------+--------+---------+-------------+---------+-------------------+-----------------+-----------------+------------+ | hostgroup | srv_host | srv_port | status | ConnUsed | ConnFree | ConnOK | ConnERR | MaxConnUsed | Queries | Queries_GTID_sync | Bytes_data_sent | Bytes_data_recv | Latency_us | +-----------+-------------+----------+--------+----------+----------+--------+---------+-------------+---------+-------------------+-----------------+-----------------+------------+ | 10 | mysql-pxc-2 | 3306 | ONLINE | 6 | 4 | 29 | 0 | 9 | 235774 | 0 | 53481047 | 10391592 | 2001 | | 12 | mysql-pxc-0 | 3306 | ONLINE | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 700 | | 12 | mysql-pxc-1 | 3306 | ONLINE | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 733 | | 11 | mysql-pxc-0 | 3306 | ONLINE | 0 | 3 | 3 | 0 | 3 | 47566 | 0 | 6502188 | 7078945 | 700 | | 11 | mysql-pxc-1 | 3306 | ONLINE | 0 | 5 | 5 | 0 | 4 | 46988 | 0 | 6434536 | 7041653 | 733 | +-----------+-------------+----------+--------+----------+----------+--------+---------+-------------+---------+-------------------+-----------------+-----------------+------------+

60s wait_timeout on MySQL is pretty low. Can you try reverting that to the default and see if your issue remains?

Deniska80 · April 27, 2022, 5:38am

Yes, i tired set default 28800 seconds in mysql and in proxy server, but results are the same. And as i understand it’s can’t be timeouts, coz errors appears immediatly i change database connection in application, after 3-5 seconds.
And again - i can see some normal requests in queries.log, so connection, user settings and so on are normal. May be this is some “unnormal” sql-request, but how can i track them…

ethaniel · February 15, 2024, 9:11am

This issue is most likely related to Segfault when upgrading to 2.1.0 · Issue #3251 · sysown/proxysql · GitHub

Topic		Replies	Views
Percona cluster for mysql and proxysql	10	985	August 31, 2023
ProxySQL Backend SSL Connection Error ProxySQL	5	1786	July 12, 2022
Proxysql Error Lost connection to mysql server ProxySQL	4	5284	December 29, 2020
Getting error while trying to execute basic queries like show databases / select version, etc Percona Operator for MySQL mysql	11	3035	November 3, 2020
ProxySQL Crashing - Signal 11 ProxySQL	2	976	August 18, 2023

Newbie ProxySQL user - Aborted connection errors on backend

Related topics