I would check which exact statements fail with the error. Which client library do you use? Do you call stored procedures? You may also enable general query log file and then check in which order MySQL server received failed queries.
Sorry for the late answer. For now I don’t know how to identify which statements are failing.
In the database level I don’t have much info. Probably some application is logging errors but we need to find which one. We have few stored procedures not using them very often.
I have enabled general query log but there aren’t failing queries logged when we experience “Got packets out of order” messages.
There are messages:
Change user xxxxx@100.142.10.13 on information_schema using TCP/IP
Query KILL CONNECTION 3594153
Quit
There are probably specific queries which cause this messages to appear, but I (still) don’t know how to find this queries. On 5.7.33 version we weren’t receiving “packets out of order” messages.
“Got packets out of order” error is expected when the application tries to run a new query without processing results from the previous one. For example, if your Python application calls a procedure that returns a result set, ignores it, then runs another query. Therefore it is necessary to find out which queries and applications receive these errors. And yes, your application developers need to log error messages when they receive them.
Check also if you were not affected by this bug: MySQL Bugs: #102266: mysql_change_user fails due to COM_CHANGE_USER incorrectly parsed While the bug is about C API it practically affects all connectors. COM_CHANGE_USER command is issued each time client connects or reconnects to the server. Since you are using a connection pool this could be the case.
In my case, on MySQL 5.7.40, it was simply invalid request.
I have switched host ports of my Docker containers, between web and MySQL. But a web page remains open on previous web port, newly MySQL port. And JavaScript from that page periodically sends request POST /index.php, accidentally to MySQL server.
MySQL responded by [Note] Got packets out of order
Therefore even completely nonsense request to MySQL / MariaDB / Percona may cause such notice.