After patching Percona XtraDB Cluster from 5.7.33 → 5.7.35 we started to receive messages:
[Note] Got packets out of order
It looks like client - server communication problem.
How can we troubleshoot this issue? Get more info about the client?
I would check PMM Networking graphs and see if you have retransmission or dropped packet issues.
There are no drop packets or transmission errors. But occasionally we have few segments retransmitted.
I have tried to play with max_allowed_packet parameter, but even though I set very high value (512MB) we were still receiving info messages.
I would check which exact statements fail with the error. Which client library do you use? Do you call stored procedures? You may also enable general query log file and then check in which order MySQL server received failed queries.
Well, retransmitted might be an issue.
Verify that your application code is using the same mysql client library version as your PXC.
Are there specific queries that are causing this message to show?
Sorry for the late answer. For now I don’t know how to identify which statements are failing.
In the database level I don’t have much info. Probably some application is logging errors but we need to find which one. We have few stored procedures not using them very often.
I have enabled general query log but there aren’t failing queries logged when we experience “Got packets out of order” messages.
There are messages:
Change user firstname.lastname@example.org on information_schema using TCP/IP
Query KILL CONNECTION 3594153
But still don’t know how to resolve this issue.
Hello @matthewb ,
We are using this libs at application side:
- mysql:mysql-connector-java:8.0.25 (connector)
- org.sql2o:sql2o:1.6.0 (model mapper)
- com.zaxxer:HikariCP:4.0.3 (connection pool)
There are probably specific queries which cause this messages to appear, but I (still) don’t know how to find this queries. On 5.7.33 version we weren’t receiving “packets out of order” messages.
“Got packets out of order” error is expected when the application tries to run a new query without processing results from the previous one. For example, if your Python application calls a procedure that returns a result set, ignores it, then runs another query. Therefore it is necessary to find out which queries and applications receive these errors. And yes, your application developers need to log error messages when they receive them.
Check also if you were not affected by this bug: MySQL Bugs: #102266: mysql_change_user fails due to COM_CHANGE_USER incorrectly parsed While the bug is about C API it practically affects all connectors. COM_CHANGE_USER command is issued each time client connects or reconnects to the server. Since you are using a connection pool this could be the case.
In my case, on MySQL 5.7.40, it was simply invalid request.
POST /index.php, accidentally to MySQL server.
MySQL responded by [Note] Got packets out of order
Therefore even completely nonsense request to MySQL / MariaDB / Percona may cause such notice.