Receiving "[Note] Got packets out of order" messages after patching to 5.7.35 version

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Hello,

After patching Percona XtraDB Cluster from 5.7.33 → 5.7.35 we started to receive messages:
[Note] Got packets out of order

It looks like client - server communication problem.
How can we troubleshoot this issue? Get more info about the client?

Thanks

Best regards,
Marko

1 Like
2

I would check PMM Networking graphs and see if you have retransmission or dropped packet issues.

1 Like
3

Hello,

There are no drop packets or transmission errors. But occasionally we have few segments retransmitted.

I have tried to play with max_allowed_packet parameter, but even though I set very high value (512MB) we were still receiving info messages.

Best regards,
Marko

1 Like
4

I would check which exact statements fail with the error. Which client library do you use? Do you call stored procedures? You may also enable general query log file and then check in which order MySQL server received failed queries.

1 Like
5

Well, retransmitted might be an issue.

Verify that your application code is using the same mysql client library version as your PXC.

Are there specific queries that are causing this message to show?

1 Like
6

Hello @svetasmirnova,

Sorry for the late answer. For now I don’t know how to identify which statements are failing.
In the database level I don’t have much info. Probably some application is logging errors but we need to find which one. We have few stored procedures not using them very often.

I have enabled general query log but there aren’t failing queries logged when we experience “Got packets out of order” messages.

There are messages:
Change user xxxxx@100.142.10.13 on information_schema using TCP/IP
Query KILL CONNECTION 3594153
Quit

But still don’t know how to resolve this issue.

Regards,
Marko

1 Like
7

Hello @matthewb ,

We are using this libs at application side:

python:
MySQL-python==1.2.5

java stack:

  • mysql:mysql-connector-java:8.0.25 (connector)
  • org.sql2o:sql2o:1.6.0 (model mapper)
  • com.zaxxer:HikariCP:4.0.3 (connection pool)

go stack:

There are probably specific queries which cause this messages to appear, but I (still) don’t know how to find this queries. On 5.7.33 version we weren’t receiving “packets out of order” messages.

Regards,
Marko

1 Like
8

@msutic

“Got packets out of order” error is expected when the application tries to run a new query without processing results from the previous one. For example, if your Python application calls a procedure that returns a result set, ignores it, then runs another query. Therefore it is necessary to find out which queries and applications receive these errors. And yes, your application developers need to log error messages when they receive them.

1 Like
9

Check also if you were not affected by this bug: MySQL Bugs: #102266: mysql_change_user fails due to COM_CHANGE_USER incorrectly parsed While the bug is about C API it practically affects all connectors. COM_CHANGE_USER command is issued each time client connects or reconnects to the server. Since you are using a connection pool this could be the case.

1 Like
10

In my case, on MySQL 5.7.40, it was simply invalid request.

I have switched host ports of my Docker containers, between web and MySQL. But a web page remains open on previous web port, newly MySQL port. And JavaScript from that page periodically sends request POST /index.php, accidentally to MySQL server.

MySQL responded by [Note] Got packets out of order

Therefore even completely nonsense request to MySQL / MariaDB / Percona may cause such notice.

1 Like