Pt-deadlock-logger - "The client was disconnected by the server because of inactivity."

Polyglot Projects Percona Toolkit
,
1

Hello,

I have a long-running pt-deadlock-logger running on docker, that should catch all deadlocks happening, and logs to PERCONA_SCHEMA.

The problem is that it seems it keeps a “forever open” connection to the MySQL server, because it will fail with these errors:

Error saving to --dest: DBD::mysql::st execute failed: The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior. [for Statement "INSERT IGNORE INTO PERCONA_SCHEMA.deadlocks…

I don’t want to increase wait_timeout or interactive_timeout just because of this tool.

Is there a way around this? Like the tool re-opening the connection if this happens, for example.

Thanks!

2

Hi @nunop,
The default --interval for pt-deadlock-logger is 30s, which means every 30s the tool is running ‘SHOW ENGINE INNODB STATUS’ and parsing the result. It is strange that you are getting a lost connection/inactivity between executing that statement, and then inserting a row. What is your wait_timeout and interactive_timeout configured to?

3

Hi @matthewb !

wait_timeout = 300
interactive_timeout = 300

And this is the command:

pt-deadlock-logger --quiet --dest S=/sock/mysql.sock,u=perconatk,p=...,D=PERCONA_SCHEMA,t=deadlocks S=/sock/mysql.sock,u=perconatk,p=...

However… I wonder if this could have happened because I restarted the database, but didn’t restart the logger.

There’s only 1 error in the logs, and the deadlocks haven’t been logged since, so I don’t think the logger is even listening anymore. I’d assume the program would crash, but the container (I’m running on docker) is still running (since February). Just probably not doing anything anymore.

I’ll restart it now, and will make it restart every time the database is also restarted, and hopefully this will resolve the issue.

Thanks!

4

Oh, yes, that would be the case. Especially if you restart MySQL within the 30s while pt-d-l was sleeping, it would not have noticed the loss of connection and would have errored on next loop.

I would expect the tool to be a bit smarter, and auto-reconnect on failure, or exit so the container dies too.

1 Like