# MySQL 8.0.24 / 8.0.25 connection problems with \`FEDERATED\` storage engine (Communication link failure) **URL:** \ **Category:** Percona Server for MySQL\ **Created:** [July 23, 2021, 11:46am UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487 "2021-07-23T11:46:34Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![onlime](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/onlime/32/229_2.png) [@onlime](https://forums.percona.com/u/onlime)\ **Post date:** [July 23, 2021, 11:46am UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/1 "2021-07-23T11:46:34Z")
Since latest **Percona Server for MySQL 8.0.25** upgrade, I am struggling with severe connection issues between two servers. I could track this down to [`FEDERATED` storage engine](https://dev.mysql.com/doc/refman/8.0/en/federated-storage-engine.html). On application/client level, the sporadic errors appear like this: ```auto Communication link failure: 1160 Got an error writing communication packets Communication link failure: 1156 Got packets out of order ``` It’s hard to debug as those errors don’t appear in `mysql.err`, not even with `log_error_verbosity = 3`. Looks like internal connections from `FEDERATED` tables to server are not getting logged. Those errors only pop up after a couple of hours. Initially the workaround was to restart MySQL on the host with `FEDERATED` tables (see below, host `backend`) every 2-3 hours, which made the connection dropping disappear. Here’s my setup (all on Percona Server for MySQL 8.0.25): - **Host `backend`** : PHP 8.0 application (using PDO/mysqlnd) with MySQL triggers that copy data from main application db to second `mailsync` database. `mailsync` database only contains tables of storage engine `FEDERATED` that write data to remote host `mail` - **Host `mail`** : REPLICATION MASTER for database `mailsync` - **Host `mx1`** : 1st REPLICATION SLAVE / REPLICA for database `mailsync` - **Host `mx2`** : 2nd REPLICATION SLAVE / REPLICA for database `mailsync` I have that rather complex setup running in production for 7 months now on Percona Server for MySQL 8.0, without any issues until and including MySQL 8.0.23. Data in `FEDERATED` tables is accessed more or less frequently (5-10 times/hour). Only the latest upgrade to 8.0.25 broke it. The connection issues appear immediately without running into any timeout, and also affect simple `SELECT` queries accessing very little data. I suspect this has something to do with the newly introduced connection management in MySQL 8.0.24: [https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-24.html#mysqld-8-0-24-connection-management](https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-24.html#mysqld-8-0-24-connection-management) > **Connection Management Notes** > > Previously, if a client did not use the connection to the server within the period specified by the [`wait_timeout`](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout) system variable and the server closed the connection, the client received no notification of the reason. Typically, the client would see Lost connection to MySQL server during query ([`CR_SERVER_LOST`](https://dev.mysql.com/doc/mysql-errors/8.0/en/client-error-reference.html#error_cr_server_lost)) or MySQL server has gone away ([`CR_SERVER_GONE_ERROR`](https://dev.mysql.com/doc/mysql-errors/8.0/en/client-error-reference.html#error_cr_server_gone_error)). > > In such cases, the server now writes the reason to the connection before closing it, and client receives a more informative error message, The client was disconnected by the server because of inactivity. See wait\_timeout and interactive\_timeout for configuring this behavior. ([`ER_CLIENT_INTERACTION_TIMEOUT`](https://dev.mysql.com/doc/mysql-errors/8.0/en/server-error-reference.html#error_er_client_interaction_timeout)). > > The previous behavior still applies for client connections to older servers and connections to the server by older clients. As I have that same setup of 4 related MySQL servers running in production in two different companies (my own webhosting company and a similar scaled hosting company I manage the infrastructure for), I now have a great way I can prove it is really MySQL \> 8.0.23 and `FEDERATED` storage engine related. I have downgraded the 4 servers (by full data dump and fresh MySQL reinstall / reloading all data) of one company to Percona Server for MySQL 8.0.23 - the problem no longer pops up! On MySQL 8.0.25 I have also found a workaround to make the problem (nearly… 2 days of testing is not enough yet) disappear: ```ini [mysqld] interactive_timeout = 86400 wait_timeout = 86400 ``` Raising the `wait_timeout` on all 3 hosts (mainly `mail` as `FEDERATED` table “server”, but I also raised those values on MySQL replication slaves, which was probably not needed) from default 8h to 24h is currently my best workaround. Before I tried to lower `wait_timeout` to 1h, and indeed the connection issues got much more frequent, 1+ hrs after restarting MySQL. So it looks like MySQL internally keeps track of its `FEDERATED` connections to the remote server and once the server with drops inactive connections after `wait_timeout`, `FEDERATED` still tries to re-use them and struggles with the new [MySQL 8.0.24+ Connection Management](https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-24.html#mysqld-8-0-24-connection-management). In my eyes, `FEDERATED` should silently try to reconnect in case a previous server connection got dropped due to a timeout, as it was before. Can you tell if this is a known bug that was possibly already fixed in latest [MySQL 8.0.26](https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-26.html) or if I am the first one reporting those issues? Thanks, Philip ---
**Author:** ![bitone](https://avatars.discourse-cdn.com/v4/letter/b/ea666f/32.png) [@bitone](https://forums.percona.com/u/bitone)\ **Post date:** [November 10, 2021, 1:16pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/2 "2021-11-10T13:16:16Z")
Hi, same issue here. Are you sure that this is connected to the MySLQ update to \> 8.0.24? On Debian there were very long timeout settings by default. After changing them we also started to get these error messages on federated tables. ---
**Author:** ![Arnaud](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@Arnaud](https://forums.percona.com/u/Arnaud)\ **Post date:** [October 25, 2022, 4:17pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/3 "2022-10-25T16:17:54Z")
this bug was identified and fixe in 8.0.28 [[PS-7999] FEDERATED engine not reconnecting on wait\_timeout exceeded - Percona JIRA](https://jira.percona.com/browse/PS-7999) ---
**Author:** ![onlime](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/onlime/32/229_2.png) [@onlime](https://forums.percona.com/u/onlime)\ **Post date:** [October 26, 2022, 3:30pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/4 "2022-10-26T15:30:12Z")
very nice, thanks for the link! Didn’t occur for quite some time, so I can confirm it was fixed in MySQL 8.0.28. I have now reverted `connect_timeout` and `interactive_timeout` back from 24h to its defaults (8h). That was my workaround which probably never had a real effect. ---
**Author:** ![onlime](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/onlime/32/229_2.png) [@onlime](https://forums.percona.com/u/onlime)\ **Post date:** [November 24, 2022, 11:09am UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/5 "2022-11-24T11:09:15Z")
While this bug never popped up in MySQL 8.0.28 and 8.0.29, it now starts again bugging us since having upgraded to Percona Server for MySQL 8.0.30 ( `8.0.30-22`, tested on Debian Bullseye) so, it looks like there is a regression for [[PS-7999] FEDERATED engine not reconnecting on wait\_timeout exceeded](https://jira.percona.com/browse/PS-7999) in MySQL 8.0.30. @Kamil_Holubicki Can you please reopen that issue? ---
**Author:** ![Kamil\_Holubicki](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kamil_holubicki/32/5693_2.png) [@Kamil\_Holubicki](https://forums.percona.com/u/Kamil_Holubicki)\ **Post date:** [November 24, 2022, 2:22pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/6 "2022-11-24T14:22:56Z")
Hello @onlime Thank you for the info. Could you please create a new Jira ticket [https://jira.percona.com/projects/PXC](https://jira.percona.com/projects/PXC)? Please provide detailed steps to allow us to reproduce the problem. If possible, could you please try MySQL 8.0.31 (Oracle) and confirm you see the problem as well? ---
**Author:** ![bitone](https://avatars.discourse-cdn.com/v4/letter/b/ea666f/32.png) [@bitone](https://forums.percona.com/u/bitone)\ **Post date:** [April 25, 2023, 5:35pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/7 "2023-04-25T17:35:34Z")
Hi @Kamil_Holubicki @onlime is there any update on this issue? We’re using the latest version 8.0.32-2 on Debian 11 and the error still occurs. ---
**Author:** ![onlime](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/onlime/32/229_2.png) [@onlime](https://forums.percona.com/u/onlime)\ **Post date:** [April 25, 2023, 7:47pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/8 "2023-04-25T19:47:56Z")
Hey @bitone – Last time I tested this in production on Percona for MySQL 8.0.31 and the problem still occurred. Again, I had to revert back to my workaround which does the trick (potentially the problem could still occur, but didn’t during the last 2 months with that configuration), raising those timeouts from default 8h to 24h: ```sql interactive_timeout: 86400 wait_timeout: 86400 ``` so far, this also works on 8.0.32 for more than a month. But yes, this issue should be reopened and addressed. @Arnaud ? ---
**Author:** ![bitone](https://avatars.discourse-cdn.com/v4/letter/b/ea666f/32.png) [@bitone](https://forums.percona.com/u/bitone)\ **Post date:** [April 25, 2023, 10:05pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/9 "2023-04-25T22:05:18Z")
@onlime Thank you for your quick update! We tried this solution in the past. But with quite a lot of users we had 10+ connections more each day. So I wrote a script which reads a table of all sleeping threads and kills the ones who were waiting for more than a few hours… Not nice but at least it reduces the problem to an acceptable level. ---
**Author:** ![Kamil\_Holubicki](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kamil_holubicki/32/5693_2.png) [@Kamil\_Holubicki](https://forums.percona.com/u/Kamil_Holubicki)\ **Post date:** [April 26, 2023, 8:24am UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/10 "2023-04-26T08:24:29Z")
I did the following test: 2 Percona Servers, both 8.0.32 Server 2: config: ```auto port=5000 wait_timeout=4 interactive_timeout=4 ``` ```auto create database test; create table test.f (id int primary key); insert into test.f values(1); ``` Server 1: config: ```auto federated=1 ``` ```auto create database test; create server test foreign data wrapper test1 options(user 'root', password '', host '127.0.0.1', port 5000, database 'test'); create table test.f (id int primary key) engine=federated connection='test'; mysql> select * from test.f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0,02 sec) ``` //wait more than 4 seconds ```auto mysql> select * from test.f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0,01 sec) ``` I debugged it a little and reconnection is done in client.cc::cli\_advanced\_command() Looks like everything works as expected. Could you please try the above and post the result? ---
**Author:** ![bitone](https://avatars.discourse-cdn.com/v4/letter/b/ea666f/32.png) [@bitone](https://forums.percona.com/u/bitone)\ **Post date:** [April 27, 2023, 3:24pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/11 "2023-04-27T15:24:18Z")
Sorry I didn’t have time yet to test this. This is how it looks on our server right now: ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/9/97e890c153027de6bcb98c3bb7622f6cd5473549.png) And that’s exactly how it looked before the fix from version 8.0.28… But maybe it’s another issue this time? I wonder why the Fed. engine keeps the connections open but doesn’t reuse them… ---
**Author:** ![bitone](https://avatars.discourse-cdn.com/v4/letter/b/ea666f/32.png) [@bitone](https://forums.percona.com/u/bitone)\ **Post date:** [May 1, 2023, 11:36pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/12 "2023-05-01T23:36:09Z")
> [@Kamil\_Holubicki](#): > > ` select * from test.f;` @Kamil_Holubicki Finally I did as you said. And this is the result: ```auto ERROR 1156 (08S01): Got packets out of order mysql> select * from f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0.02 sec) mysql> select * from f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0.01 sec) mysql> select * from f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0.01 sec) mysql> select * from f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0.01 sec) mysql> select * from f; ERROR 1156 (08S01): Got packets out of order ``` So when I wait for \> 4 seconds, it seems fine. But when I do it in random intervals, I get the “Got packets out of order” error quite frequently. Something must have changed again to the bad in the past few releases… It’s quite obvious in our server log from a CMS that uses some synchronized data of a remote federated DB. After the patch in 8.0.28 these errors did totally disappear. And now they came back and we didn’t change anything except doing the regular apt update + upgrade. ---
**Author:** ![bitone](https://avatars.discourse-cdn.com/v4/letter/b/ea666f/32.png) [@bitone](https://forums.percona.com/u/bitone)\ **Post date:** [May 1, 2023, 11:44pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/13 "2023-05-01T23:44:41Z")
It seams that I did not do the test well. I probably waited to short. I repeated the test and now the error appears each time after waiting 5 or 6 seconds: ```auto mysql> select * from f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0.01 sec) mysql> select * from f; ERROR 1156 (08S01): Got packets out of order mysql> select * from f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0.02 sec) mysql> select * from f; ERROR 1156 (08S01): Got packets out of order mysql> select * from f; +----+ | id | +----+ | 1 | +----+ 1 row in set (0.02 sec) mysql> select * from f; ERROR 1156 (08S01): Got packets out of order ``` It seems to me that the federated engine expects that the connection is still there but it isn’t. Well, great 🙄 I did some digging in the Git history. Here the error was fixed: [PS-7999: FEDERATED engine not reconnecting on wait\_timeout exceeded · percona/percona-server@9b9cfc9 · GitHub](https://github.com/percona/percona-server/commit/9b9cfc9f4ab617f23149556c6663907e157b0a74) And here the fix was reverted (on Nov 24, 2022): [Revert "PS-7999: FEDERATED engine not reconnecting on wait\_timeout ex… · percona/percona-server@8a44ea6 · GitHub](https://github.com/percona/percona-server/commit/8a44ea641c5410452e199a0aba46a805c05f60e3) Why? After doing some more reading: It say in Percona Git: > This reverts commit [9b9cfc9](https://github.com/percona/percona-server/commit/9b9cfc9f4ab617f23149556c6663907e157b0a74) in favour of upstream [mysql/mysql-server@70ef8034a](https://github.com/mysql/mysql-server/commit/70ef8034a) But the this upstream fix was never applied to Percona ha\_federated.cc Mai 2nd: Opened bug report [[PS-8747] FEDERATED engine not handling wait\_timeout MySQL Bug #33500956 - Percona JIRA](https://jira.percona.com/browse/PS-8747) ---
**Author:** ![Kamil\_Holubicki](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kamil_holubicki/32/5693_2.png) [@Kamil\_Holubicki](https://forums.percona.com/u/Kamil_Holubicki)\ **Post date:** [May 4, 2023, 9:49am UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/14 "2023-05-04T09:49:20Z")
Thank you for opening the ticket in Jira. Let’s continue the investigation there. ---
**Author:** ![bitone](https://avatars.discourse-cdn.com/v4/letter/b/ea666f/32.png) [@bitone](https://forums.percona.com/u/bitone)\ **Post date:** [May 24, 2023, 4:17pm UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/15 "2023-05-24T16:17:36Z")
Good news: This will be fixed in the pending release 8.0.33-25! Thank you @Kamil_Holubicki ---
**Author:** ![yuyuks](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/yuyuks/32/19978_2.png) [@yuyuks](https://forums.percona.com/u/yuyuks)\ **Post date:** [January 31, 2025, 7:53am UTC](https://forums.percona.com/t/mysql-8-0-24-8-0-25-connection-problems-with-federated-storage-engine-communication-link-failure/11487/16 "2025-01-31T07:53:58Z")
I’m using 8.0.40 and unfortunately the problem still exists. I searched all over the internet for a solution and finally found this. I will encounter this problem several times at intervals of 8 hours, and then it will be stable for 8 hours. So I guess the problem is when reconnecting. My only solution now is to write a cron job to keep trying to connect to keep it online.