We recently upgraded a database server from Percona MySQL version 5.7.39 to version 8.0.29-21. As part of that upgrade process we moved from Percona Xtrabackup 24-2.4.5-1 to 80-8.0.29-22. Immediately after changing the Xtrabackup version our backups began failing with an rsync buffer overflow condition. We’ve been able to replicate this on a separate test system using a restored copy of the database in question, and have also been able to create test scenarios to replicate the behavior. The host server is using rsync version 3.1.2. The database instance on the system in question contains only a few MyISAM customer databases; the remainder are exclusively InnoDB. When the backup reaches the stage in which it is backing up the non-InnoDB tables and files it will teminate with a buffer overflow and the following error:
[ERROR] [MY-011825] [Xtrabackup] rsync failed with error code 1
The issue appears to be related to the number of databases in the instance or the proportion of MyISAM and InnoDB tables. For example, we created a test instance with 100 databases, 20 tables each (10 MyISAM/10 InnoDB) and the backup worked consistently. When we increased the number to 100 databases the test failed. We have yet to find an exact tipping point. We have also been able to recreate this issue using Xtrabackup version 80-8.0.30-23. The local logs on the Linux server don’t provide any more information than the buffer overflow dump, and our efforts with tools such as strace didn’t either.
We have other Percona MySQL version 8.0.29-21/Xtrabackup 80-8.0.29-22 installations that are working as desired; however these systems each contain less than a dozen databases with minimal tables and only use the InnoDB engine. Our installations that use Percona database 5.7.39 and the older versions of Xtrabackup work fine with any combination of large numbers of databases/tables; we’ve created some similar test configurations and experience no backup issues whatsoever. Our current workaround is to not use the “–rsync” option for our Xtrabackup 8.0 routine and to rely on the standard copy function but the locking that occurs is not optimal.
Our searches have found references to very similar issues in the past but provided no concrete solutions. We’ve also opened a bug report via Percona JIRA but thought it might be prudent to bring it up for discussion on this forum as well.