Investigating much slower ET on full xtrabackup

MySQL & MariaDB
1

I’ve been investigating an issue where a backup of a 2.8G mysql database (to a data domain NFS mount) sent from about 6 hours to 22 - 40 hours. The database is mysql-community-common-5.7.19-1. While looking for changes, I found that xtrabackup patched from 2.4.23 to 2.4.24 when this problem started.

Has anyone seen performance issues with 2.4.24?

Other than the time, the other constraint I see is that our outbound net traffic could hit 3Gbps before, but now it only reaches 450Mbps. I’ve been looking for network or appliance constraints, then found this version change.

Thanks

2

Have you 100% verified this by rolling back to xtrabackup 2.4.23 and repeating a backup?

1 Like
3

@LenR
I’ve just verified and xtrabackup 2.4.24 only added exponential backoff algorithm to xbcloud.
The only changes to xtrabackup between .23 and .24 came from merging 5.7.35 into the codebase. Looking the release notes MySQL :: MySQL 5.7 Release Notes :: Changes in MySQL 5.7.35 (2021-07-20, General Availability) there isn’t anything that seems related to NFS.

On top of @matthewb already asked, check if mount options are still the same. Perhaps a reboot and non-persistent mount options could be the culprit.

1 Like
4

Thanks, testing now “during the day”, I’m getting a little less thruput than on the weekend backup, but I’d expect that. The good news is that the xtrabackup version doesn’t matter.

This is exactly what I expected, I was just asking in case it was a known problem, again, I would have been very surprised if it was.

We had chased a previous red herring, Microsoft mdatp was installed at about the same time and has caused performance issues on other servers. However, removing it didn’t help either.

1 Like