Xtrabackup fails intermittently for large database

MySQL & MariaDB Percona XtraBackup
1

Hi,

We have large database of around 8TBs and the backup fails intermittently with below error. Full database backups runs every day and the it would fail with below error once or twice a week. Backups are getting streamed to S3 bucket using xbstream and it is a on-prem system and not EC2 or GCP.

~~handler failed with backup failed with errors: 2024-08-19T00:25:01.046033-00:00 6 [ERROR] [MY-011825] [Xtrabackup] failed to copy datafile ./dbname/dbf.ibd

This happens intermittently, we take backups from replicas, so tried taking backups from multiple replicas but same issue on all.

Server version: 8.0.35-27 Percona Server (GPL)
xtrabackup version 8.0.35-30 based on MySQL server 8.0.35 Linux (x86_64)

Please let me if there is any solution to this.

2

If you are streaming to S3, you should be using XBCloud to handle that aspect. PXB and XBCloud both have improved parallel channels/FIFO support to maximize network throughput. Check out the docs and blog for how to use this.