Xbcloud get from azure starts timing out after some data transferred

MySQL & MariaDB Percona XtraBackup
1

I have a scheduled backup of a Percona MySQL 8.0 database for my UAT environment on Azure Blob Storage. I am attempting to restore one of these backups to a VM at GCP (Google Cloud Platform). All of the prerequisites to do this are complete.

I am running the following command on the VM command line to restore the backup:

xbcloud get azure-uat-20230830-b --azure-storage-account=backupuat --azure-access-key=Bej…vQ== --azure-container-name=backupsuat --max-backoff=300000 --max-retries=30 --storage=azure 2>download.log | xbstream -x -C /var/lib/mysql/

The first time I did this it completed but I forgot to pipe to xbstream so I tried again. The second time I ran the command it got 92GB (out of about 120) into the restore and then threw a timeout error. The third time it got 80GB into the restore before erroring. The last attempt was 16GB before the same error.

My first thought was it is something at Azure throttling how much data I can download in a time period. I’ve searched for documentation and cannot find anything that suggests this is happening.

Here is a tail of download.log that shows some files downloaded successfully and the timeout:

root@db-sandbox:/var/lib/mysql# tail -f download.log
230831 20:23:14 xbcloud: [0] Download successfull azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000523, size 10485821
230831 20:23:14 xbcloud: [0] Downloading azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000524.
230831 20:23:14 xbcloud: [0] Download successfull azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000524, size 10485821
230831 20:23:14 xbcloud: [0] Downloading azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000525.
230831 20:23:15 xbcloud: [0] Download successfull azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000525, size 10485821
230831 20:23:15 xbcloud: [0] Downloading azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000526.
230831 20:23:15 xbcloud: [0] Download successfull azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000526, size 10485821
230831 20:23:15 xbcloud: [0] Downloading azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000527.
230831 20:23:16 xbcloud: [0] Download successfull azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000527, size 10485821
230831 20:23:16 xbcloud: [0] Downloading azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000528.
230831 20:25:16 xbcloud: Operation failed. Error: Timeout was reached
230831 20:25:16 xbcloud: Sleeping for 2384 ms before retrying azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000528 [1]
230831 20:25:19 xbcloud: [0] Download failed. Cannot download azure-uat-20230830-b/listings/media_20230104.ibd.00000000000000000528.
230831 20:25:19 xbcloud: Download failed.```
2

Hi Richard,
I have a few questions

  1. Can you please get the xbcloud version ?
  2. Are you able to download successfully with older versions?
  3. What is the size of media_2023104.ibd file? (If you have access to the database)
3

xbcloud Ver 8.0.34-29 for Linux (x86_64) (revision id: 5ba706ee)

I have not tried with any other version of xbcloud

It fails on different files each time when trying to restore.

In the Azure storage account, media_20230104.ibd.00000000000000000528 shows that it is 10MiB

media_20230104.ibd in the database it was backed up from is 16294871040 bytes on disk, roughly 16GB

4

We have introduced timeout in 8.0.34 PXB-3073: An unresponsive host may cause xbcloud to become stuck. Added a --timeout option to xbcloud.

can you please try with xbcloud 8.0.33?

Right, each file is divided into chunks of 10MB. And 16G is not too big.

5

Default timeout for reponse is 120 seconds, may be retry with larger timeout.

Please test and share your feedback.

6

The --timeout option resolved the issue. I set it to --timeout=300 and the restore worked perfectly.
Thank you for your help with this issue.

7

@richardmarlow Glad to help. Thanks for verifying. We will check if we can increase the default timeout in the next release.