xbcloud: Probe failed. Please check your credentials and endpoint settings.

MySQL & MariaDB Percona XtraBackup
1

Greetings!

I have a MySQL backup schedule that runs full backups each Monday, incremental backups Tue-Fri, encrypts them and uploads them to AWS S3 using xbcloud.

Everything worked fine until Tuesday when xbcloud started to fail with the following error:


191212 11:01:18 xbcloud: Probe failed. Please check your credentials and endpoint settings.

Monday evening the full backup completed successfully and there were no changes that I know of on the MySQL server between Mon / Tue.

Running in verbose, I can see it tries to connect to a probe-bucket, which does not exist:


xbcloud --verbose put mysql-apt-config_0.8.9-1_all.deb 
* Trying 52.219.74.104...
* TCP_NODELAY set
* Connected to probe-bucket.s3.eu-central-1.amazonaws.com (52.219.74.104) port 443 (#0) 
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH 
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs 
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.s3.eu-central-1.amazonaws.com 
* start date: Nov 9 00:00:00 2019 GMT
* expire date: Dec 10 12:00:00 2020 GMT
* subjectAltName: host "probe-bucket.s3.eu-central-1.amazonaws.com" matched cert's "*.s3.eu-central-1.amazonaws.com" 
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2 
* SSL certificate verify ok.
> HEAD / HTTP/1.1
Host: probe-bucket.s3.eu-central-1.amazonaws.com

My configuration file looks like:


[xbcloud]
storage=s3
s3-access-key=something
s3-secret-key=else
s3-bucket=my-backup-bucket
s3-region=eu-central-1

What I tried so far:
[LIST]
[]specify S3 options via command line instead of conf file, same error
[]append all the other options like S3 api version, lookup and so on, no luck
[]test the credentials outside of the server, ensure the user can upload data to S3, this works fine
[]upgrade xtrabackup from 2.4.16 to 2.4.17, same error
[/LIST] At this point I am really running out of ideas so I am hoping someone can give me another hint as this backup process has been working fine for months now.

Thanks!

2

Hello,

we started having the exact same issue at the same time. Posting verbose output here:

xbcloud --verbose put test4.txt
* About to connect() to s3.amazonaws.com port 443 (#0)
* Trying 54.231.98.83...
* Connected to s3.amazonaws.com (54.231.98.83) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=s3.amazonaws.com,O="Amazon.com, Inc.",L=Seattle,ST=Washington,C=US
* start date: Nov 09 00:00:00 2019 GMT
* expire date: Dec 02 12:00:00 2020 GMT
* common name: s3.amazonaws.com
* issuer: CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US
> HEAD /probe-bucket/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip
Authorization: AWS4-HMAC-SHA256 Credential=AKIATMZ5RRM4FXZTZKVU/20191212/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=924c46642ae4f35ab6de1a1140aaa817f01ddacaa199e3620f914d0763b4a144
Host: s3.amazonaws.com
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20191212T171513Z

< HTTP/1.1 301 Moved Permanently
< x-amz-bucket-region: ap-northeast-1
< x-amz-request-id: 593F2CAFF2110C4C
< x-amz-id-2: b7i0XgPeusIW7GN+6WpYrD2MDYfVXrIj7MYHG8ITGpTPjC5qJ/nG/ADWX1BhrPAVOw6aOj11h5k=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 12 Dec 2019 17:15:12 GMT
< Server: AmazonS3
<
* Connection #0 to host s3.amazonaws.com left intact
191212 12:15:13 xbcloud: Probe failed. Please check your credentials and endpoint settings.

Configuration:

[xbcloud]
storage=s3
s3-endpoint=https://s3.amazonaws.com/
s3-region=us-east-1
s3-access-key=secret
s3-secret-key=secret
s3-bucket=mybucket

I can upload files with the aws-cli.

3

First of all, thank you for reporting these issues.

Our engineering team are currently working to deliver a patch. Something has been changed in the AWS configuration that has caused an issue for PXB. probe-bucket no longer exists on S3.

If uploading files via aws-cli is an option temporarily then this may be the best route until we can confirm this is fixed.

I am sorry, right this minute I don’t have the deep technical details for this as I am letting the people who understand get on with the fix right now.

4

Percona XtraBackup to AWS S3: Issue Alert

We want to make you aware that, based on information shared with Percona via forum posts and a bug report, a recent change by Amazon to the configuration of AWS S3 is causing the upload of automated backups from Percona XtraBackup via xbcloud to AWS S3 to fail.

This is not a security incident, and no data has been compromised, however, stable production uses of Percona XtraBackup may experience issues.

Our engineers are working urgently on a fix, but in the interim please take the necessary steps to secure backup files in a way that does not depend on xbcloud uploads to AWS S3. While xbcloud uploads are impacted, you can upload files to AWS S3 manually using AWS CLI.

We will release further information as it becomes available.

Thanks again for your reports

5

[B]UPDATE: we have released fixes for this issue, if you use Percona XtraBackup in this scenario where AWS S3 is used as storage for backup files, please upgrade to the latest version. Here are the release notes:

[URL=“Percona XtraBackup 2.4.18 - Percona XtraBackup”]https://www.percona.com/doc/percona-....4/2.4.18.html[/URL]
[URL=“Percona XtraBackup 8.0.9 - Percona XtraBackup”]https://www.percona.com/doc/percona-...8.0/8.0.9.html[/URL][/B]

6

I confirm the issue has been fixed with 2.4.18, many thanks for the quick fix!

7

I confirm too for 2.4.18. Thanks guys!