Not the answer you need?
Register and ask your own question!

xbcloud: Probe failed. Please check your credentials and endpoint settings.

nvusernvuser EntrantCurrent User Role Novice
Greetings!

I have a MySQL backup schedule that runs full backups each Monday, incremental backups Tue-Fri, encrypts them and uploads them to AWS S3 using xbcloud.

Everything worked fine until Tuesday when xbcloud started to fail with the following error:
191212 11:01:18 xbcloud: Probe failed. Please check your credentials and endpoint settings.

Monday evening the full backup completed successfully and there were no changes that I know of on the MySQL server between Mon / Tue.

Running in verbose, I can see it tries to connect to a probe-bucket, which does not exist:
xbcloud --verbose put mysql-apt-config_0.8.9-1_all.deb                                                               
*   Trying 52.219.74.104...
* TCP_NODELAY set
* Connected to probe-bucket.s3.eu-central-1.amazonaws.com (52.219.74.104) port 443 (#0)                                                       
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH                                                                
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs                                                                                                                      
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.s3.eu-central-1.amazonaws.com                                            
*  start date: Nov  9 00:00:00 2019 GMT
*  expire date: Dec 10 12:00:00 2020 GMT
*  subjectAltName: host "probe-bucket.s3.eu-central-1.amazonaws.com" matched cert's "*.s3.eu-central-1.amazonaws.com"                         
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2                                                           
*  SSL certificate verify ok.
> HEAD / HTTP/1.1
Host: probe-bucket.s3.eu-central-1.amazonaws.com

My configuration file looks like:
[xbcloud]
storage=s3
s3-access-key=something
s3-secret-key=else
s3-bucket=my-backup-bucket
s3-region=eu-central-1

What I tried so far:
  • specify S3 options via command line instead of conf file, same error
  • append all the other options like S3 api version, lookup and so on, no luck
  • test the credentials outside of the server, ensure the user can upload data to S3, this works fine
  • upgrade xtrabackup from 2.4.16 to 2.4.17, same error
At this point I am really running out of ideas so I am hoping someone can give me another hint as this backup process has been working fine for months now.

Thanks!

Comments

  • cesarecesare Entrant Current User Role Beginner
    Hello,

    we started having the exact same issue at the same time. Posting verbose output here:
    xbcloud --verbose put test4.txt
    * About to connect() to s3.amazonaws.com port 443 (#0)
    *   Trying 54.231.98.83...
    * Connected to s3.amazonaws.com (54.231.98.83) port 443 (#0)
    * Initializing NSS with certpath: sql:/etc/pki/nssdb
    *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
      CApath: none
    * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    * Server certificate:
    *       subject: CN=s3.amazonaws.com,O="Amazon.com, Inc.",L=Seattle,ST=Washington,C=US
    *       start date: Nov 09 00:00:00 2019 GMT
    *       expire date: Dec 02 12:00:00 2020 GMT
    *       common name: s3.amazonaws.com
    *       issuer: CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US
    > HEAD /probe-bucket/ HTTP/1.1
    Accept: */*
    Accept-Encoding: gzip
    Authorization: AWS4-HMAC-SHA256 Credential=AKIATMZ5RRM4FXZTZKVU/20191212/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=924c46642ae4f35ab6de1a1140aaa817f01ddacaa199e3620f914d0763b4a144
    Host: s3.amazonaws.com
    X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    X-Amz-Date: 20191212T171513Z
    
    < HTTP/1.1 301 Moved Permanently
    < x-amz-bucket-region: ap-northeast-1
    < x-amz-request-id: 593F2CAFF2110C4C
    < x-amz-id-2: b7i0XgPeusIW7GN+6WpYrD2MDYfVXrIj7MYHG8ITGpTPjC5qJ/nG/ADWX1BhrPAVOw6aOj11h5k=
    < Content-Type: application/xml
    < Transfer-Encoding: chunked
    < Date: Thu, 12 Dec 2019 17:15:12 GMT
    < Server: AmazonS3
    <
    * Connection #0 to host s3.amazonaws.com left intact
    191212 12:15:13 xbcloud: Probe failed. Please check your credentials and endpoint settings.
    


    Configuration:
    [xbcloud]
    storage=s3
    s3-endpoint=https://s3.amazonaws.com/
    s3-region=us-east-1
    s3-access-key=secret
    s3-secret-key=secret
    s3-bucket=mybucket
    


    I can upload files with the aws-cli.
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Patron
    First of all, thank you for reporting these issues.

    Our engineering team are currently working to deliver a patch. Something has been changed in the AWS configuration that has caused an issue for PXB. probe-bucket no longer exists on S3.

    If uploading files via aws-cli is an option temporarily then this may be the best route until we can confirm this is fixed.

    I am sorry, right this minute I don't have the deep technical details for this as I am letting the people who understand get on with the fix right now.
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Patron
    Percona XtraBackup to AWS S3: Issue Alert

    We want to make you aware that, based on information shared with Percona via forum posts and a bug report, a recent change by Amazon to the configuration of AWS S3 is causing the upload of automated backups from Percona XtraBackup via xbcloud to AWS S3 to fail.

    This is not a security incident, and no data has been compromised, however, stable production uses of Percona XtraBackup may experience issues.

    Our engineers are working urgently on a fix, but in the interim please take the necessary steps to secure backup files in a way that does not depend on xbcloud uploads to AWS S3. While xbcloud uploads are impacted, you can upload files to AWS S3 manually using AWS CLI.

    We will release further information as it becomes available.


    Thanks again for your reports
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Patron
    UPDATE: we have released fixes for this issue, if you use Percona XtraBackup in this scenario where AWS S3 is used as storage for backup files, please upgrade to the latest version. Here are the release notes:

    https://www.percona.com/doc/percona-....4/2.4.18.html
    https://www.percona.com/doc/percona-...8.0/8.0.9.html
  • nvusernvuser Entrant Current User Role Novice
    I confirm the issue has been fixed with 2.4.18, many thanks for the quick fix!
  • cesarecesare Entrant Current User Role Beginner
    I confirm too for 2.4.18. Thanks guys!
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.