Xtrabackup with xbstream S3 endlessly prints “log scanned up to”

I’m running this command:

xtrabackup --backup --compress --stream=xbstream --datadir=/var/lib/mysql/ --target-dir=/percona/backups \
--user=root --password=password | \
xbcloud put --storage=s3 \
--s3-endpoint='s3.amazonaws.com' \
--s3-bucket='percona-backups' \
$(date -I)-full_backup

If I run just xtrabackup without xbcloud it works fine, and the backup is saved locally. I have access to IAM Profile for AWS S3 with correct role, I tested it.

if I try to do xbcloud get s3://cashli-dev-percona-backups/test.txt it just never returns the response on stdout. I have internet blocked on the instance but the s3 endpoints can be access by services, because it’s enabled on endpoints and also aws s3 ls s3://your-bucket-name works fine

Hi @Ben_Chiciudean , seems like you are having a firewall issue. xbcloud will have to make redirects if aws gives it a different endpoint. Can you please run xbcloud get command passing --verbose flag and share the results here.

You might want to obfuscate sensitive information such as your access and secret keys

Thanks

this is with --verbose flag:

*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
Accept: */*
Accept-Encoding: gzip
X-aws-ec2-metadata-token-ttl-seconds: 21600
Content-Length: 0

I run it with xbcloud get --verbose s3://my-bucket/test.txt it gets stuck at the Content-Legth

If I do:

curl -X GET http://169.254.169.254/latest/meta-data/iam/info

It retrieves the instance profile arn correctly but the /latest/api/token no response

Hi Ben,

This is how tokens via IAM works, We generate a temporary token, this token is used to fetch instance profile and retrieve the token used to authenticate into S3. Can you please those commands manually:

#Generate a token
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

#Check if we have any instance profile (You should see your IAM role here)
curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/


#Retirve the instance profile data ( adjust YOUR_INSTANCE_PROFILE to the output from previous command)
curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/YOUR_INSTANCE_PROFILE

Please send me the output of those

it won’t generate the token:

bash-4.4$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:01:29 --:--:--     0^C

how does it work for aws s3 cli?

I must be using IMDSv1 for the EC2 Instance

I got it working by increasing the hop limit for metadata service:

aws ec2 modify-instance-metadata-options \
    --instance-id <instance_id> \
    --http-put-response-hop-limit 2 \
    --http-endpoint enabled

this made sure that I was able to access the IMDSv2 from the docker container