Percona Operator MongoDB backup fail on compatible S3

MongoDB Percona Operator for MongoDB
1

Hi, We are using Percona Operator MongoDB version 1.13 to deploy our mongodb’s but we are getting error when configure or manual schedule backup:

E [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed

caused by: Head “https://XXXXXXXX:/bucket/.pbm.init”: net/http: invalid header field value “AWS4-HMAC-SHA256 Credential=OYIKMPD6AZSOLXUXOH99\n/20230220/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a0d4a0c68382dfe2db57930ce581f67f7574e642bf58905ecccf1b25a582d238” for key Authorization.

2 possible causes:

  1. Encriptation protocol issue.
  2. The timezone of the backup agente container does not match with the S3 storage. There are way to change the Pod timezone deployed by operator?

We are not able to check logs of the S3 Storage. Any idea?

Thanks and regards.

2

Hey @rfaraj ,

in the most of the cases this error indicates that S3 storage credentials are wrong. The most common issue is the EOL or \n in the end of the key.

See here for example: Having difficulties enabling backups with MongoDB Operator 1.10 (header errors) - #3 by Duc_Ngo

Can you please double check that you have the correct keys in the Secret?

3

Hello @Sergey_Pronin ,

I had already read this post that you mention and I have performed the validation of the credentials several times.

echo “<AWS_ACCESS_KEY_ID>” | base64 --wrap=0
echo "<AWS_SECRET_ACCESS_KEY> | base64 --wrap=0

It is correct right?

I have tried with the minion that includes the Gitlab deployment and it works without problem.

The issue is only happening when we try to backup against Netapp’s StorageGrid.

I have also tried to define wrong credentials and it keeps returning the same error.

Thanks and regards.

4

Can you try echo -n KEY | base64 --wrap=0 ?

5

Hi @Sergey_Pronin ,

With the -n option in the echo command it works, I don’t normally use this option to encrypt k8s secrets.

Thanks for your support.