Having difficulties enabling backups with MongoDB Operator 1.10 (header errors)

MongoDB Percona Operator for MongoDB
1

I have the following messages repeating in the logs for my backup-agents

Head "https://s3.amazonaws.com/<bucket-location>/.pbm.init": net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=...

I have tried with both the percona/percona-server-mongodb-operator:1.10.0-backup and perconalab/percona-server-mongodb-operator:main-backup images but get the same result. I have also confirmed that my AWS/S3 keys (provided via credentialsSecret named secret) do not include a newline (mentioned elsewhere), and are properly encoded. Also confirmed that everything is in the same namespace.

Kinda at my wits end at this point. Any help the community can offer would be greatly appreciated.

3 Likes
2

Hey @Mike_Javorski ,

could you please share your cr.yaml? Are you getting this error when you try to take the backup or it happens without taking a backup?

1 Like
3

I have the same error when run backup command
kubectl apply -f backup/backup.yaml

Here is my cr.yaml on storages section

backup:
    enabled: true
    restartOnFailure: true
    image: percona/percona-server-mongodb-operator:1.10.0-backup
    serviceAccountName: percona-server-mongodb-operator
#    resources:
#      limits:
#        cpu: "300m"
#        memory: "0.5G"
#      requests:
#        cpu: "300m"
#        memory: "0.5G"
    storages:
      s3-us-west:
        type: s3
        s3:
          bucket: mongodbs-backup-test
          credentialsSecret: mongodbs-backup-s3
          region: us-west-2
          prefix: ""

and here is secrete config file

apiVersion: v1
kind: Secret
metadata:
  name: mongodbs-backup-s3
type: Opaque
data:
  AWS_ACCESS_KEY_ID: <MY BASE64 ENCODED KEY>
  AWS_SECRET_ACCESS_KEY: <MY BASE64 ENCODED KEY>
1 Like
4

Hi @Sergey_Pronin,

did you solve this problem ?
I have a same problem with Operator 1.11 and minio…
The problem is an extra new line character in header “\n” at the end of AWS_ACCESS_KEY_ID, it is probably false string serialization in pbm-agent. -

EDIT: here is all right, but this error indeed occurs if extra new line character in header provided (wrong base64 encoding)

net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=______ \n/20220130/us-east-1/s3/aws4_request,

For info - this occurs in backup-agent sidecar - [agentCheckup] check storage connection: storage check failed with: get S3 object header

1 Like