ERROR with storage: check storage connection: storage check failed with: get S3 object header: Forbidden: Forbidden

The backup-agent is continously reporting that they have a forbidden request.

k logs -f pod/mongodb-cluster-rs0-1 -c backup-agent
2024-05-15T11:58:35.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: Forbidden: Forbidden
        status code: 403, request id: 3G4YBFG1QYCMDYJ6, host id: niliFnpNtKmcDQqnLTjWHAUJ2PxjmVDoKTo1+Gw/ikA/GejsglxtI6tFITfPCOYS7+Md1W8Dd9g=

But with Cyberduck im able to download+upload files. s3:* is everything allowed.

cr.yaml

# https://github.com/percona/percona-server-mongodb-operator/blob/v1.15.0/deploy/cr.yaml
---
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: mongodb-cluster
  namespace: percona-mongodb
  finalizers:
    - delete-psmdb-pods-in-order
spec:
  crVersion: 1.15.0
  image: percona/percona-server-mongodb:6.0.9-7
  imagePullPolicy: IfNotPresent
  allowUnsafeConfigurations: false
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: '0 2 * * *'
    setFCV: false
  secrets:
    users: mongodb-cluster
    encryptionKey: mongodb-cluster-encryption-key
  pmm:
    enabled: true
    image: percona/pmm-client:2.39.0
    serverHost: monitoring-service.percona-pmm.svc.cluster.local
  replsets:
    - name: rs0
      size: 3
      affinity:
        antiAffinityTopologyKey: 'kubernetes.io/hostname'
      tolerations:
        - key: 'node-app'
          operator: 'Equal'
          value: 'mongodb'
          effect: 'NoExecute'
      podDisruptionBudget:
        maxUnavailable: 1
      expose:
        enabled: false
        exposeType: ClusterIP
      resources:
        limits:
          cpu: '2'
          memory: '4Gi'
        requests:
          cpu: '0.5'
          memory: '1Gi'
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 100Gi

      nonvoting:
        enabled: false
        size: 3
        affinity:
          antiAffinityTopologyKey: 'kubernetes.io/hostname'
        podDisruptionBudget:
          maxUnavailable: 1
        resources:
          limits:
            cpu: '300m'
            memory: '0.5G'
          requests:
            cpu: '300m'
            memory: '0.5G'
        volumeSpec:
          persistentVolumeClaim:
            resources:
              requests:
                storage: 3Gi
      arbiter:
        enabled: false
        size: 1
        affinity:
          antiAffinityTopologyKey: 'kubernetes.io/hostname'
        resources:
          limits:
            cpu: '300m'
            memory: '0.5G'
          requests:
            cpu: '300m'
            memory: '0.5G'

  sharding:
    enabled: false
    configsvrReplSet:
      size: 3
      affinity:
        antiAffinityTopologyKey: 'kubernetes.io/hostname'
      podDisruptionBudget:
        maxUnavailable: 1
      expose:
        enabled: false
        exposeType: ClusterIP
      resources:
        limits:
          cpu: '300m'
          memory: '0.5G'
        requests:
          cpu: '300m'
          memory: '0.5G'
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi

    mongos:
      size: 3
      affinity:
        antiAffinityTopologyKey: 'kubernetes.io/hostname'
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: '300m'
          memory: '0.5G'
        requests:
          cpu: '300m'
          memory: '0.5G'
      expose:
        exposeType: ClusterIP

  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.3.0
    serviceAccountName: percona-server-mongodb-operator
    resources:
      limits:
        memory: 1Gi
      requests:
        memory: 500Mi
        cpu: 300m
    pitr:
      enabled: false
      compressionType: gzip
      compressionLevel: 6
    storages:
      aws-s3:
        type: s3
        s3:
          bucket: 'change_me'
          credentialsSecret: 'mongodb-backup-s3'
          region: 'eu-north-1'

Bucket S3 user iam permission:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::change_me",
                "arn:aws:s3:::change_me/*"
            ],
            "Sid": "AllowAccess"
        }
    ]
}

Ah not sure what was the solution.

I recreated the credentials. The error still appeared but then i created a manual backup and it seems to work now.

Didn’t tried the manual backup until i read