# ERROR with storage: check storage connection: storage check failed with: get S3 object header: Forbidden: Forbidden

**URL:** <https://forums.percona.com/t/error-with-storage-check-storage-connection-storage-check-failed-with-get-s3-object-header-forbidden-forbidden/30333>\
**Category:** Percona Operator for MongoDB\
**Created:** [May 15, 2024, 12:01pm UTC](https://forums.percona.com/t/error-with-storage-check-storage-connection-storage-check-failed-with-get-s3-object-header-forbidden-forbidden/30333 "2024-05-15T12:01:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![RogerSik](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rogersik/32/16446_2.png) [@RogerSik](https://forums.percona.com/u/RogerSik)\
**Post date:** [May 15, 2024, 12:01pm UTC](https://forums.percona.com/t/error-with-storage-check-storage-connection-storage-check-failed-with-get-s3-object-header-forbidden-forbidden/30333/1 "2024-05-15T12:01:10Z")

</div>

The backup-agent is continously reporting that they have a forbidden request.

```auto
k logs -f pod/mongodb-cluster-rs0-1 -c backup-agent

```

```auto
2024-05-15T11:58:35.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: Forbidden: Forbidden
        status code: 403, request id: 3G4YBFG1QYCMDYJ6, host id: niliFnpNtKmcDQqnLTjWHAUJ2PxjmVDoKTo1+Gw/ikA/GejsglxtI6tFITfPCOYS7+Md1W8Dd9g=

```

But with Cyberduck im able to download+upload files. s3:\* is everything allowed.

cr.yaml

```auto
# https://github.com/percona/percona-server-mongodb-operator/blob/v1.15.0/deploy/cr.yaml
---
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: mongodb-cluster
  namespace: percona-mongodb
  finalizers:
    - delete-psmdb-pods-in-order
spec:
  crVersion: 1.15.0
  image: percona/percona-server-mongodb:6.0.9-7
  imagePullPolicy: IfNotPresent
  allowUnsafeConfigurations: false
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: '0 2 * * *'
    setFCV: false
  secrets:
    users: mongodb-cluster
    encryptionKey: mongodb-cluster-encryption-key
  pmm:
    enabled: true
    image: percona/pmm-client:2.39.0
    serverHost: monitoring-service.percona-pmm.svc.cluster.local
  replsets:
    - name: rs0
      size: 3
      affinity:
        antiAffinityTopologyKey: 'kubernetes.io/hostname'
      tolerations:
        - key: 'node-app'
          operator: 'Equal'
          value: 'mongodb'
          effect: 'NoExecute'
      podDisruptionBudget:
        maxUnavailable: 1
      expose:
        enabled: false
        exposeType: ClusterIP
      resources:
        limits:
          cpu: '2'
          memory: '4Gi'
        requests:
          cpu: '0.5'
          memory: '1Gi'
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 100Gi

      nonvoting:
        enabled: false
        size: 3
        affinity:
          antiAffinityTopologyKey: 'kubernetes.io/hostname'
        podDisruptionBudget:
          maxUnavailable: 1
        resources:
          limits:
            cpu: '300m'
            memory: '0.5G'
          requests:
            cpu: '300m'
            memory: '0.5G'
        volumeSpec:
          persistentVolumeClaim:
            resources:
              requests:
                storage: 3Gi
      arbiter:
        enabled: false
        size: 1
        affinity:
          antiAffinityTopologyKey: 'kubernetes.io/hostname'
        resources:
          limits:
            cpu: '300m'
            memory: '0.5G'
          requests:
            cpu: '300m'
            memory: '0.5G'

  sharding:
    enabled: false
    configsvrReplSet:
      size: 3
      affinity:
        antiAffinityTopologyKey: 'kubernetes.io/hostname'
      podDisruptionBudget:
        maxUnavailable: 1
      expose:
        enabled: false
        exposeType: ClusterIP
      resources:
        limits:
          cpu: '300m'
          memory: '0.5G'
        requests:
          cpu: '300m'
          memory: '0.5G'
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi

    mongos:
      size: 3
      affinity:
        antiAffinityTopologyKey: 'kubernetes.io/hostname'
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: '300m'
          memory: '0.5G'
        requests:
          cpu: '300m'
          memory: '0.5G'
      expose:
        exposeType: ClusterIP

  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.3.0
    serviceAccountName: percona-server-mongodb-operator
    resources:
      limits:
        memory: 1Gi
      requests:
        memory: 500Mi
        cpu: 300m
    pitr:
      enabled: false
      compressionType: gzip
      compressionLevel: 6
    storages:
      aws-s3:
        type: s3
        s3:
          bucket: 'change_me'
          credentialsSecret: 'mongodb-backup-s3'
          region: 'eu-north-1'

```

Bucket S3 user iam permission:

```auto
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::change_me",
                "arn:aws:s3:::change_me/*"
            ],
            "Sid": "AllowAccess"
        }
    ]
}

```

---

<div class="post-metadata">

**Author:** ![RogerSik](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rogersik/32/16446_2.png) [@RogerSik](https://forums.percona.com/u/RogerSik)\
**Post date:** [May 15, 2024, 12:07pm UTC](https://forums.percona.com/t/error-with-storage-check-storage-connection-storage-check-failed-with-get-s3-object-header-forbidden-forbidden/30333/2 "2024-05-15T12:07:52Z")

</div>

Ah not sure what was the solution.

I recreated the credentials. The error still appeared but then i created a manual backup and it seems to work now.

Didn’t tried the manual backup until i read

> [@ERROR with storage: storage check failed with: get S3 object header: RequestError: send request failed](https://forums.percona.com/t/error-with-storage-storage-check-failed-with-get-s3-object-header-requesterror-send-request-failed/10433/8):
>
> Even you change the endpointUrl in cr.yaml, it will be updated in PBM (backup-agent) container only after you try to take another backup or perform the restoration. Could you please set endpointUrl in cr.yaml with http and try to take the backup through psmdb-backup CR?