Check storage connection: storage: no init file, attempt to create failed: upload to S3: AccessDenied: Access Denied

MongoDB Percona Backup for MongoDB
1

Hi friends, i have installed mongodb operator and the psmdb to my test AWS EKS Cluster.

im having trouble setting up s3 backup using kubernetes secret. Can someone tell me what i could be doing wrong here?

I get the following error:

2023-04-26T14:04:07.000+0000 E [agentCheckup] check storage connection: storage: no init file, attempt to create failed: upload to S3: AccessDenied: Access Denied
status code: 403, request id: 3TEBJCYVHD97FJQG, host id: QZvnwwmOM3AGqZvKsMPsTAZCqAcZ199f/lRDM+Q4ChNeGSfGvzA1OzFpa+puqn+HGRIQy19N7Kc=

kubectl -n mongodb describe secret prod-aws-mongodb

Name:         prod-aws-mongodb
Namespace:    mongodb
Labels:       <none>
Annotations:  reconcile.external-secrets.io/data-hash: 46e613eded7184a44bd53358c5dcfa69

Type:  Opaque

Data
====
AWS_ACCESS_KEY_ID:      20 bytes
AWS_SECRET_ACCESS_KEY:  40 bytes

values.yaml

  storages:
    s3-eu-west:
      type: s3
      s3:
        bucket: acme-test-mongodb-backup
        credentialsSecret: prod-aws-mongodb
        region: eu-west-2
        prefix: ""
        uploadPartSize: 10485760
        maxUploadParts: 10000
        storageClass: STANDARD
        insecureSkipTLSVerify: false
  tasks:
   - name: "every-hour-backup"
     enabled: true
     schedule: "0 * * * *"
     keep: 3
     type: logical
     storageName: s3-eu-west

pbm status

Cluster:
========
rs0:
  - rs0/psmdb-db-rs0-0.psmdb-db-rs0.mongodb.svc.cluster.local:27017 [P]: pbm-agent v2.1.0 OK


PITR incremental backup:
========================
Status [OFF]

Currently running:
==================
(none)

Backups:
========
S3 eu-west-2 s3://acme-test-mongodb-backup
  Snapshots:
    2023-04-26T14:00:21Z 0.00B <logical> [ERROR: Backup stuck at `starting` stage, last beat ts: 1682517621] [2023-04-26T14:00:21Z]

pbm config


pitr:
  enabled: false
  oplogSpanMin: 0
  compression: s2
storage:
  type: s3
  s3:
    provider: aws
    region: eu-west-2
    bucket: acme-test-mongodb-backup
    credentials:
      access-key-id: '***'
      secret-access-key: '***'
    uploadPartSize: 10485760
    maxUploadParts: 10000
    storageClass: STANDARD
    insecureSkipTLSVerify: false
backup:
  priority:
    psmdb-db-rs0-0.psmdb-db-rs0.mongodb.svc.cluster.local:27017: 0.5
  compression: s2

Ive given the user behind the access credentials full access

image
image2175×231 19.4 KB

2

Looks like this is a bug in the latest version

backup:
  enabled: true
  image:
    repository: percona/percona-backup-mongodb
    tag: 2.1.0

I switched the tag to 2.0.5 and the same setup worked

Update:

maybe not, i teard everything down and rebuilt it, having the same issue on 2.0.5. It did work when i initially swapped image. because i have a backup in s3

3

Hi, How did you fix it please? I have same problem on AWS EC2

4

No i gave up using the percona snapshot and decided to use mongodump setup with a cron job. But i plan to revist this in the future

5

Alright, I have to fix it, mongodump doesn’t meet my requirement any more. I will reply if I solve it.Thank you.