How to setup AWS S3 Storage Backup using service account (IRSA)

MongoDB Percona Operator for MongoDB
1

Hi friends

I want to setup s3 bucket for backups. I have looked at the following documentation - Configure storage for backups - Percona Operator for MongoDB

Unfortunately it mentions very little about AWS IRSA but ive not been able to get it to work.

What am i doing wrong, is this even possible?

backup:
  enabled: true
  image:
    repository: percona/percona-backup-mongodb
    tag: 2.0.4
  serviceAccountName: percona-server-mongodb-operator
  storages:
    s3-eu-west:
      type: s3
      s3:
        bucket: arn:aws:s3:::<redacted>
        region: eu-west-2

kubectl -n mongodb logs -f psmdb-db-rs0-0 backup-agent

+ PBM_MONGODB_URI='mongodb://backup:rdbeye8HR0WMpTjrwl@localhost:27017/?replicaSet=rs0'
+ MONGO_SSL_DIR=/etc/mongodb-ssl
+ [[ -f /etc/mongodb-ssl/tls.crt ]]
+ export PBM_MONGODB_URI
+ exec pbm-agent-entrypoint
2023/04/25 13:49:40 [entrypoint] starting `pbm-agent`
2023-04-25T13:49:43.000+0000 I pbm-agent:
Version:   2.0.4
Platform:  linux/amd64
GitCommit: 785ee592ade9eb86be656af0af4da73b2f6055e1
GitBranch: release-2.0.4
BuildTime: 2023-02-15_17:02_UTC
GoVersion: go1.19
2023-04-25T13:49:43.000+0000 I starting PITR routine
2023-04-25T13:49:43.000+0000 I node: rs0/psmdb-db-rs0-0.psmdb-db-rs0.mongodb.svc.cluster.local:27017
2023-04-25T13:49:43.000+0000 I listening for the commands
2023-04-25T13:49:48.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:49:53.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:49:58.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:03.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:08.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:13.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:18.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:23.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:28.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result

kubectl describe sa percona-server-mongodb-operator -n mongodb

I created this SA using terraform it has a role applied to it, that role has permissions.

Name:                percona-server-mongodb-operator
Namespace:           mongodb
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<redacted>:role/g<redacted>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Policy

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<redacted>"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<redacted>"
            ]
        }
    ],
    "Version": "2012-10-17"
}

it looks like this should be possible as of 2.0.3 - Automate access to S3 buckets for PBM - Percona Backup for MongoDB

I’m not sure how to debug this, can someone help me interpret the error message?

2

Hi @Kay_Khan !
I think these messages appear until you do the first backup because when the first backup is initiated then operator sets storage for PBM so until then PBM will complain.
So to test your setup I would suggest to just try to execute the backup.
Second thing is I’m not sure you can specify the bucket name in the format you have specified it.

There is a test for this case in our repo so you can check some setup there: percona-server-mongodb-operator/e2e-tests/demand-backup-eks-credentials at main · percona/percona-server-mongodb-operator · GitHub

You can find some more info on how to test your setup in the description of this PR: K8SPSMDB-185: Allow PBM to get S3 credentials from EC2 metadata by egegunes · Pull Request #903 · percona/percona-server-mongodb-operator · GitHub

I hope this helps a bit.

3

Youre right if i use the credentialsSecret these message come up the very first time.

But when using IRSA i get the following error when the backup task is executed:

2023-04-26T14:04:07.000+0000 E [agentCheckup] check storage connection: storage: no init file, attempt to create failed: upload to S3: AccessDenied: Access Denied
status code: 403, request id: 3TEBJCYVHD97FJQG, host id: QZvnwwmOM3AGqZvKsMPsTAZCqAcZ199f/lRDM+Q4ChNeGSfGvzA1OzFpa+puqn+HGRIQy19N7Kc=

Since writing this question i have corrected the bucket name and the problem persists.

If im not mistaken, IRSA and EC2 metadata are 2 different authentication solutions?