How to setup AWS S3 Storage Backup using service account (IRSA)

MongoDB Percona Backup for MongoDB
1

Hi friends

I want to setup s3 bucket for backups. I have looked at the following documentation - Configure storage for backups - Percona Operator for MongoDB

Unfortunately it mentions very little about AWS IRSA but ive not been able to get it to work.

What am i doing wrong, is this even possible?

backup:
  enabled: true
  image:
    repository: percona/percona-backup-mongodb
    tag: 2.0.4
  serviceAccountName: percona-server-mongodb-operator
  storages:
    s3-eu-west:
      type: s3
      s3:
        bucket: arn:aws:s3:::<redacted>
        region: eu-west-2

kubectl -n mongodb logs -f psmdb-db-rs0-0 backup-agent

+ PBM_MONGODB_URI='mongodb://backup:rdbeye8HR0WMpTjrwl@localhost:27017/?replicaSet=rs0'
+ MONGO_SSL_DIR=/etc/mongodb-ssl
+ [[ -f /etc/mongodb-ssl/tls.crt ]]
+ export PBM_MONGODB_URI
+ exec pbm-agent-entrypoint
2023/04/25 13:49:40 [entrypoint] starting `pbm-agent`
2023-04-25T13:49:43.000+0000 I pbm-agent:
Version:   2.0.4
Platform:  linux/amd64
GitCommit: 785ee592ade9eb86be656af0af4da73b2f6055e1
GitBranch: release-2.0.4
BuildTime: 2023-02-15_17:02_UTC
GoVersion: go1.19
2023-04-25T13:49:43.000+0000 I starting PITR routine
2023-04-25T13:49:43.000+0000 I node: rs0/psmdb-db-rs0-0.psmdb-db-rs0.mongodb.svc.cluster.local:27017
2023-04-25T13:49:43.000+0000 I listening for the commands
2023-04-25T13:49:48.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:49:53.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:49:58.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:03.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:08.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:13.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:18.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:23.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2023-04-25T13:50:28.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result

kubectl describe sa percona-server-mongodb-operator -n mongodb

I created this SA using terraform it has a role applied to it, that role has permissions.

Name:                percona-server-mongodb-operator
Namespace:           mongodb
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<redacted>:role/g<redacted>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Policy

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<redacted>"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<redacted>"
            ]
        }
    ],
    "Version": "2012-10-17"
}

it looks like this should be possible as of 2.0.3 - Automate access to S3 buckets for PBM - Percona Backup for MongoDB

I’m not sure how to debug this, can someone help me interpret the error message?How to setup AWS S3 Storage Backup using service account (IRSA)

2

Did you ever figure it out? In my case, even though I’ve correctly set backup.serviceAccountName, the setting is seemingly ignored. Specifically with this:

  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.0.4
    serviceAccountName: s3-rw
    storages:
      default:
        s3:
          provider: s3
          bucket: my-backup-bucket
          prefix: mongodb
          region: eu-west-1
        type: s3
    pitr:
      enabled: false
    tasks:
      - enabled: true
        name: once_daily
        schedule: '* * * * *'
        storageName: default

The backup-agent container on the replicaset pod should have on its spec serviceAccountName: s3-rw, however, the container definition does not set anything. Predictably, looking in cloudtrail, I can see the requests have all fallen back to the worker instance profile. Do I understand wrong how this is meant to work?

3

No i never got percona s3 backup to work. I gave up trying and now i just have a cron job running doing a mongodump. Let me know if you find a solution. I don’t have the time to revisit this right now.

4

I’m running into this issue, the provided serviceAccountName is not being used by the operator to create the pods. the pods get runs with default service account. If I added the needed permission to that account backup runs fine. Since the backup agent is a sidecar, we need the ability to pick which service account runs the cluster instead of setting default. the helm chart doesn’t support setting the service account. I’m not sure if the crd supports it.

5

It certainly looks like a bug in Operator rather than the helm chart. More info and a workaround here @tra_for Operator doesn't apply the provided service account to the backup containers - #2 by Luis_Pabon

6

@tra_for try to set serviceAccountName on rs and configsvrReplSet (in case of sharding) level.

7

I’m using the helm chart, they got ignored. it works fine inserting in crd itself. We need the helm chart to support this? This should be pretty straightforward to add. Do you guys accept PR?

8

This issue is resolved in Percona MongoDB Operator version 1.15.