Question regarding pbm_config.yaml for aws S3 storage

MongoDB Percona Backup for MongoDB
1

Hello,

I want to configure pbm-agent to backup to S3 with server-side encryption with Amazon S3 managed keys(SSE-S3) , don’t want to use custom key. If I am doing aws s3 cp command, the option is “–sse”.

How to configure above in pbm_config.yamp file? I did it, but it shows error.

storage:
type: s3
s3:
region: us-east-1
bucket: xxxxxxxxxx
prefix: data/pbm/backup
serverSideEncryption:
sseAlgorithm: aws:kms

After restarting pbm-agent, it shows the config like below:

[root@ip-10-88-3-46 etc]# pbm config
pitr:
enabled: false
oplogSpanMin: 0
compression: s2
storage:
type: s3
s3:
provider: aws
region: us-east-1
forcePathStyle: true
bucket: xxxxxxxxx
prefix: data/pbm/backup
credentials: {}
serverSideEncryption:
sseAlgorithm: aws:kms
kmsKeyID: “”
sseCustomerAlgorithm: “”
sseCustomerKey: “”
maxUploadParts: 10000
storageClass: STANDARD
insecureSkipTLSVerify: false
backup:
oplogSpanMin: 0
compression: s2

[root@ip-10-88-3-46 etc]# pbm logs
2024-05-01T20:26:46Z E [rs0/10.88.3.46:27017] [agentCheckup] check storage connection: storage: no init file, attempt to create failed: upload to S3: KMS.ValidationException: 2 validation errors detected: Value ‘’ at ‘keyId’ failed to satisfy constraint: Member must satisfy regular expression pattern: ^\p{ASCII}+$; Value ‘’ at ‘keyId’ failed to satisfy constraint: Member must have length greater than or equal to 1
status code: 400, request id: Z954FFXT09TRQQF9, host id: FJ/4Xz+JmbztxNy2Bq4tVs1UJM+sYwM38XG0ZwQ16BxZjijKiVZiDrHy+LEpyZR5hjrdz5QlsNE=
2024-05-01T20:26:46Z E [rs0/10.88.1.179:27017] [agentCheckup] check storage connection: storage: no init file, attempt to create failed: upload to S3: KMS.ValidationException: 2 validation errors detected: Value ‘’ at ‘keyId’ failed to satisfy constraint: Member must satisfy regular expression pattern: ^\p{ASCII}+$; Value ‘’ at ‘keyId’ failed to satisfy constraint: Member must have length greater than or equal to 1

Can you please assist to get it working ? Thanks!!!

1 Like
3

Thanks! I made change and restart pbm-agent, now the log shows permission errors:

[root@ip-10-88-1-11 etc]# pbm config
pitr:
enabled: false
oplogSpanMin: 0
compression: s2
storage:
type: s3
s3:
provider: aws
region: us-east-1
forcePathStyle: true
bucket: xxxxxx
prefix: data/pbm/backup
credentials: {}
maxUploadParts: 10000
storageClass: STANDARD
insecureSkipTLSVerify: false
backup:
oplogSpanMin: 0
compression: s2
[root@ip-10-88-1-11 etc]#

[root@ip-10-88-1-11 etc]# pbm logs
2024-05-02T13:31:59Z E [rs0/10.88.3.28:27017] [agentCheckup] check storage connection: storage: no init file, attempt to create failed: upload to S3: AccessDenied: Access Denied
status code: 403, request id: WV9N008X68XQM7KR, host id: 3jfy00hreZ+moM1mkFRFfJAb3WeJljfjUX5KTNj+uo2zbR0oh0mCQUkF/P0zkdtTT06VRYDmadE=
2024-05-02T13:32:00Z E [rs0/10.88.2.25:27017] [agentCheckup] check storage connection: storage: no init file, attempt to create failed: upload to S3: AccessDenied: Access Denied

I have EC2 instance profile with read/write policy to the backup bucket. I can run aws cli command to upload file to S3 without error. What permission do I need to grant ? can you show me example ?

[root@ip-10-88-1-11 etc]# touch test-file
[root@ip-10-88-1-11 etc]# aws s3 cp test-file s3://xxxxxx/data/pbm/backup --sse
upload: ./test-file to s3://770079253716-mysql-backup/data/pbm/backup
[root@ip-10-88-1-11 etc]#

1 Like
4

@David_Clark David, Do you know how to fix above error ?

[root@ip-10-88-1-11 etc]# pbm logs
2024-05-02T13:31:59Z E [rs0/10.88.3.28:27017] [agentCheckup] check storage connection: storage: no init file, attempt to create failed: upload to S3: AccessDenied: Access Denied
status code: 403, request id: WV9N008X68XQM7KR, host id: 3jfy00hreZ+moM1mkFRFfJAb3WeJljfjUX5KTNj+uo2zbR0oh0mCQUkF/P0zkdtTT06VRYDmadE=
2024-05-02T13:32:00Z E [rs0/10.88.2.25:27017] [agentCheckup] check storage connection: storage: no init file, attempt to create failed: upload to S3: AccessDenied: Access Denied

1 Like
6

@David_Clark I follow this page to setup S3 bucket permissions

I can run aws s3 cp command to upload files from EC2 to S3 bucket without issue. That’s why I am not sure what’s wrong here. Please see the following S3 policy.

1 Like
7

I have IAM role profile defined to allow S3:* action, mongo is running on EC2.

{
    "Version": "2012-10-17",
    "Statement": [
       {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::mongo-backup"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::mongo-backup/*"
        }
    ]
}