Pbm restore on another environment not working anymore

MongoDB Percona Backup for MongoDB
1

Hi,
after upgrading pbm from 2.3.1 to 2.8.0 something related to our restore procedure stopped working.
We are using minio as s3 compatible storage solution, and the prod config is the following:

# cat /etc/pbm_config.yaml
storage:
  type: s3
  s3:
    endpointUrl: "https://s3.example.com"
    region: us-east-1
    bucket: backup-mongodb
    prefix: clu02_prod
    credentials:
      access-key-id: <read_write_user>
      secret-access-key: xxxxxxxx
backup:
  priority:
    "xxxxxxx:27017": 2.5
    "xxxxxxx:27017": 2.5
    "xxxxxxx:27017": 2.5
restore:
  batchSize: 10

prod is using read-write s3 credentials in order to manage its backups.

Test environment instead is using read-only s3 credential in order to fetch/list prod backups from (blocking any write operation); test config is the following:

# cat /etc/pbm_config.yaml
storage:
  type: s3
  s3:
    endpointUrl: "https://s3.example.com"
    region: us-east-1
    bucket: backup-mongodb
    prefix: clu02_prod
    credentials:
      access-key-id: <readonly_user>
      secret-access-key: xxxxx
restore:
  batchSize: 10

on both, PBM_MONGODB_URI point to local cluster (test → test cluster, prod → prod cluster).

If we need to refresh a test database from prod data, what we used to do was to trigger a restore procedure on test, pointing to prod bucket path (see test config file). To be able to achieve that, a pbm config --force-resync was needed in order to refresh the pbm prod backup list (pbm list).

This always worked like a charm, but after upgrading, force resync on test is not working anymore, keeping the backup list outdated and restore procedure is not usable at all to refresh data on another environment.

The only error displayed is the following (pbm):

[resync] resync: reinit storage: delete init file: delete 'backup-mongodb/.pbm.init' file from S3: 
AccessDenied: Access Denied. status code: 403, request id: 1827691CA65ED6BB, host id: xxxxxxxx

minio:

2025-02-25T10:38:47.844 [206 Partial Content] s3.GetObject s3.example.com/backup-mongodb/clu02_prod/.pbm.init xxx.xxx.xxx.xxx    3.781ms      ⇣  3.731067ms  ↑ 83 B ↓ 5 B
2025-02-25T10:38:47.849 [403 Forbidden] s3.DeleteObject s3.example.com/backup-mongodb/clu02_prod/.pbm.init xxx.xxx.xxx.xxx    133µs       ⇣  122.505µs  ↑ 93 B ↓ 359 B

Any idea on how to make this working again?

2

As you see in the logs, PBM wants to remove an init file and it cannot because your test environment doesn’t provide write/delete permissions. As per doc - it’s required you grant all perms:

if you want to workaround that, you might want to create some other way of removing init files before running a force resync from a test environment. Hope this helps.

3

Since that file is in the same bucket+prefix used by production, does its removal have any impact on the production cluster and its backup?

4

It might cause some friction on backup. See the bug we have solved by introducing the “write” permission requirement and checking the init temp file:
https://perconadev.atlassian.net/browse/PBM-1379

A better choice in your situation would be to have separate buckets for test and prod and introduce some S3 data replication instead (excluding the temp files copy)

5

Thanks.

This solution is not very efficient because I need to duplicate backup files; it would be a better solution reading existing backup files, but if is not possible right now (maybe you can think about that feature) I’ll give it a try.

About temp file, what do you mean? Copy everything except .pbm.init or something else?