Hi,
I’ve setup Percona Server for MongoDB on a bare metal kubernetes cluster and things are mostly working fine. I’ve enabled storage on an S3 compatible storage and use SSE-C to encrypt the backups.
On the cluster i’ve tried different recovery scenarios and everything is working as I want which is great.
However, i’m running into some troubles trying to restore locally from the S3 storage
What i’ve done is setup is:
- create a readonly s3 key so the restore absolutely does not impact the backups
- start a local percona-server-mongodb:8.0.19-7 docker container (same as kubernetes cluster)
- start a local percona-backup-mongodb:2.12.0 docker container with a config pointing to the prod S3 storage with the readonly user (same as kubernetes cluster)
The issue is that pbm list by default shows nothing, so I try to force a resync to update the metadata but I get a permission error
docker exec -it pbm pbm config --force-resync --wait
Error: waiting for resync [opid "6a22d1645c1d961583a1e455"]: resync: reinit storage: delete init file: delete 'mongo-backups/.pbm.init' file from S3: operation error S3: DeleteObject, https response error StatusCode: 403, RequestID: tx8c2b460608fe497a85ba8-006a22d165, HostID: tx8c2b460608fe497a85ba8-006a22d165, api error AccessDenied: Access Denied.
I understand that pbm is trying to delete .pbm.init but I’m not too keen on the restore being able to modify the storage. It feels like I’m missing something, is it normal for the resync to modify the pbm.init? any suggestions on how to approach this?
Thanks!
P.S: percona/percona-backup-mongodb - Docker Image documentation isn’t correct, I believe the commands at the bottom should be docker exec and not docker run
percona config
storage:
type: s3
s3:
region: gra
endpointUrl: https://s3.gra.io.cloud.ovh.net
forcePathStyle: true
bucket: mongo-backups
credentials:
access-key-id: redacted
secret-access-key: redacted
serverSideEncryption:
sseCustomerAlgorithm: AES256
sseCustomerKey: redacted
docker commands
docker network create pbm
docker run --network pbm --name pbm_mongo -d percona/percona-server-mongodb:8.0.19-7 --replSet=rs0
docker exec -it pbm_mongo mongosh --eval='rs.initiate()'
docker run --network pbm --name pbm --mount type=bind,source=$(pwd)/pbm_config.yaml,target=/tmp/pbm_config.yaml,readonly=true -e PBM_MONGODB_URI="mongodb://pbm_mongo:27017" -d percona/percona-backup-mongodb:2.12.0
docker exec -it pbm pbm config --file /tmp/pbm_config.yaml
docker exec -it pbm pbm config --force-resync --wait
EDIT: looking at the code write is necessary – my question is can I resync or at least import locally in read only?
// Resync sync oplog, backup, and restore meta from provided storage.
//
// It checks for read and write permissions, drops all meta from the database
// and populate it again by reading meta from the storage.
func Resync(
ctx context.Context,
conn connect.Client,
cfg *config.StorageConf,
node string,
includeRestores bool,
) error {
l := log.LogEventFromContext(ctx)
stg, err := util.StorageFromConfig(cfg, node, l)
if err != nil {
return errors.Wrap(err, "unable to get backup store")
}
err = storage.HasReadAccess(ctx, stg)
if err != nil {
if !errors.Is(err, storage.ErrUninitialized) {
return errors.Wrap(err, "check read access")
}
err = util.Initialize(ctx, stg)
if err != nil {
return errors.Wrap(err, "init storage")
}
} else {
// check write permission and update PBM version
err = util.Reinitialize(ctx, stg)
if err != nil {
return errors.Wrap(err, "reinit storage")
}
}
err = SyncBackupList(ctx, conn, cfg, "", node)
if err != nil {
l.Error("failed sync backup metadata: %v", err)
}
err = resyncOplogRange(ctx, conn, stg)
if err != nil {
l.Error("failed sync oplog range: %v", err)
}
err = resyncPhysicalRestores(ctx, conn, stg, includeRestores)
if err != nil {
l.Error("failed sync physical restore metadata: %v", err)
}
return nil
}