Description:
I have a MongoDB cluster in Kubernetes and automated backups running fine to an S3 location.
Steps to Reproduce:
When I try to restore from a backup, using with a configuration like below
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
name: restore-logical-percona-mongodb
namespace: data-mongodb
spec:
clusterName: percona-mongodb
backupName: percona-mongodb-backup-03
storageName: minio
or even if I specify the backupSource with exactly the same configuration like the one for the backups:
backupSource:
type: logical
destination: s3://[bucket]/[date]
s3:
credentialsSecret: credentials-storage-s3
region: k8s
bucket: [bucket]
endpointUrl: http://[url]
insecureSkipTLSVerify: true
prefix: ""
The restore is always failing.
I also tried with an Azure storage and I get the same behaviour, the backups are working but the restore is failing a similar error.
Version:
operator: 1.17.0
Logs:
error: "waiting for start: cluster failed: failed to ensure snapshot file 2024-11 08T13:34:01Z/cfg/metadata.json:
get S3 object header: NoCredentialProviders: no valid providers in chain.
Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors"
Expected Result:
Restore the cluster from the backup.
Actual Result:
The restore object PerconaServerMongoDBRestore is in error state with the above error mesage.
Anyone could help with this issue?
It is a blocker at the moment for us moving forward with Percona Mongo DB
Hey @Iulian_Paraian - backup and restore issues are the worst, as it is quite hard to troubleshoot those.
To debug easier - can you please provide additional logs from backup container? In each replica set node pod there is a sidecar container (backup-agent) that takes backups, it provides much more details about what went wrong.
Hi @Sergey_Pronin
I checked the backup-agent and it is logging the same thing:
From an S3 backup
2024-11-25T10:43:08.000+0000 I [restore/2024-11-25T10:43:07.857597331Z] recovery started
2024-11-25T10:43:08.000+0000 D [restore/2024-11-25T10:43:07.857597331Z] waiting for 'starting' status
2024-11-25T10:43:09.000+0000 E [restore/2024-11-25T10:43:07.857597331Z] restore: failed to ensure snapshot file 2024-11-25T10:38:45Z/rs-0/metadata.json: get S3 object header: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
From Azure blob storage backup:
2024-11-25T10:32:52.000+0000 D [restore/2024-11-25T10:32:51.830161887Z] waiting for 'starting' status
2024-11-25T10:32:53.000+0000 E [restore/2024-11-25T10:32:51.830161887Z] restore: get backup storage: check container: GET https://xxx.blob.core.windows.net/mongodb-backup
--------------------------------------------------------------------------------
RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
ERROR CODE: AuthenticationFailed
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:a04ba4b8-801e-002c-3925-3f4e8f000000
Time:2024-11-25T10:32:53.1753992Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'kCtxmpUWoVoSpIV8e66gEG+71+O9fx5D/sIkXYHqv/E=' is not the same as any computed signature. Server used following string to sign: 'GET
x-ms-date:Mon, 25 Nov 2024 10:32:53 GMT
x-ms-version:2023-11-03
/xxx/mongodb-backup
restype:container'.</AuthenticationErrorDetail></Error>
The restore configuration is the same with the one for the backups, that are both working in an S3 bucket and Azure blob storage.
And you are 100% sure that credentials are valid, right? I assume, you use the same creds to store backups and to restore.
Yes, both the backups and the restores are using the same secret.
Hi @Sergey_Pronin,
Any other suggestions?
I had the same issue when testing an upgrade from PBM 2.0.4 to 2.10 with Azure backups.
The cause was that the account key was not being added to the admin.pbmBackups entity: store.azure.credentials.key was ““. Setting it fixed the issue.