Description:
I have a MongoDB cluster in Kubernetes and automated backups running fine to an S3 location.
Steps to Reproduce:
When I try to restore from a backup, using with a configuration like below
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
name: restore-logical-percona-mongodb
namespace: data-mongodb
spec:
clusterName: percona-mongodb
backupName: percona-mongodb-backup-03
storageName: minio
or even if I specify the backupSource with exactly the same configuration like the one for the backups:
backupSource:
type: logical
destination: s3://[bucket]/[date]
s3:
credentialsSecret: credentials-storage-s3
region: k8s
bucket: [bucket]
endpointUrl: http://[url]
insecureSkipTLSVerify: true
prefix: ""
The restore is always failing.
I also tried with an Azure storage and I get the same behaviour, the backups are working but the restore is failing a similar error.
Version:
operator: 1.17.0
Logs:
error: "waiting for start: cluster failed: failed to ensure snapshot file 2024-11 08T13:34:01Z/cfg/metadata.json:
get S3 object header: NoCredentialProviders: no valid providers in chain.
Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors"
Expected Result:
Restore the cluster from the backup.
Actual Result:
The restore object PerconaServerMongoDBRestore is in error state with the above error mesage.
Anyone could help with this issue?
It is a blocker at the moment for us moving forward with Percona Mongo DB
Hey @Iulian_Paraian - backup and restore issues are the worst, as it is quite hard to troubleshoot those.
To debug easier - can you please provide additional logs from backup container? In each replica set node pod there is a sidecar container (backup-agent) that takes backups, it provides much more details about what went wrong.
Hi @Sergey_Pronin
I checked the backup-agent and it is logging the same thing:
From an S3 backup
2024-11-25T10:43:08.000+0000 I [restore/2024-11-25T10:43:07.857597331Z] recovery started
2024-11-25T10:43:08.000+0000 D [restore/2024-11-25T10:43:07.857597331Z] waiting for 'starting' status
2024-11-25T10:43:09.000+0000 E [restore/2024-11-25T10:43:07.857597331Z] restore: failed to ensure snapshot file 2024-11-25T10:38:45Z/rs-0/metadata.json: get S3 object header: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
From Azure blob storage backup:
2024-11-25T10:32:52.000+0000 D [restore/2024-11-25T10:32:51.830161887Z] waiting for 'starting' status
2024-11-25T10:32:53.000+0000 E [restore/2024-11-25T10:32:51.830161887Z] restore: get backup storage: check container: GET https://xxx.blob.core.windows.net/mongodb-backup
--------------------------------------------------------------------------------
RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
ERROR CODE: AuthenticationFailed
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:a04ba4b8-801e-002c-3925-3f4e8f000000
Time:2024-11-25T10:32:53.1753992Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'kCtxmpUWoVoSpIV8e66gEG+71+O9fx5D/sIkXYHqv/E=' is not the same as any computed signature. Server used following string to sign: 'GET
x-ms-date:Mon, 25 Nov 2024 10:32:53 GMT
x-ms-version:2023-11-03
/xxx/mongodb-backup
restype:container'.</AuthenticationErrorDetail></Error>
The restore configuration is the same with the one for the backups, that are both working in an S3 bucket and Azure blob storage.
And you are 100% sure that credentials are valid, right? I assume, you use the same creds to store backups and to restore.
Yes, both the backups and the restores are using the same secret.
Hi @Sergey_Pronin,
Any other suggestions?