We tried to restore a backup to a new Kubernetes cluster.
Kubernetes cluster 1.20.5
Percona operator 1.11
Mongo 4.4.10-11
That means all databases incl admin db
Backup was taken via full backup and additionals pitr backups.
Restore workes ok.
After the restore the system accounts ( userAmdin, clusterAdmin etc) kan no longer login with the passwords which were valid for the backup (Access denied)
Logs shows i.ex:
"AuthenticationFailed: SCRAM authentication failed, storedKey mismatch " for clusterMonitor user
Even invalid certificate is shown in the logs:
“SSL peer certificate validation failed”
Pods are periodically going to non-ready state and are restarted due to not being healthy.
Funny enough : a non system user which was created before the backup was taken kan still login and sees all data.
We followed the Percona docs but maybe we miss something ?
Do we have to do additional tasks ?
Thx in advance for any help
/Frank
2 Likes
Update on this:
We investigated and could see that this was caused by the operator creating the “external” user-secret on it’s own when creating the cluster.
The secret consists av hard coded users with random passwords so the passwords passed via the external secret were not used.
Due to this, we were not able to login after the restore.
This is a strange behaviour and we will open a new topic for this
1 Like
@frank2b this is indeed strange. Thank you for sharing. Please let me know once you create the ticket with the steps to reproduce. We will try to address this in the next release.
1 Like
@Sergey_Pronin Pls check this issue:
where I described it in more detail and the steps to reproduce it.
Not that simple due to timing issue.
B.r
Frank
1 Like