Environment:
Kubernetes 1.20.5
Mongodb operator 1.11
Mongodb 4.4.10
Deployment of op and db via Helm chart
Config:
simple replicat set with 3 nodes (no shards)
user-secret provided by “external secrets” (values from Vault) under db deployment
Problem:
The operator deployment works without any issues.
Deployment of the database works as well.
BUT:
The operator mostly creates the external secret itself with wrong values (hardcoded users and random passwords from percona-server-mongodb-operator/secrets.go at main · percona/percona-server-mongodb-operator · GitHub, line 79 etc)
This leads to a running cluster with “wrong passwords and users” where we ran into login problem after a restore (of course).
Question:
Why is the secret created at all when an external secret is defined ?
It is not clear to us and from our point of view, this should never happens.
It seems to be a timing problem when the secret is not in place when the operator starts deploying the database. When deploying the secret manually in advance, the beahviour is correct (what we could test).
It takes a few second for external secret to provide the secret. Sometimes, the operator states “secret not found” in the logs which is correct, but mostely, the operator creates the secret itself.
This behaviour prevents us from auto-deployment via pipelines, too, because the operator is not cluster-wide-aware yet and has to be deployed per db instance (where each runs in it’s own namespace).
Any suggestion very welcome.
Thx in advance
/Frank