# Operator cannot start PMM because keys don't exist in the secrets, or secrets and internal secrets are out of sync? **URL:** \ **Category:** Percona Operator for MongoDB\ **Tags:** pmm\ **Created:** [September 3, 2025, 10:27am UTC](https://forums.percona.com/t/operator-cannot-start-pmm-because-keys-dont-exist-in-the-secrets-or-secrets-and-internal-secrets-are-out-of-sync/39266 "2025-09-03T10:27:54Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![willr](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/willr/32/21916_2.png) [@willr](https://forums.percona.com/u/willr)\ **Post date:** [September 3, 2025, 10:27am UTC](https://forums.percona.com/t/operator-cannot-start-pmm-because-keys-dont-exist-in-the-secrets-or-secrets-and-internal-secrets-are-out-of-sync/39266/1 "2025-09-03T10:27:54Z")
Sorry, this is probably very basic but there’s a gap in the documentation between PSMDB Operator, MongoDB and PMM around exactly how to supply authentication secrets to pmm-admin. tldr: Please could someone tell me the right way to specify PMM\_SERVER\_PASSWORD, PMM\_SERVER\_USER and PMM\_SERVER\_API\_KEY in the MongoDB Operator? I’m using terraform if that makes any difference. I have tried following the basic [installation steps](https://docs.percona.com/percona-operator-for-mongodb/monitoring.html), but the operator doesn’t expose any values for the secrets.yml file. Creating those secrets in advance got me some of the way but I remain confused about what the deployed secret name should be. By accessing the pmm-client container I could get pmm-admin to register but I could not add the MongoDB service. Studying the mongodb [helm chart](https://github.com/percona/percona-helm-charts/blob/b5b958c9ad482e725f486852fe9bd6e335ba91a6/charts/psmdb-db/values.yaml) reveals that setting `systemUsers` will populate ‘the secret’; this creates a secret called [database-name-secrets](https://github.com/percona/percona-helm-charts/blob/b5b958c9ad482e725f486852fe9bd6e335ba91a6/charts/psmdb-db/templates/cluster-secret.yaml), but now the operator refuses to start the PMM container because ‘keys don’t exist in the secrets, or secrets and internal secrets are out of sync’. PMM Server and client are v2.44, operator is latest. ```bash 2025-09-03T10:07:41.407Z ERROR secrets {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-production-psmdb","namespace":"mongodb"}, "namespace": "mongodb", "name": "mdb-production-psmdb", "reconcileID": "66d665ca-d1dd-402a-8994-c1fe967efb57", "percona-server-mongodb-users": "internalSecrets", "error": "Can't enable PMM: \"PMM_SERVER_API_KEY\" or \"PMM_SERVER_USER\" with \"PMM_SERVER_PASSWORD\" keys don't exist in the secrets, or secrets and internal secrets are out of sync", "errorVerbose": "Can't enable PMM: \"PMM_SERVER_API_KEY\" or \"PMM_SERVER_USER\" with \"PMM_SERVER_PASSWORD\" keys don't exist in the secrets, or secrets and internal secrets are out of sync\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileUsers\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/users.go:54\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"} ``` ## ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [September 3, 2025, 11:00am UTC](https://forums.percona.com/t/operator-cannot-start-pmm-because-keys-dont-exist-in-the-secrets-or-secrets-and-internal-secrets-are-out-of-sync/39266/2 "2025-09-03T11:00:42Z")
Hi, have you seen this page? [5. Monitor the database with PMM - Percona Operator for MongoDB](https://docs.percona.com/percona-operator-for-mongodb/monitoring-tutorial.html#install-pmm-client) The name of the secret needs to contain to be `-secrets` basically you need to deploy the secrets.yaml file that is included with Operator with either API key or user/pass combination. ---
**Author:** ![willr](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/willr/32/21916_2.png) [@willr](https://forums.percona.com/u/willr)\ **Post date:** [September 3, 2025, 3:35pm UTC](https://forums.percona.com/t/operator-cannot-start-pmm-because-keys-dont-exist-in-the-secrets-or-secrets-and-internal-secrets-are-out-of-sync/39266/4 "2025-09-03T15:35:23Z")
Hi Ivan. Thanks for your reply. Yes, that’s the page I linked to. I have populated the secret by setting `serviceUsers` in the values file: ```auto Name: mdb-production-psmdb-secrets Namespace: mongodb Labels: app.kubernetes.io/instance=mdb-production app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=psmdb-db app.kubernetes.io/version=1.20.1 helm.sh/chart=psmdb-db-1.20.1 Annotations: meta.helm.sh/release-name: mdb-production meta.helm.sh/release-namespace: mongodb Type: Opaque Data ==== MONGODB_USER_ADMIN_PASSWORD: 12 bytes PMM_SERVER_PASSWORD: 12 bytes PMM_SERVER_USER: 5 bytes MONGODB_BACKUP_USER: 5 bytes MONGODB_CLUSTER_ADMIN_PASSWORD: 12 bytes MONGODB_CLUSTER_ADMIN_USER: 5 bytes MONGODB_CLUSTER_MONITOR_PASSWORD: 12 bytes MONGODB_CLUSTER_MONITOR_USER: 5 bytes MONGODB_DATABASE_ADMIN_USER: 5 bytes MONGODB_USER_ADMIN_USER: 5 bytes PMM_SERVER_API_KEY: 80 bytes MONGODB_BACKUP_PASSWORD: 12 bytes MONGODB_DATABASE_ADMIN_PASSWORD: 12 bytes ``` The operator also creates other secrets: ```auto internal-mdb-production-psmdb-users Opaque 20 116m percona-server-mongodb-users Opaque 10 116m ``` And I get the error mentioned above: ```auto Can't enable PMM... keys don’t exist in the secrets, or secrets and internal secrets are out of sync. ``` I deleted all these secrets before running the operator this time, so they were all created in the same run. ETA: @Ivan_Groenewold further down this rabbit hole I see that `percona-server-mongodb-users` is the default user secret name [defined in the operator](https://github.com/percona/percona-server-mongodb-operator/blob/4fd25561859eff8b07bc9d25d1b0462c64b5437d/pkg/apis/psmdb/v1/psmdb_defaults.go#L43). That secret is being created from the default cr-secrets.yaml and then copied as `internal-percona-server-mongodb-users`. The `-secrets` doesn’t seem to do anything at all? ---
**Author:** ![Natalia\_Marukovich](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/natalia_marukovich/32/6407_2.png) [@Natalia\_Marukovich](https://forums.percona.com/u/Natalia_Marukovich)\ **Post date:** [September 4, 2025, 10:42am UTC](https://forums.percona.com/t/operator-cannot-start-pmm-because-keys-dont-exist-in-the-secrets-or-secrets-and-internal-secrets-are-out-of-sync/39266/5 "2025-09-04T10:42:54Z")
Hi, @willr ! As you noted operator created its own secrets. It means that you need to specify the secret you want to use in cr. Could you please check this doc [Application and system users - Percona Operator for MongoDB](https://docs.percona.com/percona-operator-for-mongodb/users.html#system-users) ?