Operator cannot start PMM because keys don't exist in the secrets, or secrets and internal secrets are out of sync?

MongoDB Percona Operator for MongoDB
1

Sorry, this is probably very basic but there’s a gap in the documentation between PSMDB Operator, MongoDB and PMM around exactly how to supply authentication secrets to pmm-admin.

tldr: Please could someone tell me the right way to specify PMM_SERVER_PASSWORD, PMM_SERVER_USER and PMM_SERVER_API_KEY in the MongoDB Operator? I’m using terraform if that makes any difference.

I have tried following the basic installation steps, but the operator doesn’t expose any values for the secrets.yml file. Creating those secrets in advance got me some of the way but I remain confused about what the deployed secret name should be. By accessing the pmm-client container I could get pmm-admin to register but I could not add the MongoDB service.

Studying the mongodb helm chart reveals that setting systemUsers will populate ‘the secret’; this creates a secret called database-name-secrets, but now the operator refuses to start the PMM container because ‘keys don’t exist in the secrets, or secrets and internal secrets are out of sync’.

PMM Server and client are v2.44, operator is latest.

2025-09-03T10:07:41.407Z	ERROR	secrets	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-production-psmdb","namespace":"mongodb"}, "namespace": "mongodb", "name": "mdb-production-psmdb", "reconcileID": "66d665ca-d1dd-402a-8994-c1fe967efb57", "percona-server-mongodb-users": "internalSecrets", "error": "Can't enable PMM: \"PMM_SERVER_API_KEY\" or \"PMM_SERVER_USER\" with \"PMM_SERVER_PASSWORD\" keys don't exist in the secrets, or secrets and internal secrets are out of sync", "errorVerbose": "Can't enable PMM: \"PMM_SERVER_API_KEY\" or \"PMM_SERVER_USER\" with \"PMM_SERVER_PASSWORD\" keys don't exist in the secrets, or secrets and internal secrets are out of sync\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileUsers\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/users.go:54\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}

1 Like
2

Hi, have you seen this page? 5. Monitor the database with PMM - Percona Operator for MongoDB

The name of the secret needs to contain to be <your_cluster_name>-secrets

basically you need to deploy the secrets.yaml file that is included with Operator with either API key or user/pass combination.

4

Hi Ivan. Thanks for your reply. Yes, that’s the page I linked to. I have populated the secret by setting serviceUsers in the values file:

Name:         mdb-production-psmdb-secrets
Namespace:    mongodb
Labels:       app.kubernetes.io/instance=mdb-production
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=psmdb-db
              app.kubernetes.io/version=1.20.1
              helm.sh/chart=psmdb-db-1.20.1
Annotations:  meta.helm.sh/release-name: mdb-production
              meta.helm.sh/release-namespace: mongodb

Type:  Opaque

Data
====
MONGODB_USER_ADMIN_PASSWORD:       12 bytes
PMM_SERVER_PASSWORD:               12 bytes
PMM_SERVER_USER:                   5 bytes
MONGODB_BACKUP_USER:               5 bytes
MONGODB_CLUSTER_ADMIN_PASSWORD:    12 bytes
MONGODB_CLUSTER_ADMIN_USER:        5 bytes
MONGODB_CLUSTER_MONITOR_PASSWORD:  12 bytes
MONGODB_CLUSTER_MONITOR_USER:      5 bytes
MONGODB_DATABASE_ADMIN_USER:       5 bytes
MONGODB_USER_ADMIN_USER:           5 bytes
PMM_SERVER_API_KEY:                80 bytes
MONGODB_BACKUP_PASSWORD:           12 bytes
MONGODB_DATABASE_ADMIN_PASSWORD:   12 bytes

The operator also creates other secrets:

internal-mdb-production-psmdb-users    Opaque                                20     116m
percona-server-mongodb-users           Opaque                                10     116m

And I get the error mentioned above:

Can't enable PMM... keys don’t exist in the secrets, or secrets and internal secrets are out of sync.

I deleted all these secrets before running the operator this time, so they were all created in the same run.

ETA: @Ivan_Groenewold further down this rabbit hole I see that percona-server-mongodb-users is the default user secret name defined in the operator. That secret is being created from the default cr-secrets.yaml and then copied as internal-percona-server-mongodb-users. The <your_cluster_name>-secrets doesn’t seem to do anything at all?

5

Hi, @willr ! As you noted operator created its own secrets. It means that you need to specify the secret you want to use in cr. Could you please check this doc Application and system users - Percona Operator for MongoDB ?