Installation of PSMDB fails with TLS error.
This is fresh installation and followed the below procedure:
- Deploy the operator using the following command:
$ kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.19.1/deploy/bundle.yaml
- Deploy MongoDB cluster with:
$ kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.19.1/deploy/cr-minimal.yaml
It failed with below error:
2025-03-20T12:01:39.861Z INFO createSSLByCertManager updating cert-manager certificates {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4aa5e6ed-25a6-4ff3-8daf-d11653e20c8c"}
2025-03-20T12:01:39.861Z INFO Creating old secrets {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4aa5e6ed-25a6-4ff3-8daf-d11653e20c8c"}
2025-03-20T12:01:39.881Z INFO applying new certificates {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4aa5e6ed-25a6-4ff3-8daf-d11653e20c8c"}
2025-03-20T12:01:40.943Z ERROR Reconciler error {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4aa5e6ed-25a6-4ff3-8daf-d11653e20c8c", "error": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert\". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly", "errorVerbose": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert\". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:389\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-03-20T12:01:51.438Z INFO createSSLByCertManager updating cert-manager certificates {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4180d379-d2ba-4fee-a662-6f71b94693f5"}
2025-03-20T12:01:51.438Z INFO Creating old secrets {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4180d379-d2ba-4fee-a662-6f71b94693f5"}
2025-03-20T12:01:51.449Z INFO applying new certificates {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4180d379-d2ba-4fee-a662-6f71b94693f5"}
2025-03-20T12:01:52.503Z ERROR Reconciler error {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4180d379-d2ba-4fee-a662-6f71b94693f5", "error": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert\". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly", "errorVerbose": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert\". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:389\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
===================================================
kubectl describe psmdb cluster-3 -n mongodb-test
Name: cluster-3
Namespace: mongodb-test
Labels: <none>
Annotations: <none>
API Version: psmdb.percona.com/v1
Kind: PerconaServerMongoDB
Metadata:
Creation Timestamp: 2025-03-20T12:01:22Z
Finalizers:
percona.com/delete-psmdb-pods-in-order
Generation: 1
Resource Version: 80855269
UID: 16703d55-151e-4be1-a04a-c2df58b2ef2e
Spec:
Backup:
Enabled: true
Image: percona/percona-backup-mongodb:2.8.0-multi
Pitr:
Compression Level: 6
Compression Type: gzip
Enabled: false
Oplog Only: false
Cr Version: 1.19.1
Image: percona/percona-server-mongodb:7.0.15-9-multi
Image Pull Policy: Always
Pmm:
Enabled: false
Image: percona/pmm-client:2.44.0
Server Host: monitoring-service
Replsets:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Arbiter:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Enabled: false
Resources:
Limits:
Cpu: 300m
Memory: 0.5G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 1
Expose:
Enabled: false
Type: ClusterIP
Name: rs0
Nonvoting:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Enabled: false
Pod Disruption Budget:
Max Unavailable: 1
Resources:
Limits:
Cpu: 300m
Memory: 0.5G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 3
Volume Spec:
Persistent Volume Claim:
Resources:
Requests:
Storage: 3Gi
Pod Disruption Budget:
Max Unavailable: 1
Resources:
Limits:
Cpu: 300m
Memory: 0.5G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 3
Volume Spec:
Persistent Volume Claim:
Resources:
Requests:
Storage: 3Gi
Secrets:
Encryption Key: cluster-3-mongodb-encryption-key
Users: cluster-3-secrets
Sharding:
Configsvr Repl Set:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Expose:
Enabled: false
Type: ClusterIP
Pod Disruption Budget:
Max Unavailable: 1
Resources:
Limits:
Cpu: 300m
Memory: 0.5G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 3
Volume Spec:
Persistent Volume Claim:
Resources:
Requests:
Storage: 3Gi
Enabled: true
Mongos:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Expose:
Type: ClusterIP
Pod Disruption Budget:
Max Unavailable: 1
Resources:
Limits:
Cpu: 300m
Memory: 0.5G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 3
Update Strategy: SmartUpdate
Upgrade Options:
Apply: disabled
Schedule: 0 2 * * *
Set FCV: false
Version Service Endpoint: https://check.percona.com
Status:
Conditions:
Last Transition Time: 2025-03-20T12:01:22Z
Status: True
Type: sharding
Last Transition Time: 2025-03-20T12:01:24Z
Message: TLS secrets handler: "create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly
Reason: ErrorReconcile
Status: True
Type: error
Message: Error: TLS secrets handler: "create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly
Ready: 0
Size: 0
State: error
Events: <none>
================
I can see the secret & cert is there.
kubectl get secret -n mongodb-test
NAME TYPE DATA AGE
cluster-3-ca-cert kubernetes.io/tls 3 53s
cluster-3-ca-cert-old Opaque 3 3m24s
cluster-3-secrets Opaque 10 10m
internal-cluster-3-users Opaque 20 10m