Using CertManager and TLS with MongoDB Helm charts results in error?

thedukedk · January 24, 2025, 11:15am

Description:

We already have cert-manager deployed and working in our clusters using SmallStep CA as an issuer. Works fine for us. I installed the psmdb-operator with little to no changes to its default values.

I try to install a psmdb-db with its helm chart and get an error because the operator tries to create a CA cert that is already controlled by our own CA(SmallStep).

Not sure what the proper approach is here. We only want certificates on our clusters that are signed by our own CA and we know get renewed based on our needs and wants.

Is there some way to use the certificates generated by certManager from our CA?

If we generate internal and external certifcates prior to deploying a MongoDB statefulset with the psmdb-db helm chart. Can we just tell it to use the existing certificates by passing it the appropriate secrets?

Steps to Reproduce:

I am using both the psmdb-operator and the psmdb-db Helm charts. Both version 1.19.0.

psmdb-operator

replicaCount: 3 in values file
watchAllNamespaces: true

psmdb-db

I did a deploy with backup and sharding disabled. I then tried setting the below TLS values but got the same result.

tls.mode: allowTLS
tls.issuerConf.name: step-cluster-issuer
tls.issuerConf.kind: StepClusterIssuer
tls.issuerConf.group: certmanager.step.sm

The name, kind and group values are ussually what we set when on the issuerRef whenever we want a certificate issued. So I figured I would give it a try…

Logs:

TLS secrets handler: "create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object percona/first-psmdb-psmdb-db-ca-cert is already owned by another Certificate controller first-psmdb-psmdb-db-ca-cert". Please create your TLS secret first-psmdb-psmdb-db-ssl manually or setup cert-manager correctly

Expected Result:

Valid certificates, generated by certManager with our CA as the issuer, for internal
and external communication.

MongoDB StatefulSet, Pods, Services, etc are created.

Actual Result:

Certificates get created but the PerconaServerMongoDB kind is stuck in an error state with the above error message. No PODS are ever created.

apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JS....
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tL...
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: ""
    cert-manager.io/certificate-name: first-psmdb-psmdb-db-ca-cert
    cert-manager.io/common-name: first-psmdb-psmdb-db-ca
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: first-psmdb-psmdb-db-psmdb-ca-issuer
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2025-01-24T10:37:40Z"
  labels:
    controller.cert-manager.io/fao: "true"
  name: first-psmdb-psmdb-db-ca-cert
  namespace: percona
  ownerReferences:
  - apiVersion: cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Certificate
    name: first-psmdb-psmdb-db-ca-cert
    uid: 588823e5-b309-4233-bbe9-e41770ecf7dd
  resourceVersion: "351676508"
  uid: 0ecae70c-ec25-4e3c-8643-869fc3e57606
type: kubernetes.io/tls

Additional Information:

…

thedukedk · January 24, 2025, 1:35pm

As an update I generated two certificates using our standard approach. Then passed those in with the following config for the chart.

tls:
  mode: allowTLS
  tls.allowInvalidCertificates: false
  issuerConf:
    name: step-cluster-issuer
    kind: StepClusterIssuer
    group: certmanager.step.sm
secrets:
  ssl: first-psmdb-ssl-external
  sslInternal: first-psmdb-ssl-internal
  encryptionKey: first-psmdb-psmdb-db-mongodb-encryption-key
  keyFile: first-psmdb-psmdb-db-mongodb-keyfile
  users: internal-first-psmdb-psmdb-db-users

This seems to get me farther. But I now see errors like the below in the statefulsets logs.

first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:33.145+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3557","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from clusterAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:33.242+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3561","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from userAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:34.289+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3564","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from clusterAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:34.389+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3568","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from userAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:35.447+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3571","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from clusterAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:35.537+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3574","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from userAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:36.588+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3577","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from clusterAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:36.686+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3579","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from userAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:37.788+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3584","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from clusterAdmin@ to @admin"}}}
first-psmdb-psmdb-db-rs0-0 mongod {"t":{"$date":"2025-01-24T13:34:37.840+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn3587","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from userAdmin@ to @admin"}}}

Julio_Pasinatto · January 28, 2025, 11:49pm

Hi @thedukedk thanks for reporting this issue, I’ll try to reproduce it.

In the meantime, you might find these TLS setup docs helpful, if you haven’t already checked them: Transport encryption (TLS/SSL) - Percona Operator for MongoDB

Also, I noticed in your message there might be a typo in the configuration. Please make sure you’re using allowInvalidCertificates and not tls.allowInvalidCertificates.

I’ll get back to you once I have more information.

thedukedk · January 29, 2025, 1:05pm

Hi @Julio_Pasinatto

Many thanks. Been struggling quite a bit to get this running with TLS enabled.

We really need internal and external TLS enabled using our CA. We do not want the data encrypted at rest.

Generating the certs before deploying the psmdb-db with the helm chart seems to work, when tls.mode is either perferTLS or allowTLS, with the below configuration.

tls:
  mode: allowTLS # OR perferTLS
  allowInvalidCertificates: false
  issuerConf:
    name: step-cluster-issuer
    kind: StepClusterIssuer
    group: certmanager.step.sm
secrets:
  ssl: first-psmdb-ssl-external
  sslInternal: first-psmdb-ssl-internal
  encryptionKey: first-psmdb-psmdb-db-mongodb-encryption-key
  keyFile: first-psmdb-psmdb-db-mongodb-keyfile
  users: internal-first-psmdb-psmdb-db-users

But I see these errors in the logs.

mongod {"t":{"$date":"2025-01-29T11:46:24.955+00:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn427","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from clusterAdmin@ to @admin"}}}

Setting the tls.mode to requireTLS does not work at all unless i set allowInvalidCertificates to true.

zkatarn · February 19, 2025, 3:14pm

Hi

Same trouble here.

We use Percona Operator in 1.16.0 without trouble. This week, i try to upgrade in 1.19.0, and it doesn’t work. I have a conflict between Cert-manager et Percona Operator.

Architecture

We use Google Kubernetes Engine 1.30.9-gke.1009000 with Cert-manager v1.16.2

TLS Configuration

tls:
  mode: preferTLS
  allowInvalidCertificates: false

Logs from Operator

2025-02-19T15:00:07.905Z	INFO	createSSLByCertManager	updating cert-manager certificates	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mongodb-testvg","namespace":"percona-test"}, "namespace": "percona-test", "name": "mongodb-testvg", "reconcileID": "7dac656b-c75c-4107-ac45-4ea910b180c5"}
2025-02-19T15:00:07.905Z	INFO	Creating old secrets	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mongodb-testvg","namespace":"percona-test"}, "namespace": "percona-test", "name": "mongodb-testvg", "reconcileID": "7dac656b-c75c-4107-ac45-4ea910b180c5"}
2025-02-19T15:00:07.918Z	INFO	applying new certificates	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mongodb-testvg","namespace":"percona-test"}, "namespace": "percona-test", "name": "mongodb-testvg", "reconcileID": "7dac656b-c75c-4107-ac45-4ea910b180c5"}
2025-02-19T15:00:08.974Z	ERROR	Reconciler error	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mongodb-testvg","namespace":"percona-test"}, "namespace": "percona-test", "name": "mongodb-testvg", "reconcileID": "7dac656b-c75c-4107-ac45-4ea910b180c5", "error": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object percona-test/mongodb-testvg-ca-cert is already owned by another Certificate controller mongodb-testvg-ca-cert\". Please create your TLS secret mongodb-testvg-ssl manually or setup cert-manager correctly", "errorVerbose": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object percona-test/mongodb-testvg-ca-cert is already owned by another Certificate controller mongodb-testvg-ca-cert\". Please create your TLS secret mongodb-testvg-ssl manually or setup cert-manager correctly\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:389\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224

Logs from Cert-manager

I0219 14:21:20.256396       1 conditions.go:96] Setting lastTransitionTime for Issuer "mongodb-testvg-psmdb-ca-issuer" condition "Ready" to 2025-02-19 14:21:20.25637953 +0000 UTC m=+175707.548230613
I0219 14:21:20.279227       1 conditions.go:203] Setting lastTransitionTime for Certificate "mongodb-testvg-ca-cert" condition "Ready" to 2025-02-19 14:21:20.279216789 +0000 UTC m=+175707.571067862
I0219 14:21:20.279242       1 trigger_controller.go:223] "Certificate must be re-issued" logger="cert-manager.controller" key="percona-test/mongodb-testvg-ca-cert" reason="DoesNotExist" message="Issuing certificate as Secret does not exist"
I0219 14:21:20.279408       1 conditions.go:203] Setting lastTransitionTime for Certificate "mongodb-testvg-ca-cert" condition "Issuing" to 2025-02-19 14:21:20.279401129 +0000 UTC m=+175707.571252212
I0219 14:21:20.300669       1 controller.go:152] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" error="Operation cannot be fulfilled on certificates.cert-manager.io \"mongodb-testvg-ca-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0219 14:21:20.300759       1 conditions.go:203] Setting lastTransitionTime for Certificate "mongodb-testvg-ca-cert" condition "Ready" to 2025-02-19 14:21:20.300752259 +0000 UTC m=+175707.592603332
I0219 14:21:20.606403       1 controller.go:152] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" error="Operation cannot be fulfilled on certificates.cert-manager.io \"mongodb-testvg-ca-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0219 14:21:20.664148       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "mongodb-testvg-ca-cert-1" condition "Approved" to 2025-02-19 14:21:20.664138023 +0000 UTC m=+175707.955989097
I0219 14:21:20.704195       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "mongodb-testvg-ca-cert-1" condition "Ready" to 2025-02-19 14:21:20.704183933 +0000 UTC m=+175707.996035006
I0219 14:21:20.762196       1 conditions.go:192] Found status change for Certificate "mongodb-testvg-ca-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2025-02-19 14:21:20.762186582 +0000 UTC m=+175708.054037655
I0219 14:21:20.785810       1 controller.go:152] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" error="Operation cannot be fulfilled on certificates.cert-manager.io \"mongodb-testvg-ca-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0219 14:21:20.787056       1 conditions.go:192] Found status change for Certificate "mongodb-testvg-ca-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2025-02-19 14:21:20.787048361 +0000 UTC m=+175708.078899424
I0219 14:21:20.816625       1 controller.go:152] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" error="Operation cannot be fulfilled on certificates.cert-manager.io \"mongodb-testvg-ca-cert\": the object has been modified; please apply your changes to the latest version and try again"

It’s work until 1.16.1, after that i have always this error.

Thanks for your help

Topic		Replies	Views
Basic cluster with TLS not working, operator k8s/helm setup Percona Server for MongoDB mongodb , psmdb-operator	18	3054	December 18, 2022
Operator Overwrites cert-manager Managed TLS Secrets Percona Operator for MongoDB mongodb	6	230	June 19, 2024
MongoDB ReplicaSet Cluster with TLS and Letsencrypt/cert-manager Percona Server for MongoDB percona , mongodb	7	2131	May 16, 2023
Issues setting up TLS for MongoDB with cert-manager Percona Operator for MongoDB	3	1547	November 29, 2021
Mongodb cipher lock down kubernetes helm chart Awards & Recognition mongodb	2	48	November 7, 2024