Operator Overwrites cert-manager Managed TLS Secrets

filipp.akinfiev · June 16, 2024, 5:26pm

Description:

I am using the Percona MongoDB Operator with TLS enabled (preferTLS), and I am leveraging HashiCorp Vault via cert-manager as the PKI provider to generate certificates for MongoDB. My certificates are created and managed by cert-manager, and I have verified that the certificates work as expected. However, I am encountering an issue where the operator overwrites my cert-manager-managed secrets.

Here is an example of my Certificate resource:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: psmdb-db-ssl
  namespace: psmdb
spec:
  secretName: psmdb-db-ssl
  issuerRef:
    name: vault-issuer-mongo-server
  subject:
    organizations:
      - MYORG
    organizationalUnits:
      - cloud
  commonName: psmdb-db-rs0
  dnsNames:
    - "localhost"
    - "psmdb-db-rs0"
    - "psmdb-db-rs0.psmdb"
    - "psmdb-db-rs0.psmdb.svc.cluster.local"
    - "*.psmdb-db-rs0"
    - "*.psmdb-db-rs0.psmdb"
    - "*.psmdb-db-rs0.psmdb.svc.cluster.local"
    - "psmdb-db-mongos"
    - "psmdb-db-mongos.psmdb"
    - "psmdb-db-mongos.psmdb.svc.cluster.local"
    - "*.psmdb-db-mongos"
    - "*.psmdb-db-mongos.psmdb"
    - "*.psmdb-db-mongos.psmdb.svc.cluster.local"
    - "psmdb-db-cfg"
    - "psmdb-db-cfg.psmdb"
    - "psmdb-db-cfg.psmdb.svc.cluster.local"
    - "*.psmdb-db-cfg"
    - "*.psmdb-db-cfg.psmdb"
    - "*.psmdb-db-cfg.psmdb.svc.cluster.local"

The operator checks for the presence of the secret and determines whether it should manage the secret based on the IsSecretCreatedByUser function, which checks for a specific label (controller.cert-manager.io/fao).

github.com

percona/percona-server-mongodb-operator/blob/main/pkg/psmdb/tls/tls.go#L32


      
          	"sigs.k8s.io/controller-runtime/pkg/client"
          
          	api "github.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1"
          )
          
          var validityNotAfter = time.Date(9999, 12, 31, 23, 59, 59, 0, time.UTC)
          
          func IsSecretCreatedByUser(ctx context.Context, c client.Client, cr *api.PerconaServerMongoDB, secret *corev1.Secret) (bool, error) {
          	if metav1.IsControlledBy(secret, cr) {
          		return false, nil
          	}
          	if secret.Labels[cm.PartOfCertManagerControllerLabelKey] == "true" {
          		return isCertManagerSecretCreatedByUser(ctx, c, cr, secret)
          	}
          	return true, nil
          }
          
          func isCertManagerSecretCreatedByUser(ctx context.Context, c client.Client, cr *api.PerconaServerMongoDB, secret *corev1.Secret) (bool, error) {
          	if metav1.IsControlledBy(secret, cr) {
          		return false, nil
          	}

The cert-manager adds this label for all secrets managed by them, which causes the operator to incorrectly assume that the secret is not user-created and hence, overrides it.

Could you please provide guidance on the best way to handle this situation? Is there a recommended approach to ensure that the operator respects cert-manager-managed secrets?
As workaround I have disabled TLS.mode in the Custom Resource (CR) and configure it directly in the MongoDB configuration.

Version:

psmdb-operator: 1.16.0

Expected Result:

I would like to use cert-manager-managed certificates without the operator overriding them.

Thank you for your assistance!

Tomislav_Plavcic · June 17, 2024, 7:50am

Hi @filipp.akinfiev !

We should have a patch release 1.16.1 with the fix this week and the image is already available here: https://hub.docker.com/layers/percona/percona-server-mongodb-operator/1.16.1/images/sha256-5a9731e34cbdfc57adf7687d0b42fc00a91bcdc0d32d21012898f0bf9ccef015?context=explore

But please don’t use it in production until you see release notes published.
It would be great if you could try this image in test environment and provide feedback before the release (scheduled for Wednesday currently).

Thanks!

filipp.akinfiev · June 17, 2024, 8:45am

Hi @Tomislav_Plavcic, unfortunately, v. 1.16.1. does not work as expected.
How I tested:

scaled down operator + mongodb
delete secrets psmdb-db-ssl, psmdb-db-ssl-internal
replaced certificates with my

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: psmdb-db-ssl
  namespace: psmdb
spec:
  secretName: psmdb-db-ssl
  issuerRef:
    name: vault-issuer-mongo-server
  subject:
    organizations:
      - MYORG
    organizationalUnits:
      - cloud
  commonName: psmdb-db-rs0
  dnsNames:
    - "localhost"
    - "psmdb-db-rs0"
    - "psmdb-db-rs0.psmdb"
    - "psmdb-db-rs0.psmdb.svc.cluster.local"
    - "*.psmdb-db-rs0"
    - "*.psmdb-db-rs0.psmdb"
    - "*.psmdb-db-rs0.psmdb.svc.cluster.local"
    - "psmdb-db-mongos"
    - "psmdb-db-mongos.psmdb"
    - "psmdb-db-mongos.psmdb.svc.cluster.local"
    - "*.psmdb-db-mongos"
    - "*.psmdb-db-mongos.psmdb"
    - "*.psmdb-db-mongos.psmdb.svc.cluster.local"
    - "psmdb-db-cfg"
    - "psmdb-db-cfg.psmdb"
    - "psmdb-db-cfg.psmdb.svc.cluster.local"
    - "*.psmdb-db-cfg"
    - "*.psmdb-db-cfg.psmdb"
    - "*.psmdb-db-cfg.psmdb.svc.cluster.local"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: psmdb-db-ssl-internal
  namespace: psmdb
spec:
  secretName: psmdb-db-ssl-internal
  issuerRef:
    name: vault-issuer-mongo-server
  commonName: psmdb-db-rs0
  subject:
    organizations:
      - MYORG
    organizationalUnits:
      - cloud
  dnsNames:
    - localhost
    - psmdb-db-rs0
    - psmdb-db-rs0.psmdb
    - psmdb-db-rs0.psmdb.svc.cluster.local
    - "*.psmdb-db-rs0"
    - "*.psmdb-db-rs0.psmdb"
    - "*.psmdb-db-rs0.psmdb.svc.cluster.local"
    - psmdb-db-mongos
    - psmdb-db-mongos.psmdb
    - psmdb-db-mongos.psmdb.svc.cluster.local
    - "*.psmdb-db-mongos"
    - "*.psmdb-db-mongos.psmdb"
    - "*.psmdb-db-mongos.psmdb.svc.cluster.local"
    - psmdb-db-cfg
    - psmdb-db-cfg.psmdb
    - psmdb-db-cfg.psmdb.svc.cluster.local
    - "*.psmdb-db-cfg"
    - "*.psmdb-db-cfg.psmdb"
    - "*.psmdb-db-cfg.psmdb.svc.cluster.local"

after this, cert-manager has created new secrets psmdb-db-ssl, psmdb-db-ssl-internal as expected

apiVersion: v1
kind: Secret
metadata:
  name: psmdb-db-ssl
  namespace: psmdb
  uid: 6fc271fb-e1fa-47f6-9bcd-473fe257c9cf
  resourceVersion: '361155'
  creationTimestamp: '2024-06-17T08:38:45Z'
  labels:
    controller.cert-manager.io/fao: 'true'
  annotations:
    cert-manager.io/alt-names: >-
      localhost,psmdb-db-rs0,psmdb-db-rs0.psmdb,psmdb-db-rs0.psmdb.svc.cluster.local,*.psmdb-db-rs0,*.psmdb-db-rs0.psmdb,*.psmdb-db-rs0.psmdb.svc.cluster.local,psmdb-db-mongos,psmdb-db-mongos.psmdb,psmdb-db-mongos.psmdb.svc.cluster.local,*.psmdb-db-mongos,*.psmdb-db-mongos.psmdb,*.psmdb-db-mongos.psmdb.svc.cluster.local,psmdb-db-cfg,psmdb-db-cfg.psmdb,psmdb-db-cfg.psmdb.svc.cluster.local,*.psmdb-db-cfg,*.psmdb-db-cfg.psmdb,*.psmdb-db-cfg.psmdb.svc.cluster.local
    cert-manager.io/certificate-name: psmdb-db-ssl
    cert-manager.io/common-name: psmdb-db-rs0
    cert-manager.io/ip-sans: ''
    cert-manager.io/issuer-group: ''
    cert-manager.io/issuer-kind: ''
    cert-manager.io/issuer-name: vault-issuer-mongo-server
    cert-manager.io/subject-organizationalunits: cloud
    cert-manager.io/subject-organizations: MYORG
    cert-manager.io/uri-sans: ''
  managedFields:
...

operator is scaled up
operator overwrites Certificates and rewrites Secrets…

2024-06-17T08:41:05.612Z	INFO	setup	Manager starting up	{"gitCommit": "8790e85f0fe68be30f7ff1ce737f2322beddbc9a", "gitBranch": "release-1-16-1", "goVersion": "go1.22.4", "os": "linux", "arch": "amd64"}
2024-06-17T08:41:05.636Z	INFO	server version	{"platform": "kubernetes", "version": "v1.30.1"}
2024-06-17T08:41:05.648Z	INFO	controller-runtime.metrics	Starting metrics server
2024-06-17T08:41:05.648Z	INFO	starting server	{"name": "health probe", "addr": "[::]:8081"}
2024-06-17T08:41:05.648Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
I0617 08:41:05.648847       1 leaderelection.go:250] attempting to acquire leader lease psmdb/08db0feb.percona.com...
I0617 08:41:23.448927       1 leaderelection.go:260] successfully acquired lease psmdb/08db0feb.percona.com
2024-06-17T08:41:23.449Z	INFO	Starting EventSource	{"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2024-06-17T08:41:23.449Z	INFO	Starting Controller	{"controller": "psmdb-controller"}
2024-06-17T08:41:23.449Z	INFO	Starting EventSource	{"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2024-06-17T08:41:23.449Z	INFO	Starting EventSource	{"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2024-06-17T08:41:23.449Z	INFO	Starting EventSource	{"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2024-06-17T08:41:23.449Z	INFO	Starting EventSource	{"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2024-06-17T08:41:23.449Z	INFO	Starting Controller	{"controller": "psmdbrestore-controller"}
2024-06-17T08:41:23.449Z	INFO	Starting Controller	{"controller": "psmdbbackup-controller"}
2024-06-17T08:41:23.580Z	INFO	Starting workers	{"controller": "psmdbrestore-controller", "worker count": 1}
2024-06-17T08:41:23.590Z	INFO	Starting workers	{"controller": "psmdbbackup-controller", "worker count": 1}
2024-06-17T08:41:23.600Z	INFO	Starting workers	{"controller": "psmdb-controller", "worker count": 1}
2024-06-17T08:41:23.773Z	INFO	createSSLByCertManager	updating cert-manager certificates	{"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"psmdb"}, "namespace": "psmdb", "name": "psmdb-db", "reconcileID": "523c9a34-9f26-4345-9d39-bacffa68f45d"}
2024-06-17T08:41:23.773Z	INFO	Creating old secrets	{"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"psmdb"}, "namespace": "psmdb", "name": "psmdb-db", "reconcileID": "523c9a34-9f26-4345-9d39-bacffa68f45d"}
2024-06-17T08:41:23.824Z	INFO	applying new certificates	{"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"psmdb"}, "namespace": "psmdb", "name": "psmdb-db", "reconcileID": "523c9a34-9f26-4345-9d39-bacffa68f45d"}
2024-06-17T08:41:26.003Z	INFO	migrating new ca	{"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"psmdb"}, "namespace": "psmdb", "name": "psmdb-db", "reconcileID": "523c9a34-9f26-4345-9d39-bacffa68f45d"}

my PerconaServerMongoDB

secrets:
    ssl: psmdb-db-ssl
    sslInternal: psmdb-db-ssl-internal
...
tls:
    allowInvalidCertificates: false
    certValidityDuration: 2160h
    issuerConf:
      group: cert-manager.io
      kind: Issuer
      name: vault-issuer-mongo-server
    mode: preferTLS

Ideas?

Tomislav_Plavcic · June 17, 2024, 8:59am

@filipp.akinfiev Thanks for info and I will discuss this with the team.
Did 1.15.0 work for you or you just started with 1.16.0?

filipp.akinfiev · June 17, 2024, 9:04am

@Tomislav_Plavcic
yes, with 1.15. i have used config like

        configuration: |
          net:
            tls:
              mode: preferTLS

Tomislav_Plavcic · June 19, 2024, 11:03am

Hi @filipp.akinfiev !
We have a new image here: percona/percona-server-mongodb-operator:1.16.1 (https://hub.docker.com/layers/percona/percona-server-mongodb-operator/1.16.1/images/sha256-6326fefd8a2b20316a8a805edc18d577c23e9472046bee82e42e51fbb00efc91?context=explore)

which should fix your issue and we have delayed the release to the next week.
It would be great help if you could confirm if this image now fixes your problem.

Many thanks!

filipp.akinfiev · June 19, 2024, 2:46pm

Hi @Tomislav_Plavcic !
It looks good, I’ve tried it in dev and from scratch, in my case certificates stays as it must be -created by certmanager issued by vault PKI.
Thanks to percona team!

Topic		Replies	Views
Issues setting up TLS for MongoDB with cert-manager Percona Operator for MongoDB	3	1487	November 29, 2021
Is it possible to disable creating tls-secrets by the operator? Percona Operator for MongoDB	2	743	November 20, 2023
Manually generated certificates cannot be applied in 1.16.0 Percona Operator for MongoDB	4	223	June 28, 2024
MongoDB ReplicaSet Cluster with TLS and Letsencrypt/cert-manager Percona Server for MongoDB percona , mongodb	7	1965	May 16, 2023
Mongodb operator x.509 tls vault auth failed for $external Percona Operator for MongoDB	2	1329	January 26, 2022

Operator Overwrites cert-manager Managed TLS Secrets

Description:

Version:

Expected Result:

Related topics