# Manually generated certificates cannot be applied in 1.16.0

**URL:** <https://forums.percona.com/t/manually-generated-certificates-cannot-be-applied-in-1-16-0/30802>\
**Category:** Percona Operator for MongoDB\
**Created:** [June 4, 2024, 10:01am UTC](https://forums.percona.com/t/manually-generated-certificates-cannot-be-applied-in-1-16-0/30802 "2024-06-04T10:01:38Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Case](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/case/32/16803_2.png) [@Case](https://forums.percona.com/u/Case)\
**Post date:** [June 4, 2024, 10:01am UTC](https://forums.percona.com/t/manually-generated-certificates-cannot-be-applied-in-1-16-0/30802/1 "2024-06-04T10:01:38Z")

</div>

Hi, testing on AWS EKS and using 1.16.0.

an issue where the operator keeps overwriting manually generated certificates.

my certificate is stored in AWS Secret Manager, and SSL & InternalSSL Secret are create by ExternalSecret.

 ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/5/2/52ce059d1acaf04d97a1413e9111632d7b6f7ae6.png)

**Secrets:**

```auto
apiVersion: v1
data:
  ca.crt: ++++++++
  tls.crt: ++++++++
  tls.key: ++++++++
immutable: false
kind: Secret
metadata:
  annotations:
    reconcile.external-secrets.io/data-hash: ec498208a4babd7a0b95e0afa4514c2b
  creationTimestamp: '2024-06-04T08:23:29Z'
  labels:
    reconcile.external-secrets.io/created-by: f03b3d87b482d2c50649c6280414ab91
  name: mongo44-main-ssl-internal
  namespace: mongodb
  ownerReferences:
    - apiVersion: external-secrets.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: ExternalSecret
      name: mongo44-main-internal-ssl
      uid: e886ca36-8f83-418b-9cf9-b7536f2a1fea
  resourceVersion: '313531644'
  uid: 43f39be8-1759-4c19-b1ba-f77d23a2abd4
type: kubernetes.io/tls
---
apiVersion: v1
data:
  ca.crt: ++++++++
  tls.crt: ++++++++
  tls.key: ++++++++
immutable: false
kind: Secret
metadata:
  annotations:
    reconcile.external-secrets.io/data-hash: b1c240872906b635dbaa1f4736fc15a9
  creationTimestamp: '2024-06-04T08:23:13Z'
  labels:
    reconcile.external-secrets.io/created-by: 07e635490539f840f7aff1572316b1d6
  name: mongo44-main-ssl
  namespace: mongodb
  ownerReferences:
    - apiVersion: external-secrets.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: ExternalSecret
      name: mongo44-main-ssl
      uid: 5b67e9d0-1e76-45d8-80aa-258720049b98
  resourceVersion: '313531265'
  uid: d1e08019-e5ec-4d9c-b1ad-5da819700802
type: kubernetes.io/tls

```

after the above two secrets are created, apply psmdb below,

**CR:**

```auto
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: mongo44-main
  labels:
    app.kubernetes.io/name: mongo44-main
  finalizers: null
spec:
  crVersion: 1.16.0
  unmanaged: false
  updateStrategy: SmartUpdate
  clusterServiceDNSMode: ServiceMesh
  
  image: "percona/percona-server-mongodb:4.4.24-23"
  imagePullPolicy: "IfNotPresent"

  ## secrets
  secrets:
    ssl: mongo44-main-ssl
    sslInternal : mongo44-main-ssl-internal
    users: psmdb-users

  ## upgrade
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: 0 2 * * *
    setFCV: false

  ## tls
  tls: 
    mode: preferTLS

  ## backup
  backup:
    enabled: false
    image: "percona/percona-backup-mongodb:2.4.1"
    storages:
      s3-ap-northeast-2:
        s3:
          bucket: database-backup-kr-alpha
          prefix: scheduled/mongo44-main
          region: ap-northeast-2
        type: s3
    pitr:
      enabled: false
    tasks:
      - compressionLevel: 6
        compressionType: gzip
        enabled: true
        keep: 365
        name: daily-s3-ap-northeast-2
        schedule: 30 19 * * *
        storageName: s3-ap-northeast-2
        type: physical

  ## replica set
  replsets:
    - name: shard01
      size: 3
      configuration: |
        net:
          tls:
            allowConnectionsWithoutCertificates: true
            allowInvalidHostnames: true
            allowInvalidCertificates: true
        replication:
          replSetName: "alpha-mongodb-test44-shard01"
        security:
          enableEncryption: false
        systemLog:
          verbosity: 0
        setParameter:
          diagnosticDataCollectionDirectoryPath: "/data/db/diagnostic-mongod.data"
          diagnosticDataCollectionPeriodMillis: 10000
          maxIndexBuildMemoryUsageMegabytes: 250
      serviceAccountName: mongodb
      affinity:
        advanced:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: mongo44-main
                      app.kubernetes.io/name: percona-server-mongodb
                      app.kubernetes.io/replset: shard01
                  topologyKey: failure-domain.beta.kubernetes.io/zone
                weight: 100
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: mongo44-main
                      app.kubernetes.io/name: percona-server-mongodb
                  topologyKey: kubernetes.io/hostname
                weight: 1
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: mongo44-main
                    app.kubernetes.io/name: percona-server-mongodb
                    app.kubernetes.io/replset: shard01
                topologyKey: kubernetes.io/hostname
      nodeSelector:
        group: mongodb
      tolerations:
        - effect: NoSchedule
          key: group
          operator: Equal
          value: mongodb
      livenessProbe:
        failureThreshold: 20
        initialDelaySeconds: 300
        periodSeconds: 120
        startupDelaySeconds: 7200
        timeoutSeconds: 10
      readinessProbe:
        failureThreshold: 10
        initialDelaySeconds: 60
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 10
      storage:
        wiredTiger:
          collectionConfig:
            blockCompressor: snappy
          engineConfig:
            cacheSizeRatio: 0.6
            directoryForIndexes: false
            journalCompressor: snappy
          indexConfig:
            prefixCompression: true
      podDisruptionBudget:
        maxUnavailable: 1
      expose:
        enabled: true
        exposeType: LoadBalancer
        serviceAnnotations: 
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-security-groups: "eks-default,alpha-mongodb"
      resources:
        limits:
          cpu: 8
          memory: 64Gi
        requests:
          cpu: 100m
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 200Gi
          storageClassName: gp3-xfs
      nonvoting:
        enabled: false
        size: 1
        affinity:
      arbiter:
        enabled: false
        size: 1
        affinity: null
    - name: shard02
      size: 3
      configuration: |
        net:
          tls:
            allowConnectionsWithoutCertificates: true
            allowInvalidHostnames: true
            allowInvalidCertificates: true
        replication:
          replSetName: "alpha-mongodb-test44-shard02"
        security:
          enableEncryption: false
        systemLog:
          verbosity: 0
        setParameter:
          diagnosticDataCollectionDirectoryPath: "/data/db/diagnostic-mongod.data"
          diagnosticDataCollectionPeriodMillis: 10000
          maxIndexBuildMemoryUsageMegabytes: 250
      serviceAccountName: mongodb
      affinity:
        advanced:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: mongo44-main
                      app.kubernetes.io/name: percona-server-mongodb
                      app.kubernetes.io/replset: shard02
                  topologyKey: failure-domain.beta.kubernetes.io/zone
                weight: 100
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: mongo44-main
                      app.kubernetes.io/name: percona-server-mongodb
                  topologyKey: kubernetes.io/hostname
                weight: 1
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: mongo44-main
                    app.kubernetes.io/name: percona-server-mongodb
                    app.kubernetes.io/replset: shard02
                topologyKey: kubernetes.io/hostname
      nodeSelector:
        group: mongodb
      tolerations:
        - effect: NoSchedule
          key: group
          operator: Equal
          value: mongodb
      livenessProbe:
        failureThreshold: 20
        initialDelaySeconds: 300
        periodSeconds: 120
        startupDelaySeconds: 7200
        timeoutSeconds: 10
      readinessProbe:
        failureThreshold: 10
        initialDelaySeconds: 60
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 10
      storage:
        wiredTiger:
          collectionConfig:
            blockCompressor: snappy
          engineConfig:
            cacheSizeRatio: 0.6
            directoryForIndexes: false
            journalCompressor: snappy
          indexConfig:
            prefixCompression: true
      podDisruptionBudget:
        maxUnavailable: 1
      expose:
        enabled: true
        exposeType: LoadBalancer
        serviceAnnotations: 
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-security-groups: "eks-default,alpha-mongodb"
      resources:
        limits:
          cpu: 8
          memory: 64Gi
        requests:
          cpu: 100m
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 200Gi
          storageClassName: gp3-xfs
      nonvoting:
        enabled: false
        size: 1
        affinity:
      arbiter:
        enabled: false
        size: 1
        affinity: null
  
  ## shard cluster
  sharding:
    enabled: true
    balancer:
      enabled: true
    configsvrReplSet:
      size: 3
      configuration: |
        net:
          tls:
            allowConnectionsWithoutCertificates: true
            allowInvalidHostnames: true
            allowInvalidCertificates: true
        security:
          enableEncryption: false
        systemLog:
          verbosity: 0
        replication:
          replSetName: "alpha-mongodb-test44-config"
        setParameter:
          diagnosticDataCollectionDirectoryPath: "/data/db/diagnostic-mongod.data"
          diagnosticDataCollectionPeriodMillis: 10000
      serviceAccountName: mongodb
      affinity:
        advanced:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: mongo44-main
                      app.kubernetes.io/name: percona-server-mongodb
                      app.kubernetes.io/replset: cfg
                  topologyKey: failure-domain.beta.kubernetes.io/zone
                weight: 100
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: mongo44-main
                      app.kubernetes.io/name: percona-server-mongodb
                  topologyKey: kubernetes.io/hostname
                weight: 1
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: mongo44-main
                    app.kubernetes.io/name: percona-server-mongodb
                    app.kubernetes.io/replset: cfg
                topologyKey: kubernetes.io/hostname
      nodeSelector:
        group: mongodb
      tolerations:
        - effect: NoSchedule
          key: group
          operator: Equal
          value: mongodb
      livenessProbe:
        failureThreshold: 20
        initialDelaySeconds: 300
        periodSeconds: 120
        startupDelaySeconds: 7200
        timeoutSeconds: 10
      readinessProbe:
        failureThreshold: 10
        initialDelaySeconds: 60
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 10
      podDisruptionBudget:
        maxUnavailable: 1
      expose:
        enabled: true
        exposeType: LoadBalancer
        serviceAnnotations: 
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-security-groups: "eks-default,alpha-mongodb"
      resources:
        limits:
          memory: 2Gi
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 50Gi
          storageClassName: gp3-xfs
    mongos:
      size: 3
      configuration: |
        net:
          tls:
            allowConnectionsWithoutCertificates: true
            allowInvalidHostnames: true
            allowInvalidCertificates: true
          compression :
            compressors : "disabled"
        systemLog:
          verbosity: 0
        setParameter:
          diagnosticDataCollectionDirectoryPath: "/data/db/diagnostic-mongos.data"
          diagnosticDataCollectionPeriodMillis: 10000
          taskExecutorPoolSize: 1
      serviceAccountName: mongodb
      affinity:
        advanced:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/component: mongos
                      app.kubernetes.io/instance: mongo44-main
                      app.kubernetes.io/name: percona-server-mongodb
                  topologyKey: failure-domain.beta.kubernetes.io/zone
                weight: 100
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/instance: mongo44-main
                      app.kubernetes.io/name: percona-server-mongodb
                  topologyKey: kubernetes.io/hostname
                weight: 1
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchLabels:
                    app.kubernetes.io/component: mongos
                    app.kubernetes.io/instance: mongo44-main
                    app.kubernetes.io/name: percona-server-mongodb
                topologyKey: kubernetes.io/hostname
      nodeSelector:
        group: mongodb
      tolerations:
        - effect: NoSchedule
          key: group
          operator: Equal
          value: mongodb
      livenessProbe:
        failureThreshold: 20
        initialDelaySeconds: 300
        periodSeconds: 120
        startupDelaySeconds: 7200
        timeoutSeconds: 10
      readinessProbe:
        failureThreshold: 10
        initialDelaySeconds: 60
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 10
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          memory: 2Gi
      expose:
        exposeType: LoadBalancer
        servicePerPod: true
        serviceAnnotations: 
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-security-groups: "eks-default,alpha-mongodb"

```

at this time, the operator creates a new certificate and overwrites the existing SSL & InternalSSL Secret. this behavior seems to only occur in 1.16.0.

```auto
2024-05-31T10:32:17.774Z	INFO	createSSLByCertManager	updating cert-manager certificates	{"controller": "psmdb-controller", "object": {"name":"mongo44-main","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongo44-main", "reconcileID": "5aa30853-9d2b-40f9-b345-4efe0c713f62"}
2024-05-31T10:32:17.774Z	INFO	Creating old secrets	{"controller": "psmdb-controller", "object": {"name":"mongo44-main","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongo44-main", "reconcileID": "5aa30853-9d2b-40f9-b345-4efe0c713f62"}
2024-05-31T10:32:17.800Z	INFO	applying new certificates	{"controller": "psmdb-controller", "object": {"name":"mongo44-main","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongo44-main", "reconcileID": "5aa30853-9d2b-40f9-b345-4efe0c713f62"}
2024-05-31T10:32:19.854Z	INFO	migrating new ca	{"controller": "psmdb-controller", "object": {"name":"mongo44-main","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongo44-main", "reconcileID": "5aa30853-9d2b-40f9-b345-4efe0c713f62"}
2024-05-31T10:32:19.889Z	INFO	new ca is already in secret, deleting old secret	{"controller": "psmdb-controller", "object": {"name":"mongo44-main","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongo44-main", "reconcileID": "5aa30853-9d2b-40f9-b345-4efe0c713f62"}
2024-05-31T10:32:19.907Z	INFO	new ca is already in secret, deleting old secret	{"controller": "psmdb-controller", "object": {"name":"mongo44-main","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongo44-main", "reconcileID": "5aa30853-9d2b-40f9-b345-4efe0c713f62"}
2024-05-31T10:32:20.001Z	INFO	StatefulSet is changed, starting smart update	{"controller": "psmdb-controller", "object": {"name":"mongo44-main","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongo44-main", "reconcileID": "5aa30853-9d2b-40f9-b345-4efe0c713f62", "name": "mongo44-main-cfg"}

```

Since the conditions below have disappeared(1.16.0), I think that this might be the cause of ignoring the existing secret and applying it as a new certificate.

> <https://github.com/percona/percona-server-mongodb-operator/commit/282394a89cd2f4874a3709848b6d8cae800cccde>
>
> \* K8SPSMDB-1014: update cert-manager certs and issuers
> 
> https://jira.percona.c…om/browse/K8SPSMDB-1014
> 
> \* fix
> 
> \* Fix upgrade from \`1.14.0\`
> 
> \* fix test
> 
> \* fix merge
> 
> \* update \`upgrade-consistency-sharded-tls\`
> 
> \* fix \`TestReconcileStatefulSet\`
> 
> \* fix test
> 
> \* don't use dry controller for deleting deprecated issuer
> 
> \* rename variables
> 
> \* fix
> 
> \* check if sharding is enabled
> 
> \* rename \`isAllStsHasLatestSSL\`
> 
> \---------
> 
> Co-authored-by: Natalia Marukovich \<nmarukovich@gmail.com\>
> Co-authored-by: Viacheslav Sarzhan \<slava.sarzhan@percona.com\>
> Co-authored-by: Tomislav Plavcic \<tomislav.plavcic@percona.com\>

```auto
# 1.15.0, pkg/controller/perconaservermongodb/ssl.go
func (r *ReconcilePerconaServerMongoDB) reconcileSSL(ctx context.Context, cr *api.PerconaServerMongoDB) error {
    ...
    if errSecret == nil && errInternalSecret == nil {
        return nil
    } else if errSecret != nil && !k8serr.IsNotFound(errSecret) {
        ...

```

The new spec.tls setting in 1.16.0 seems to always generate a certificate via cert-manager. I’ve tried changing unsafeFlags and testing in various settings, but it always overwrites existing secrets.

PS. Setting tls.allowInvalidCertificates also has the same result whether it is true or false.

---

<div class="post-metadata">

**Author:** ![qonalex](https://avatars.discourse-cdn.com/v4/letter/q/898d66/32.png) [@qonalex](https://forums.percona.com/u/qonalex)\
**Post date:** [June 5, 2024, 10:13am UTC](https://forums.percona.com/t/manually-generated-certificates-cannot-be-applied-in-1-16-0/30802/2 "2024-06-05T10:13:58Z")

</div>

Same issue here.  
After upgrade to operator 1.16.0 it has started to renew secrets automatically using cert-manager.  
How to keep old ssl secrets after upgrading to new operator?

---

<div class="post-metadata">

**Author:** ![qonalex](https://avatars.discourse-cdn.com/v4/letter/q/898d66/32.png) [@qonalex](https://forums.percona.com/u/qonalex)\
**Post date:** [June 5, 2024, 10:16am UTC](https://forums.percona.com/t/manually-generated-certificates-cannot-be-applied-in-1-16-0/30802/3 "2024-06-05T10:16:50Z")

</div>

Actually cluster went into initializing status after that…

`2024-06-05T10:15:05.205Z	ERROR	failed to reconcile cluster	{"controller": "psmdb-controller", "object": {"name":"mongodb-stage","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongodb-stage", "reconcileID": "cec2b38b-3aef-4056-b89a-a930302522ef", "replset": "rs0", "error": "dial: ping mongo: server selection error: context deadline exceeded, current topology: { Type: ReplicaSetNoPrimary, Servers: [{`

And all replicas have Type: RSGhost status.

---

<div class="post-metadata">

**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\
**Post date:** [June 5, 2024, 11:49am UTC](https://forums.percona.com/t/manually-generated-certificates-cannot-be-applied-in-1-16-0/30802/4 "2024-06-05T11:49:07Z")

</div>

Hi, we have created [[K8SPSMDB-1101] - Percona JIRA](https://perconadev.atlassian.net/browse/K8SPSMDB-1101) to track this. Currently we are investigating the issue and should have an update shortly

---

<div class="post-metadata">

**Author:** ![Case](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/case/32/16803_2.png) [@Case](https://forums.percona.com/u/Case)\
**Post date:** [June 28, 2024, 12:15am UTC](https://forums.percona.com/t/manually-generated-certificates-cannot-be-applied-in-1-16-0/30802/5 "2024-06-28T00:15:27Z")

</div>

worked well on 1.16.1. thank you!