# Psmdb-operator crashes when psmdb-db is deployed

**URL:** <https://forums.percona.com/t/psmdb-operator-crashes-when-psmdb-db-is-deployed/31111>\
**Category:** Percona Operator for MongoDB\
**Created:** [June 19, 2024, 6:35pm UTC](https://forums.percona.com/t/psmdb-operator-crashes-when-psmdb-db-is-deployed/31111 "2024-06-19T18:35:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nickmman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nickmman/32/16962_2.png) [@nickmman](https://forums.percona.com/u/nickmman)\
**Post date:** [June 19, 2024, 6:35pm UTC](https://forums.percona.com/t/psmdb-operator-crashes-when-psmdb-db-is-deployed/31111/1 "2024-06-19T18:35:13Z")

</div>

## Description:

I’m using both the psmdb-operator and psmdb-db helm charts. I have deployed the operator (without the db deployment) and it was working fine, without errors/crashes.

However, now that I have deployed the db, the operator enters a crashloop. When the operator starts crashlooping, it causes a complete restart of all the pods from psmdb-db as well.

Additionally, from what I can see, the mongos pods are not being deployed, just cfg and rs pods.

## Steps to Reproduce:

1. Deploy psmdb-operator with helm (version 1.16.1) using the following values:

```auto
    logLevel: "DEBUG"
    logStructured: true

```

1. Deploy psmdb-db with helm (version 1.16.1) using the following values:

```auto
    backup:
      enabled: false
      pitr:
        enabled: false
      storages:
        gcp:
          type: s3
          s3:
            credentialsSecret: gcp-backup-credentials
            bucket: redacted
            region: us
            prefix: redacted/mongodb
            endpointUrl: https://storage.googleapis.com
      tasks:
      - name: daily-gcp-us
        enabled: true
        schedule: "0 0 * * *"
        keep: 3
        storageName: gcp
        compressionType: gzip
    pmm:
      enabled: false
    replsets:
      rs0:
        volumeSpec:
          pvc:
            storageClassName: ceph-block
            resources:
              requests:
                storage: 10Gi
    secrets:
      users: percona-mongodb-credentials
    sharding:
      configrs:
        volumeSpec:
          pvc:
            storageClassName: ceph-block
            resources:
              requests:
                storage: 10Gi
      mongos:
        terminationGracePeriodSeconds: 40
    tls:
      issuerConf:
        name: trust-manager
        kind: ClusterIssuer

```

1. Observe the psmdb-db cluster be created, the operator starts crashing and the cluster itself will follow.

## Version:

psmdb-operator v1.16.1  
psmdb-db v1.16.1  
Kubernetes v1.28.7

## Logs:

```auto
2024-06-12T21:42:34.549Z INFO setup Manager starting up {"gitCommit": "54e1b18dd9dac8e0ed5929bb2c91318cd6829a48", "gitBranch": "release-1-16-0", "goVersion": "go1.22.3", "os": "linux", "arch": "amd64"}
2024-06-12T21:42:34.565Z INFO server version {"platform": "kubernetes", "version": "v1.28.7+k3s1"}
2024-06-12T21:42:34.570Z INFO controller-runtime.metrics Starting metrics server
2024-06-12T21:42:34.570Z INFO starting server {"name": "health probe", "addr": "[::]:8081"}
2024-06-12T21:42:34.570Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
I0612 21:42:34.570960 1 leaderelection.go:250] attempting to acquire leader lease mongodb/08db0feb.percona.com...
I0612 21:42:53.320941 1 leaderelection.go:260] successfully acquired lease mongodb/08db0feb.percona.com
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdb-controller"}
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdbrestore-controller"}
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdbbackup-controller"}
2024-06-12T21:42:53.444Z INFO Starting workers {"controller": "psmdbbackup-controller", "worker count": 1}
2024-06-12T21:42:53.445Z INFO Starting workers {"controller": "psmdb-controller", "worker count": 1}
2024-06-12T21:42:53.445Z INFO Starting workers {"controller": "psmdbrestore-controller", "worker count": 1}
E0612 21:42:53.685207 1 runtime.go:79] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 313 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1f11320, 0x298b1f0})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:49 +0x6b
panic({0x1f11320?, 0x298b1f0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
k8s.io/client-go/util/retry.OnError.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:51 +0x30
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/wait.go:145 +0x3e
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000baaa18)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/backoff.go:461 +0x5a
k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
        /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:50 +0xa5
k8s.io/client-go/util/retry.RetryOnConflict(...)
        /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:104
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:368 +0x16d0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 141
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:218 +0x486
2024-06-12T21:42:53.730Z INFO Observed a panic in reconciler: assignment to entry in nil map {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "7676acba-b62f-4d00-a4dc-51c0e17bc27c"}
panic: assignment to entry in nil map [recovered]
        panic: assignment to entry in nil map [recovered]
        panic: assignment to entry in nil map

goroutine 313 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:111 +0x1e5
panic({0x1f11320?, 0x298b1f0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:56 +0xcd
panic({0x1f11320?, 0x298b1f0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
k8s.io/client-go/util/retry.OnError.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:51 +0x30
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/wait.go:145 +0x3e
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000efea18)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/backoff.go:461 +0x5a
k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
        /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:50 +0xa5
k8s.io/client-go/util/retry.RetryOnConflict(...)
        /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:104
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:368 +0x16d0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 141
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:218 +0x486

```

## Expected Result:

Mongos pods are deployed, operator doesn’t crash.

## Actual Result:

No mongos pods are deployed, the operator starts crashlooping and the psmdb-db cluster restarts as well.

## Additional Information:

Deployed using ArgoCD but still using helm.  
Github issues opened:

> <https://github.com/percona/percona-helm-charts/issues/338>
>
> I'm using both the psmdb-operator and psmdb-db helm charts. I have deployed the …operator (without the db deployment) and it was working fine, without errors/crashes.
> 
> However, now that I have deployed the db, the operator enters a crashloop. When the operator starts crashlooping, it causes a complete restart of all the pods from psmdb-db as well.
> 
> Logs from operator:
> \`\`\`
> 2024-06-12T21:42:34.549Z INFO setup Manager starting up {"gitCommit": "54e1b18dd9dac8e0ed5929bb2c91318cd6829a48", "gitBranch": "release-1-16-0", "goVersion": "go1.22.3", "os": "linux", "arch": "amd64"}
> 2024-06-12T21:42:34.565Z INFO server version {"platform": "kubernetes", "version": "v1.28.7+k3s1"}
> 2024-06-12T21:42:34.570Z INFO controller-runtime.metrics Starting metrics server
> 2024-06-12T21:42:34.570Z INFO starting server {"name": "health probe", "addr": "\[::\]:8081"}
> 2024-06-12T21:42:34.570Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
> I0612 21:42:34.570960 1 leaderelection.go:250\] attempting to acquire leader lease mongodb/08db0feb.percona.com...
> I0612 21:42:53.320941 1 leaderelection.go:260\] successfully acquired lease mongodb/08db0feb.percona.com
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdb-controller", "source": "kind source: \*v1.PerconaServerMongoDB"}
> 2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdb-controller"}
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: \*v1.PerconaServerMongoDBRestore"}
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: \*v1.PerconaServerMongoDBBackup"}
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: \*v1.Pod"}
> 2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdbrestore-controller"}
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: \*v1.Pod"}
> 2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdbbackup-controller"}
> 2024-06-12T21:42:53.444Z INFO Starting workers {"controller": "psmdbbackup-controller", "worker count": 1}
> 2024-06-12T21:42:53.445Z INFO Starting workers {"controller": "psmdb-controller", "worker count": 1}
> 2024-06-12T21:42:53.445Z INFO Starting workers {"controller": "psmdbrestore-controller", "worker count": 1}
> E0612 21:42:53.685207 1 runtime.go:79\] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
> goroutine 313 \[running\]:
> k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1f11320, 0x298b1f0})
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:75 +0x85
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:49 +0x6b
> panic({0x1f11320?, 0x298b1f0?})
> /usr/local/go/src/runtime/panic.go:770 +0x132
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
> k8s.io/client-go/util/retry.OnError.func1()
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:51 +0x30
> k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/wait.go:145 +0x3e
> k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000baaa18)
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/backoff.go:461 +0x5a
> k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:50 +0xa5
> k8s.io/client-go/util/retry.RetryOnConflict(...)
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:104
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb\_controller.go:368 +0x16d0
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114 +0xb7
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311 +0x3bc
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261 +0x1be
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2()
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222 +0x79
> created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2 in goroutine 141
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:218 +0x486
> 2024-06-12T21:42:53.730Z INFO Observed a panic in reconciler: assignment to entry in nil map {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "7676acba-b62f-4d00-a4dc-51c0e17bc27c"}
> panic: assignment to entry in nil map \[recovered\]
> panic: assignment to entry in nil map \[recovered\]
> panic: assignment to entry in nil map
> 
> goroutine 313 \[running\]:
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile.func1()
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:111 +0x1e5
> panic({0x1f11320?, 0x298b1f0?})
> /usr/local/go/src/runtime/panic.go:770 +0x132
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:56 +0xcd
> panic({0x1f11320?, 0x298b1f0?})
> /usr/local/go/src/runtime/panic.go:770 +0x132
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
> k8s.io/client-go/util/retry.OnError.func1()
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:51 +0x30
> k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/wait.go:145 +0x3e
> k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000efea18)
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/backoff.go:461 +0x5a
> k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:50 +0xa5
> k8s.io/client-go/util/retry.RetryOnConflict(...)
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:104
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb\_controller.go:368 +0x16d0
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114 +0xb7
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311 +0x3bc
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261 +0x1be
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2()
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222 +0x79
> created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2 in goroutine 141
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:218 +0x486
> \`\`\`
> 
> I'm using version 1.16.1 in both charts.
> My values files are as follows:
> 
> \`\`\`
> perconaMongodb:
> enabled: true
> version: 1.16.1
> values:
> backup:
> enabled: false
> pitr:
> enabled: true
> storages:
> gcp:
> type: s3
> s3:
> credentialsSecret: gcp-backup-credentials
> bucket: redacted
> region: us
> prefix: dev-onprem/mongodb
> endpointUrl: https://storage.googleapis.com
> tasks:
> - name: daily-gcp-us
> enabled: true
> schedule: "0 0 \* \* \*"
> keep: 3
> storageName: gcp
> compressionType: gzip
> pmm:
> enabled: true
> replsets:
> rs0:
> volumeSpec:
> pvc:
> storageClassName: ceph-block
> resources:
> requests:
> storage: 10Gi
> rs1:
> resources:
> limits:
> cpu: "300m"
> memory: "0.5G"
> requests:
> cpu: "300m"
> memory: "0.5G"
> size: 3
> volumeSpec:
> pvc:
> storageClassName: ceph-block
> resources:
> requests:
> storage: 10Gi
> secrets:
> users: percona-mongodb-credentials
> sharding:
> configrs:
> volumeSpec:
> pvc:
> storageClassName: ceph-block
> resources:
> requests:
> storage: 10Gi
> tls:
> issuerConf:
> name: redacted
> kind: ClusterIssuer
> perconaMongodbOperator:
> enabled: true
> version: 1.16.1
> values:
> watchNamespace: "mongodb"
> \`\`\`
> Everything else is using the default values.

> <https://github.com/percona/percona-server-mongodb-operator/issues/1571>
>
> \### Report
> 
> I'm using both the \[psmdb-operator\](https://github.com/percona/per…cona-helm-charts/tree/main/charts/psmdb-operator) and \[psmdb-db\](https://github.com/percona/percona-helm-charts/tree/main/charts/psmdb-db) helm charts. I have deployed the operator (without the db deployment) and it was working fine, without errors/crashes.
> 
> However, now that I have deployed the db, the operator enters a crashloop. When the operator starts crashlooping, it causes a complete restart of all the pods from psmdb-db as well.
> \`\`\`
> 2024-06-12T21:42:34.549Z INFO setup Manager starting up {"gitCommit": "54e1b18dd9dac8e0ed5929bb2c91318cd6829a48", "gitBranch": "release-1-16-0", "goVersion": "go1.22.3", "os": "linux", "arch": "amd64"}
> 2024-06-12T21:42:34.565Z INFO server version {"platform": "kubernetes", "version": "v1.28.7+k3s1"}
> 2024-06-12T21:42:34.570Z INFO controller-runtime.metrics Starting metrics server
> 2024-06-12T21:42:34.570Z INFO starting server {"name": "health probe", "addr": "\[::\]:8081"}
> 2024-06-12T21:42:34.570Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
> I0612 21:42:34.570960 1 leaderelection.go:250\] attempting to acquire leader lease mongodb/08db0feb.percona.com...
> I0612 21:42:53.320941 1 leaderelection.go:260\] successfully acquired lease mongodb/08db0feb.percona.com
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdb-controller", "source": "kind source: \*v1.PerconaServerMongoDB"}
> 2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdb-controller"}
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: \*v1.PerconaServerMongoDBRestore"}
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: \*v1.PerconaServerMongoDBBackup"}
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: \*v1.Pod"}
> 2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdbrestore-controller"}
> 2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: \*v1.Pod"}
> 2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdbbackup-controller"}
> 2024-06-12T21:42:53.444Z INFO Starting workers {"controller": "psmdbbackup-controller", "worker count": 1}
> 2024-06-12T21:42:53.445Z INFO Starting workers {"controller": "psmdb-controller", "worker count": 1}
> 2024-06-12T21:42:53.445Z INFO Starting workers {"controller": "psmdbrestore-controller", "worker count": 1}
> E0612 21:42:53.685207 1 runtime.go:79\] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
> goroutine 313 \[running\]:
> k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1f11320, 0x298b1f0})
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:75 +0x85
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:49 +0x6b
> panic({0x1f11320?, 0x298b1f0?})
> /usr/local/go/src/runtime/panic.go:770 +0x132
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
> k8s.io/client-go/util/retry.OnError.func1()
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:51 +0x30
> k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/wait.go:145 +0x3e
> k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000baaa18)
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/backoff.go:461 +0x5a
> k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:50 +0xa5
> k8s.io/client-go/util/retry.RetryOnConflict(...)
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:104
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb\_controller.go:368 +0x16d0
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114 +0xb7
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311 +0x3bc
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261 +0x1be
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2()
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222 +0x79
> created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2 in goroutine 141
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:218 +0x486
> 2024-06-12T21:42:53.730Z INFO Observed a panic in reconciler: assignment to entry in nil map {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "7676acba-b62f-4d00-a4dc-51c0e17bc27c"}
> panic: assignment to entry in nil map \[recovered\]
> panic: assignment to entry in nil map \[recovered\]
> panic: assignment to entry in nil map
> 
> goroutine 313 \[running\]:
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile.func1()
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:111 +0x1e5
> panic({0x1f11320?, 0x298b1f0?})
> /usr/local/go/src/runtime/panic.go:770 +0x132
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/runtime/runtime.go:56 +0xcd
> panic({0x1f11320?, 0x298b1f0?})
> /usr/local/go/src/runtime/panic.go:770 +0x132
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
> k8s.io/client-go/util/retry.OnError.func1()
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:51 +0x30
> k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/wait.go:145 +0x3e
> k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000efea18)
> /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/backoff.go:461 +0x5a
> k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:50 +0xa5
> k8s.io/client-go/util/retry.RetryOnConflict(...)
> /go/pkg/mod/k8s.io/client-go@v0.30.0/util/retry/util.go:104
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
> github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
> /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb\_controller.go:368 +0x16d0
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114 +0xb7
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311 +0x3bc
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261 +0x1be
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2()
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222 +0x79
> created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2 in goroutine 141
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:218 +0x486
> \`\`\`
> 
> \### More about the problem
> 
> My values files are as follows:
> 
> \`\`\`
> perconaMongodb:
> enabled: true
> version: 1.16.1
> values:
> backup:
> enabled: false
> pitr:
> enabled: true
> storages:
> gcp:
> type: s3
> s3:
> credentialsSecret: gcp-backup-credentials
> bucket: redacted
> region: us
> prefix: dev-onprem/mongodb
> endpointUrl: https://storage.googleapis.com
> tasks:
> - name: daily-gcp-us
> enabled: true
> schedule: "0 0 \* \* \*"
> keep: 3
> storageName: gcp
> compressionType: gzip
> pmm:
> enabled: true
> replsets:
> rs0:
> volumeSpec:
> pvc:
> storageClassName: ceph-block
> resources:
> requests:
> storage: 10Gi
> rs1:
> resources:
> limits:
> cpu: "300m"
> memory: "0.5G"
> requests:
> cpu: "300m"
> memory: "0.5G"
> size: 3
> volumeSpec:
> pvc:
> storageClassName: ceph-block
> resources:
> requests:
> storage: 10Gi
> secrets:
> users: percona-mongodb-credentials
> sharding:
> configrs:
> volumeSpec:
> pvc:
> storageClassName: ceph-block
> resources:
> requests:
> storage: 10Gi
> tls:
> issuerConf:
> name: redacted
> kind: ClusterIssuer
> perconaMongodbOperator:
> enabled: true
> version: 1.16.1
> values:
> watchNamespace: "mongodb"
> \`\`\`
> Everything else is using the default values.
> 
> \### Steps to reproduce
> 
> 1. Deploy psmdb-operator via helm
> 2. Deploy psmdb-db via helm
> 3. Wait until deployment finishes, operator will start crashlooping shortly
> 
> 
> \### Versions
> 
> 1. Kubernetes: v1.28.7
> 2. Operator: 1.16.0
> 3. Database: 7.0.8-5
> 
> 
> \### Anything else?
> 
> Deployment is handled via ArgoCD.

---

<div class="post-metadata">

**Author:** ![Tomislav\_Plavcic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/tomislav_plavcic/32/9717_2.png) [@Tomislav\_Plavcic](https://forums.percona.com/u/Tomislav_Plavcic)\
**Post date:** [June 24, 2024, 4:02pm UTC](https://forums.percona.com/t/psmdb-operator-crashes-when-psmdb-db-is-deployed/31111/2 "2024-06-24T16:02:05Z")

</div>

Hi @nickmman !

Can you just for the test try to set following annotation in the values for the psmdb-db chart:

```auto
annotations:
  kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"psmdb.percona.com/v1","kind":"PerconaServerMongoDB"}

```

---

<div class="post-metadata">

**Author:** ![nickmman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nickmman/32/16962_2.png) [@nickmman](https://forums.percona.com/u/nickmman)\
**Post date:** [June 24, 2024, 8:39pm UTC](https://forums.percona.com/t/psmdb-operator-crashes-when-psmdb-db-is-deployed/31111/3 "2024-06-24T20:39:00Z")

</div>

Hi @Tomislav_Plavcic,

Thanks for your reply!

That appears to have worked, the only problem now is that the operator has the following log output every 5 seconds:  
`{"level":"info","ts":1719261452.0629106,"msg":"StatefulSet is not up to date","controller":"psmdb-controller","object":{"name":"psmdb-db","namespace":"mongodb"},"namespace":"mongodb","name":"psmdb-db","reconcileID":"cd6c8d1d-0264-40b9-a63a-fbfb054e551c","sts":"psmdb-db-cfg"}`

---

<div class="post-metadata">

**Author:** ![nickmman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nickmman/32/16962_2.png) [@nickmman](https://forums.percona.com/u/nickmman)\
**Post date:** [June 24, 2024, 9:42pm UTC](https://forums.percona.com/t/psmdb-operator-crashes-when-psmdb-db-is-deployed/31111/4 "2024-06-24T21:42:53Z")

</div>

Another thing that I’m seeing is that all of the pods in psmdb-db are being terminated and created anew, without any interaction from my part.  
These are the logs from the operator:

```auto
2024-06-24T21:38:03.938Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "2c0bae00-a854-4d85-99e9-131e27ca22ff", "error": "reconcile statefulsets: reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret", "errorVerbose": "reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:464\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
2024-06-24T21:38:04.490Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "6ca89c0e-eec9-4fe2-9006-e322b6f98dd8", "error": "reconcile statefulsets: reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret", "errorVerbose": "reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:464\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
2024-06-24T21:38:05.037Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "1338928e-9d19-41ec-9652-42c596f959fd", "error": "reconcile statefulsets: reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret", "errorVerbose": "reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:464\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
2024-06-24T21:38:05.637Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "351aa100-42d5-4c55-b671-a440ead08f90", "error": "reconcile statefulsets: reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret", "errorVerbose": "reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:464\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
2024-06-24T21:38:06.186Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "facf4721-bbea-44d5-82ed-2560108122f7", "error": "reconcile statefulsets: reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret", "errorVerbose": "reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:464\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
2024-06-24T21:38:06.764Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "a5f6f01c-e866-43f2-8391-928dbd5ec1f3", "error": "reconcile statefulsets: reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret", "errorVerbose": "reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:464\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
2024-06-24T21:38:07.289Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "1c7c3c05-a47c-4020-9be2-c4c70ab91563", "error": "reconcile statefulsets: reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret", "errorVerbose": "reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:464\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
2024-06-24T21:38:10.868Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "3e7e9577-f3bb-4023-b96c-4245af1dbd93", "sts": "psmdb-db-cfg"}
2024-06-24T21:38:11.041Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "3e7e9577-f3bb-4023-b96c-4245af1dbd93", "sts": "psmdb-db-cfg"}
2024-06-24T21:38:11.042Z INFO add new job {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "3e7e9577-f3bb-4023-b96c-4245af1dbd93", "name": "ensure-version/mongodb/psmdb-db", "schedule": "0 2 * * *"}
2024-06-24T21:38:11.065Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "3e7e9577-f3bb-4023-b96c-4245af1dbd93", "previous": "error", "current": "initializing"}
2024-06-24T21:38:11.292Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1", "sts": "psmdb-db-cfg"}
2024-06-24T21:38:13.517Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1", "sts": "psmdb-db-cfg"}
2024-06-24T21:38:13.677Z INFO StatefulSet is changed, starting smart update {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1", "name": "psmdb-db-mongos"}
2024-06-24T21:38:13.679Z DEBUG Checking for active jobs {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1", "currentJob": {"Name":"","Type":0}}
2024-06-24T21:38:46.238Z INFO Pod started {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1", "pod": "psmdb-db-mongos-2"}
2024-06-24T21:39:16.645Z INFO Pod started {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1", "pod": "psmdb-db-mongos-1"}
2024-06-24T21:39:48.047Z INFO Pod started {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1", "pod": "psmdb-db-mongos-0"}
2024-06-24T21:39:48.064Z INFO smart update finished for mongos statefulset {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1"}
2024-06-24T21:39:48.071Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "78f1ee3b-b328-4959-bda3-49b4d9426bd1", "sts": "psmdb-db-cfg"}
2024-06-24T21:39:48.293Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "sts": "psmdb-db-cfg"}
2024-06-24T21:39:48.331Z INFO StatefulSet is changed, starting smart update {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "name": "psmdb-db-cfg"}
2024-06-24T21:39:48.333Z DEBUG Checking for active jobs {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "currentJob": {"Name":"","Type":0}}
2024-06-24T21:39:48.458Z INFO apply changes to secondary pod {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "pod": "psmdb-db-cfg-2"}
2024-06-24T21:40:23.015Z INFO Pod started {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "pod": "psmdb-db-cfg-2"}
2024-06-24T21:40:23.044Z INFO apply changes to secondary pod {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "pod": "psmdb-db-cfg-1"}
2024-06-24T21:40:52.367Z INFO Pod started {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "pod": "psmdb-db-cfg-1"}
2024-06-24T21:40:52.392Z INFO doing step down... {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "force": false}
2024-06-24T21:40:52.423Z INFO apply changes to primary pod {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "e7aaa0fd-999e-4629-9898-908015916412", "pod": "psmdb-db-cfg-0"}

```