# Mongo replset fails to restart if backup is switch on/off **URL:** \ **Category:** Percona Operator for MongoDB\ **Tags:** percona, mongodb\ **Created:** [September 3, 2021, 8:40am UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026 "2021-09-03T08:40:34Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![vhphan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vhphan/32/4187_2.png) [@vhphan](https://forums.percona.com/u/vhphan)\ **Post date:** [September 3, 2021, 8:40am UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026/1 "2021-09-03T08:40:34Z")
Hi, I have recently encountered this issue: - Create Mongo replset, backup disable → Works fine. - Modify the CR to enable backup → PSMDB stucks in initializing state. Error logs from the operator after modifying the CR: ```json { "level":"error", "ts":1630657894.349765, "logger":"controller-runtime.controller", "msg":"Reconciler error", "controller":"psmdb-controller", "request":"percona-mongodb/loci-dev", "error":"reconcile StatefulSet for rs0: failed to run smartUpdate: failed to check active jobs: getting pbm object: create PBM connection to 10.171.130.44:30969,10.171.130.45:32443,10.171.130.17:30392: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: 10.171.130.44:30969, Type: RSGhost, Average RTT: 677173 }, { Addr: 10.171.130.45:32443, Type: RSSecondary, Average RTT: 569239 }, { Addr: 10.171.130.17:30392, Type: RSGhost, Average RTT: 723416 },] }", "errorVerbose":"reconcile StatefulSet for rs0: failed to run smartUpdate: failed to check active jobs: getting pbm object: create PBM connection to 10.171.130.44:30969,10.171.130.45:32443,10.171.130.17:30392: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: 10.171.130.44:30969, Type: RSGhost, Average RTT: 677173 }, { Addr: 10.171.130.45:32443, Type: RSSecondary, Average RTT: 569239 }, { Addr: 10.171.130.17:30392, Type: RSGhost, Average RTT: 723416 },] }\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:365\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371", "stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88" } ``` Error logs from the backup-agent container: ```auto 2021-09-03T08:40:11.612+0000 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0 2021-09-03T08:40:11.612+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any nodes for set rs0. Please check network connectivity and the status of the set. This has happened for 2 checks in a row. ``` ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [September 6, 2021, 9:08am UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026/2 "2021-09-06T09:08:26Z")
Hello @vhphan , I cannot reproduce it. I have taken 1.9.0 Operator, deployed default cr.yaml with disabled backups. Enabled backups by changing `spec.backup.enabled` to `true`. PSMDB is ready and no errors. Is there anything else about your deployment? ---
**Author:** ![vhphan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vhphan/32/4187_2.png) [@vhphan](https://forums.percona.com/u/vhphan)\ **Post date:** [September 27, 2021, 8:31am UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026/3 "2021-09-27T08:31:17Z")
No… It’s all default. Idk if it’s again a specific problem with my k8s provider. I will update if there is anything else I can find. ---
**Author:** ![reab](https://avatars.discourse-cdn.com/v4/letter/r/35a633/32.png) [@reab](https://forums.percona.com/u/reab)\ **Post date:** [September 14, 2022, 2:33pm UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026/4 "2022-09-14T14:33:51Z")
Having same issue. Operator 1.13 Mongodb 4.4 @vhphan have found the issue? ---
**Author:** ![Tomislav\_Plavcic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/tomislav_plavcic/32/9717_2.png) [@Tomislav\_Plavcic](https://forums.percona.com/u/Tomislav_Plavcic)\ **Post date:** [September 19, 2022, 8:19am UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026/5 "2022-09-19T08:19:32Z")
Hi @reab ! For me it seems to work, but I didn’t use the ssl certificates from the “deploy” directory. I have just left the operator to generate the certificates (just didn’t apply the example ones from the directory), because the ssl certificates which are placed in deploy directory have some hardcoded altNames for namespace “psmdb” and it seems they won’t work properly in other namespaces. Could you please re-check and followup with info if that works for you or not? ---
**Author:** ![reab](https://avatars.discourse-cdn.com/v4/letter/r/35a633/32.png) [@reab](https://forums.percona.com/u/reab)\ **Post date:** [September 19, 2022, 2:24pm UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026/6 "2022-09-19T14:24:11Z")
Hi @Tomislav_Plavcic the operator generated the certificates as I am using helm to enable backup. The issue appeared after upgrading to Operator 1.13, I have the cluster running with `allowUnsafeConfigurations: false` and when I changed it to `allowUnsafeConfigurations: true` The issue disappeared. It seems it’s related to the change made to fix this [[K8SPSMDB-515] Allow setting requireTLS mode for MongoDB through the Operator - Percona JIRA](https://jira.percona.com/browse/K8SPSMDB-515) See [here](https://github.com/percona/percona-server-mongodb-operator/blob/a445d86ebeff07aee6b1485140c14a349d806708/pkg/psmdb/container.go#L161) My question now, is how to have `allowUnsafeConfigurations: false` without getting the SSL error For your reference the error is: `check for concurrent jobs: getting pbm object: create PBM connection to Cluster-Y create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: IP:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for IP because it doesn't contain any IP SANs }, { Addr: IP:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for IP because it doesn't contain any IP SANs }, { Addr: IP:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for IP because it doesn't contain any IP SANs }, { Addr: IP:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for IP because it doesn't contain any IP SANs }, { Addr: IP:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for IP because it doesn't contain any IP SANs },] }` ---
**Author:** ![Tomislav\_Plavcic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/tomislav_plavcic/32/9717_2.png) [@Tomislav\_Plavcic](https://forums.percona.com/u/Tomislav_Plavcic)\ **Post date:** [September 19, 2022, 6:52pm UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026/7 "2022-09-19T18:52:09Z")
Hi @reab ! How are you running with operator 1.13.0 when the (official) helm chart is still not released? I have tried to run 1.13.0 (not with helm) with `allowUnsafeConfigurations: false` and I don’t see your error: ```auto NAME READY STATUS RESTARTS AGE my-cluster-name-rs0-0 2/2 Running 0 6m7s percona-server-mongodb-operator-6677c8cbf7-ckjcg 1/1 Running 0 6m44s ``` Also mind you that we have some similar issue reported related to migration from/to safe configuration: [[K8SPSMDB-780] Failed to downscale/upscale cluster to unsafe configuration - Percona JIRA](https://jira.percona.com/browse/K8SPSMDB-780) Maybe it’s somehow related. ---
**Author:** ![dmn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dmn/32/6170_2.png) [@dmn](https://forums.percona.com/u/dmn)\ **Post date:** [December 9, 2022, 5:09pm UTC](https://forums.percona.com/t/mongo-replset-fails-to-restart-if-backup-is-switch-on-off/12026/8 "2022-12-09T17:09:44Z")
I am also running the 1.13 operator and have run into this. I can stand up a small cluster, I have backups disabled from the onset, (In my case I’m testing out the some other features so I dont need backups enabled) I make a change, such as to the mongod configuration, and the operator fails with the same error. It is trying to check pbm but a) I dont have backups enabled and b) it is doing so by IP, which is not going to be part of the certificate(The cert is generated by the operator, I am not providing one) ```auto 2022-12-09T17:03:04.908Z INFO controller_psmdb StatefulSet is changed, starting smart update {"name": "main-psmdb-db-cfg"} 2022-12-09T17:03:34.928Z ERROR controller.psmdb-controller Reconciler error {"name": "main-psmdb-db", "namespace": "ns-team-ads-test", "error": "reconcile StatefulSet for cfg: failed to run smartUpdate: failed to check active jobs: getting pbm object: create PBM connection to main-psmdb-db-rs0-0.main-psmdb-db-rs0.ns-team-ads-test.svc.cluster.local:27017,main-psmdb-db-rs0-2.main-psmdb-db-rs0.ns-team-ads-test.svc.cluster.local:27017,main-psmdb-db-rs0-1.main-psmdb-db-rs0.ns-team-ads-test.svc.cluster.local:27017: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: 192.168.87.140:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for 192.168.87.140 because it doesn't contain any IP SANs }, { Addr: 192.168.228.98:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for 192.168.228.98 because it doesn't contain any IP SANs }, { Addr: 192.168.204.96:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for 192.168.204.96 because it doesn't contain any IP SANs },] }", "errorVerbose": "reconcile StatefulSet for cfg: failed to run smartUpdate: failed to check active jobs: getting pbm object: create PBM connection to main-psmdb-db-rs0-0.main-psmdb-db-rs0.ns-team-ads-test.svc.cluster.local:27017,main-psmdb-db-rs0-2.main-psmdb-db-rs0.ns-team-ads-test.svc.cluster.local:27017,main-psmdb-db-rs0-1.main-psmdb-db-rs0.ns-team-ads-test.svc.cluster.local:27017: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: 192.168.87.140:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for 192.168.87.140 because it doesn't contain any IP SANs }, { Addr: 192.168.228.98:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for 192.168.228.98 because it doesn't contain any IP SANs }, { Addr: 192.168.204.96:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: cannot validate certificate for 192.168.204.96 because it doesn't contain any IP SANs },] }\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:415\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 ```