# MongoDB Cluster cannot failover when down time all pods and using mode External (NodePort and LB) **URL:** https://forums.percona.com/t/mongodb-cluster-cannot-failover-when-down-time-all-pods-and-using-mode-external-nodeport-and-lb/29842 **Category:** Percona Operator for MongoDB **Tags:** psmdb-operator **Created:** [April 25, 2024, 8:57am UTC](https://forums.percona.com/t/mongodb-cluster-cannot-failover-when-down-time-all-pods-and-using-mode-external-nodeport-and-lb/29842 "2024-04-25T08:57:31Z") **Posts on this page:** 4 **Page:** 1
### Author: ![Trinh.Duc.Chung](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/trinh.duc.chung/32/11146_2.png) [@Trinh.Duc.Chung](https://forums.percona.com/u/Trinh.Duc.Chung) #### Post date: [April 25, 2024, 8:57am UTC](https://forums.percona.com/t/mongodb-cluster-cannot-failover-when-down-time-all-pods-and-using-mode-external-nodeport-and-lb/29842/1 "2024-04-25T08:57:31Z")
## Description: MongoDB Cluster cannot failover when down time all pods ## Steps to Reproduce: `Kubectl delete (all pods in replicaset) --force -n namespace` or shutdown all nodes in K8s cluster ## Version: Percona Operator for MongoDB\* 1.15.0 ## Logs: Logs in Operator: 2024-04-25T08:48:42.241Z ERROR failed to reconcile cluster {“controller”: “psmdb-controller”, “object”: {“name”:“mongo-psmdb-db”,“namespace”:“tungdt”}, “namespace”: “tungdt”, “name”: “mongo-psmdb-db”, “reconcileID”: “7614bf82-1bf1-43e0-b4ab-f07b6c5a358c”, “replset”: “rs0”, “error”: “dial: ping mongo: server selection error: context deadline exceeded, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mongo-psmdb-db-rs0-0.mongo-psmdb-db-rs0.tungdt.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup mongo-psmdb-db-rs0-0.mongo-psmdb-db-rs0.tungdt.svc.cluster.local on 10.43.0.10:53: no such host }, { Addr: mongo-psmdb-db-rs0-1.mongo-psmdb-db-rs0.tungdt.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup mongo-psmdb-db-rs0-1.mongo-psmdb-db-rs0.tungdt.svc.cluster.local on 10.43.0.10:53: no such host }, { Addr: mongo-psmdb-db-rs0-2.mongo-psmdb-db-rs0.tungdt.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup mongo-psmdb-db-rs0-2.mongo-psmdb-db-rs0.tungdt.svc.cluster.local on 10.43.0.10:53: no such host },] }”, “errorVerbose”: “server selection error: context deadline exceeded, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mongo-psmdb-db-rs0-0.mongo-psmdb-db-rs0.tungdt.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup mongo-psmdb-db-rs0-0.mongo-psmdb-db-rs0.tungdt.svc.cluster.local on 10.43.0.10:53: no such host }, { Addr: mongo-psmdb-db-rs0-1.mongo-psmdb-db-rs0.tungdt.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup mongo-psmdb-db-rs0-1.mongo-psmdb-db-rs0.tungdt.svc.cluster.local on 10.43.0.10:53: no such host }, { Addr: mongo-psmdb-db-rs0-2.mongo-psmdb-db-rs0.tungdt.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup mongo-psmdb-db-rs0-2.mongo-psmdb-db-rs0.tungdt.svc.cluster.local on 10.43.0.10:53: no such host },] }\nping mongo\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo.Dial\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo/mongo.go:112\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb.MongoClient\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/client.go:62\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*mongoClientProvider).Mongo\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/connections.go:38\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).mongoClientWithRole\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/connections.go:60\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:87\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb\_controller.go:498\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm\_amd64.s:1598\ndial\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:93\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb\_controller.go:498\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm\_amd64.s:1598”} [github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(\*ReconcilePerconaServerMongoDB).Reconcile](http://github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb\_controller.go:500 ## Expected Result: The cluster returns to normal operation and rs.status() displays information about the ready state of the cluster ## Actual Result: The cluster enters the RS Ghost state and becomes inoperable ## Additional Information: ![photo_2024-04-25_15-57-02](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/2/2/220aefaba9f418f0657c33801985d61136aa5813.jpeg) ---
### Author: ![Trinh.Duc.Chung](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/trinh.duc.chung/32/11146_2.png) [@Trinh.Duc.Chung](https://forums.percona.com/u/Trinh.Duc.Chung) #### Post date: [April 25, 2024, 9:00am UTC](https://forums.percona.com/t/mongodb-cluster-cannot-failover-when-down-time-all-pods-and-using-mode-external-nodeport-and-lb/29842/2 "2024-04-25T09:00:39Z")
Can anyone help me to fix issue? ---
### Author: ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin) #### Post date: [April 29, 2024, 8:29am UTC](https://forums.percona.com/t/mongodb-cluster-cannot-failover-when-down-time-all-pods-and-using-mode-external-nodeport-and-lb/29842/3 "2024-04-29T08:29:17Z")
Hello @Trinh.Duc.Chung , I reproduced it. Will discuss with the team internally and get back to you. UPD: I believe you also submitted this PR: [K8SPSMDB-1074 fixed MongoDB Cluster cannot failover when down time all pods using mode External (NodePort and LB) by chungtd by chungtd203338 · Pull Request #1535 · percona/percona-server-mongodb-operator · GitHub](https://github.com/percona/percona-server-mongodb-operator/pull/1535) We will have a look. Also a quick question - is there any reason why you need to use External mode vs using splitHorizons? ---
### Author: ![Trinh.Duc.Chung](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/trinh.duc.chung/32/11146_2.png) [@Trinh.Duc.Chung](https://forums.percona.com/u/Trinh.Duc.Chung) #### Post date: [May 2, 2024, 10:23am UTC](https://forums.percona.com/t/mongodb-cluster-cannot-failover-when-down-time-all-pods-and-using-mode-external-nodeport-and-lb/29842/4 "2024-05-02T10:23:32Z")
Hi Mr.@Sergey_Pronin! Thank you for your response regarding my issue! I indeed created that pull request. I discovered how to resolve this error by consulting the official Mongodb documentation. You can view it here: [https://www.mongodb.com/docs/kubernetes-operator/master/tutorial/connect-from-outside-k8s/](https://www.mongodb.com/docs/kubernetes-operator/master/tutorial/connect-from-outside-k8s/). As for why I need to use External mode instead of using splitHorizons because some of our infrastructure does not have loadbalancer, besides splithorizons Mongodb operator is fixing port 27017, we will have cases where we need to deploy nodeport mode and Use service loadbalancer with port other than 27017. Looking forward to the response on my pull request creation. We are in need of an operator that can fix this error as soon as possible. Thanks !