# Percona MongoDB instance don't launch with replicas after downtime

**URL:** <https://forums.percona.com/t/percona-mongodb-instance-dont-launch-with-replicas-after-downtime/24251>\
**Category:** Percona Operator for MongoDB\
**Created:** [August 7, 2023, 3:59am UTC](https://forums.percona.com/t/percona-mongodb-instance-dont-launch-with-replicas-after-downtime/24251 "2023-08-07T03:59:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dung-tien-nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dung-tien-nguyen/32/11999_2.png) [@dung-tien-nguyen](https://forums.percona.com/u/dung-tien-nguyen)\
**Post date:** [August 7, 2023, 3:59am UTC](https://forums.percona.com/t/percona-mongodb-instance-dont-launch-with-replicas-after-downtime/24251/1 "2023-08-07T03:59:59Z")

</div>

## Description:

With Percona server Mongodb in k8s using Operator,  
After a case of unexpected full node downtime, the pods are back running but do not join replicas and are in a ghost state.  
Does Percona have any solution, what is the manual solution to fix this problem, please help me…

## Steps to Reproduce:

Shutdown k8s node or using kubectl delete --force all\_pod\_in\_rs

## Version:

I’m using Operator 1.14.0, MongoDB server 6.0.4-3

## Logs:

When exec into pod, run cmd: rs.status(), output: MongoServerError: Our replica set config is invalid or we are not a member of it

## Expected Result:

A automate solution or manual solution while waiting for roadmap

## Actual Result:

## Additional Information:

 ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/a/a4a744e35bfb90b9d71a24a9bbb83f64e706e109.png)

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [September 1, 2023, 7:54am UTC](https://forums.percona.com/t/percona-mongodb-instance-dont-launch-with-replicas-after-downtime/24251/2 "2023-09-01T07:54:02Z")

</div>

Hello @dung-tien-nguyen ,

I was not able to reproduce it with the same versions. Killed the Pods multiple times, can connect and cluster is healthy.

It also important to connect to a replicaset in the right way - for example:

```auto
mongosh "mongodb+srv://clusterAdmin:clusterAdmin123456@my-cluster-name-rs0.default.svc.cluster.local/admin?replicaSet=rs0&ssl=false"

```

What is psmdb object showing and Pods?

```auto
kubectl get pods
kubectl get psmdb

```

---

<div class="post-metadata">

**Author:** ![dung-tien-nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dung-tien-nguyen/32/11999_2.png) [@dung-tien-nguyen](https://forums.percona.com/u/dung-tien-nguyen)\
**Post date:** [September 1, 2023, 3:24pm UTC](https://forums.percona.com/t/percona-mongodb-instance-dont-launch-with-replicas-after-downtime/24251/3 "2023-09-01T15:24:22Z")

</div>

This error occurs when specifying the **clusterServiceDNSMode=External value** , and exposing using the **NodePort** or **LoadBalancer** service.  
**clusterServiceDNSMode=Internal** does not have this problem. @Sergey_Pronin

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [September 4, 2023, 10:23am UTC](https://forums.percona.com/t/percona-mongodb-instance-dont-launch-with-replicas-after-downtime/24251/4 "2023-09-04T10:23:26Z")

</div>

@dung-tien-nguyen you are right, it does not work that way.  
In the next release of the operator (coming in September), we will have split horizon feature. Thus you can have clusterServiceDNSMode=Internal, but at the same time use loadbalancers. This should address it.

---

<div class="post-metadata">

**Author:** ![dung-tien-nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dung-tien-nguyen/32/11999_2.png) [@dung-tien-nguyen](https://forums.percona.com/u/dung-tien-nguyen)\
**Post date:** [September 5, 2023, 3:21am UTC](https://forums.percona.com/t/percona-mongodb-instance-dont-launch-with-replicas-after-downtime/24251/5 "2023-09-05T03:21:13Z")

</div>

but when set **clusterServiceDNSMode=Internal** and accessing from outside the k8s cluster, users can only connect to single mongoDB instances, right?