# Mongo pods constantly restart due to failed liveness probe

**URL:** <https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757>\
**Category:** Percona Operator for MongoDB\
**Tags:** percona, mongodb\
**Created:** [August 12, 2021, 10:33am UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757 "2021-08-12T10:33:27Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![vhphan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vhphan/32/4187_2.png) [@vhphan](https://forums.percona.com/u/vhphan)\
**Post date:** [August 12, 2021, 10:33am UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/1 "2021-08-12T10:33:27Z")

</div>

Hello,

When I create a PerconaServerMongoDB in replicaset mode, my pods restart constantly due to failed liveness probes with this error: _“connection error: no reachable servers”_.

I can connect to the pod and run _/data/db/mongodb-healthcheck k8s liveness --ssl --sslInsecure --sslCAFile /etc/mongodb-ssl/ca.crt --sslPEMKeyFile /tmp/tls.pem_ as in the liveness probe definition and indeed it fails with the same connection error.

I am running on RKE 1.18.6. My CR will be in the comment.

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![vhphan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vhphan/32/4187_2.png) [@vhphan](https://forums.percona.com/u/vhphan)\
**Post date:** [August 12, 2021, 10:34am UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/2 "2021-08-12T10:34:35Z")

</div>

I cannot upload yaml file here so I will just put the raw file. 😕

```auto
apiVersion: psmdb.percona.com/v1-9-0
kind: PerconaServerMongoDB
metadata:
  name: mongodb
  finalizers:
    - delete-psmdb-pvc
spec:
  crVersion: 1.9.0
  image: "percona/percona-server-mongodb:4.4.6-8"
  imagePullPolicy: Always
  allowUnsafeConfigurations: true
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: 4.4-recommended
    schedule: 0 2 * * *
    setFCV: false
  secrets:
    users: 
  pmm:
    enabled: false
    image: percona/pmm-client:2.18.0
    serverHost: monitoring-service
  replsets:
    - name: rs0
      size: 3
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      # advanced:
      # nodeAffinity:
      # requiredDuringSchedulingIgnoredDuringExecution:
      # nodeSelectorTerms:
      # - matchExpressions:
      # - key: kubernetes.io/e2e-az-name
      # operator: In
      # values:
      # - e2e-az1
      # - e2e-az2
      # tolerations:
      # - key: "node.alpha.kubernetes.io/unreachable"
      # operator: "Exists"
      # effect: "NoExecute"
      # tolerationSeconds: 6000
      # priorityClassName: high-priority
      # annotations:
      # iam.amazonaws.com/role: role-arn
      # labels:
      # rack: rack-22
      # nodeSelector:
      # disktype: ssd
      livenessProbe:
        failureThreshold: 4
        initialDelaySeconds: 60
        periodSeconds: 30
        successThreshold: 1
        timeoutSeconds: 10
        startupDelaySeconds: 7200
      # runtimeClassName: image-rc
      # sidecars:
      # - image: busybox
      # command: ["/bin/sh"]
      # args: ["-c", "while true; do echo echo $(date -u) 'test' >> /dev/null; sleep 5;done"]
      # name: rs-sidecar-1
      podDisruptionBudget:
        maxUnavailable: 1
        minAvailable: 3
      expose:
        enabled: true
        exposeType: LoadBalancer
        serviceAnnotations:
          service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
      arbiter:
        enabled: false
      resources:
        limits:
          cpu: "2"
          memory: 8Gi
          requests:
            cpu: "2"
            memory: 8Gi
      volumeSpec:
        # emptyDir: {}
        # hostPath:
        # path: /data
        # type: Directory
        persistentVolumeClaim:
          storageClassName: percona-mongodb
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

  sharding:
    enabled: false

  mongod:
    net:
      hostPort: 0

  backup:
    enabled: false
    restartOnFailure: true
    image: percona/percona-server-mongodb-operator:1.9.0-backup
    serviceAccountName: percona-server-mongodb-operator
    resources:

    storages:
    # s3-us-west:
    # type: s3
    # s3:
    # bucket: S3-BACKUP-BUCKET-NAME-HERE
    # credentialsSecret: my-cluster-name-backup-s3
    # region: us-west-2
    # minio:
    # type: s3
    # s3:
    # bucket: MINIO-BACKUP-BUCKET-NAME-HERE
    # region: us-east-1
    # credentialsSecret: my-cluster-name-backup-minio
    # endpointUrl: http://minio.psmdb.svc.cluster.local:9000/minio/
    pitr:
      enabled: true
    tasks:
# - name: daily-s3-us-west
# enabled: true
# schedule: "0 0 * * *"
# keep: 3
# storageName: s3-us-west
# compressionType: gzip
# - name: weekly-s3-us-west
# enabled: false
# schedule: "0 0 * * 0"
# keep: 5
# storageName: s3-us-west
  # compressionType: gzip

```

---

<div class="post-metadata">

**Author:** ![jamoser](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@jamoser](https://forums.percona.com/u/jamoser)\
**Post date:** [August 12, 2021, 1:06pm UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/3 "2021-08-12T13:06:06Z")

</div>

may be add some users ?

---

<div class="post-metadata">

**Author:** ![vhphan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vhphan/32/4187_2.png) [@vhphan](https://forums.percona.com/u/vhphan)\
**Post date:** [August 12, 2021, 1:48pm UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/4 "2021-08-12T13:48:36Z")

</div>

Oh yeah I commented that out before pasting it here, forgot to put the user secrets back. But even with user registered I still have the same error. ☹

---

<div class="post-metadata">

**Author:** ![jamoser](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@jamoser](https://forums.percona.com/u/jamoser)\
**Post date:** [August 12, 2021, 2:05pm UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/5 "2021-08-12T14:05:28Z")

</div>

may be you should do something like

kubectl get pods

so we can see if you installation is correct

because your cr.yaml looks not correct ← load the original file in a editor where you can see clearly what belongs to what … and tell what you actually want (only replicaset ? with sharding ?)

---

<div class="post-metadata">

**Author:** ![vhphan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vhphan/32/4187_2.png) [@vhphan](https://forums.percona.com/u/vhphan)\
**Post date:** [August 12, 2021, 2:39pm UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/6 "2021-08-12T14:39:51Z")

</div>

I did load the original file from Github and modify it to meet my need, so it looks a bit different than the original one. Can you point out which part of my cr.yaml doesn’t look correct please?

Just to make sure, I tried putting the original file on, same error. I hope it isn’t a platform-specific issue. I made it work a few weeks ago with GKE and now I’m trying to make it work on an on-premise RKE cluster.

---

<div class="post-metadata">

**Author:** ![jamoser](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@jamoser](https://forums.percona.com/u/jamoser)\
**Post date:** [August 12, 2021, 3:05pm UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/7 "2021-08-12T15:05:30Z")

</div>

> > made it work a few weeks ago with GKE

so I guess this is not a Percona Operator issue and more like that you don’t know RKE

---

<div class="post-metadata">

**Author:** ![vhphan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vhphan/32/4187_2.png) [@vhphan](https://forums.percona.com/u/vhphan)\
**Post date:** [August 18, 2021, 9:47am UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/8 "2021-08-18T09:47:48Z")

</div>

Thanks, very useful. =.=!

Actually it turns out that the problem only happen when I expose the replicaset with LoadBalancer or NodePort.

When I use NodePort and connect to one of the pod and launch:  
`mongo --host <node_ip> --port <node_port>`

It returns `connection refused`.

By curiosity I tried spinning up a single pod running `percona/percona-server-mongodb:4.4.6-8` image and start `mongod --bind_ip_all --port 27017`, then expose the port as a NodePort service, I can connect to this test mongo normally.

---

<div class="post-metadata">

**Author:** ![Shubham\_Harilal\_Saro](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/shubham_harilal_saro/32/14919_2.png) [@Shubham\_Harilal\_Saro](https://forums.percona.com/u/Shubham_Harilal_Saro)\
**Post date:** [April 23, 2024, 7:32pm UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/9 "2024-04-23T19:32:20Z")

</div>

@vhphan

What was the solution. I am facing the same issue

---

<div class="post-metadata">

**Author:** ![owlee](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/owlee/32/16279_2.png) [@owlee](https://forums.percona.com/u/owlee)\
**Post date:** [January 9, 2025, 5:08pm UTC](https://forums.percona.com/t/mongo-pods-constantly-restart-due-to-failed-liveness-probe/11757/10 "2025-01-09T17:08:57Z")

</div>

@Shubham_Harilal_Saro@vhphan

Did you find any solution? still suffering this in 2025, been looking for a solution for quite some time now.