# Liveness probe failed replisetStatus command need authentication

**URL:** <https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526>\
**Category:** Percona Operator for MongoDB\
**Created:** [May 20, 2021, 1:28am UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526 "2021-05-20T01:28:17Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![wenjian](https://avatars.discourse-cdn.com/v4/letter/w/9de053/32.png) [@wenjian](https://forums.percona.com/u/wenjian)\
**Post date:** [May 20, 2021, 1:28am UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/1 "2021-05-20T01:28:17Z")

</div>

I am using the operator 1.8.0 to deploy a 3 nodes MongoDB cluster with LPV. in a 3 nodes k8s

All the pods are running when I firstly deploy.

When I see the logs of one pod, I can see the error log shows me that `clusterMonitor` authentication failed:

```auto
{"t":{"$date":"2021-05-20T01:17:44.934+00:00"},"s":"I", "c":"ACCESS", "id":20249, "ctx":"conn33","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-1","principalName":"clusterMonitor","authenticationDatabase":"admin","client":"127.0.0.1:36118","result":"UserNotFound: Could not find user \"clusterMonitor\" for db \"admin\""}}

```

and describe the pod shows the error I put in the title:

```auto
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 12m (x7 over 17m) default-scheduler 0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate, 2 node(s) had volume node affinity conflict.
  Normal Scheduled 12m default-scheduler Successfully assigned percona/my-cluster-name-rs0-2 to k8s-workernode-1
  Normal Pulling 12m kubelet Pulling image "percona/percona-server-mongodb-operator:1.8.0"
  Normal Pulled 10m kubelet Successfully pulled image "percona/percona-server-mongodb-operator:1.8.0" in 1m13.862005939s
  Normal Created 10m kubelet Created container mongo-init
  Normal Started 10m kubelet Started container mongo-init
  Warning Unhealthy 9m43s kubelet Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2021-05-20T01:15:54Z"}
{"level":"error","msg":"replSetGetStatus returned error command replSetGetStatus requires authentication","time":"2021-05-20T01:15:54Z"}
  Warning Unhealthy 9m14s kubelet Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2021-05-20T01:16:23Z"}
{"level":"error","msg":"replSetGetStatus returned error command replSetGetStatus requires authentication","time":"2021-05-20T01:16:23Z"}
  Warning Unhealthy 8m44s kubelet Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2021-05-20T01:16:53Z"}
{"level":"error","msg":"replSetGetStatus returned error command replSetGetStatus requires authentication","time":"2021-05-20T01:16:53Z"}
  Warning Unhealthy 8m14s kubelet Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2021-05-20T01:17:23Z"}
{"level":"error","msg":"replSetGetStatus returned error command replSetGetStatus requires authentication","time":"2021-05-20T01:17:23Z"}
  Normal Created 8m13s (x2 over 10m) kubelet Created container mongod
  Normal Pulled 8m13s (x2 over 10m) kubelet Container image "percona/percona-server-mongodb:4.4.5-7" already present on machine
  Normal Started 8m12s (x2 over 10m) kubelet Started container mongod
  Warning Unhealthy 6m44s kubelet Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2021-05-20T01:18:53Z"}
{"level":"error","msg":"replSetGetStatus returned error command replSetGetStatus requires authentication","time":"2021-05-20T01:18:53Z"}
  Warning Unhealthy 6m14s kubelet Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2021-05-20T01:19:23Z"}
{"level":"error","msg":"replSetGetStatus returned error command replSetGetStatus requires authentication","time":"2021-05-20T01:19:23Z"}

```

When I attached into the pod then execute `mongo` shell to connect local mongodb instance, I can only see 1 user in admin database, that is ‘userAdmin’

 ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/3/3b219384c0fc937951f021e636ec622aa2e77b2c.png)

When I try to connect another pod using statefulset stable network ID, for example execute below command in pod `my-cluster-name-rs0-0`:

` mongo "mongodb://my-cluster-name-rs0-1.my-cluster-name-rs0.percona.svc.cluster.local" --verbose`

> percona is my namespace

I got Host not found exception.

Someone could please help on this issue?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![wenjian](https://avatars.discourse-cdn.com/v4/letter/w/9de053/32.png) [@wenjian](https://forums.percona.com/u/wenjian)\
**Post date:** [May 20, 2021, 1:43am UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/2 "2021-05-20T01:43:53Z")

</div>

any additional info I need to attach, please let me know

---

<div class="post-metadata">

**Author:** ![wenjian](https://avatars.discourse-cdn.com/v4/letter/w/9de053/32.png) [@wenjian](https://forums.percona.com/u/wenjian)\
**Post date:** [May 20, 2021, 2:23am UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/3 "2021-05-20T02:23:01Z")

</div>

redeploy again, same issue

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [May 24, 2021, 11:20am UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/4 "2021-05-24T11:20:10Z")

</div>

Hello @wenjian ,

could you please share the cr.yaml that you use to deploy the cluster?  
Are you changing the secrets when deploying?

---

<div class="post-metadata">

**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\
**Post date:** [June 6, 2021, 5:53pm UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/5 "2021-06-06T17:53:38Z")

</div>

@Sergey_Pronin Does usage of existing secrets cause any issue to the cluster?

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [June 6, 2021, 7:50pm UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/6 "2021-06-06T19:50:57Z")

</div>

@SpoorthiPalakshaiah what do you mean by “existing” ?

But changing system user secrets is a process that might trigger the pods restart or containers reload.

---

<div class="post-metadata">

**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\
**Post date:** [June 8, 2021, 7:51pm UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/7 "2021-06-08T19:51:49Z")

</div>

I’m using same secrets as in the secrets.yaml in documentation; But still the cluster is unstable;  
Liveness and readiness probes fails few times but the pod stays in running state; What could be the reason for this behavior in the cluster

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [June 14, 2021, 8:37am UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/8 "2021-06-14T08:37:53Z")

</div>

The failures of liveness and readiness probes are not triggered by secrets.  
The most likely reason would be resources saturation.  
What does your monitoring tells you? What is the current utilization (CPU/RAM) for Mongo related containers?

---

<div class="post-metadata">

**Author:** ![wenjian](https://avatars.discourse-cdn.com/v4/letter/w/9de053/32.png) [@wenjian](https://forums.percona.com/u/wenjian)\
**Post date:** [June 29, 2021, 6:48am UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/9 "2021-06-29T06:48:12Z")

</div>

I changed to using hostPath, now everything is fine.

I assumed that the issue was caused by some miss configured PV / PVC maybe

---

<div class="post-metadata">

**Author:** ![vhphan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vhphan/32/4187_2.png) [@vhphan](https://forums.percona.com/u/vhphan)\
**Post date:** [August 24, 2021, 6:56am UTC](https://forums.percona.com/t/liveness-probe-failed-replisetstatus-command-need-authentication/10526/10 "2021-08-24T06:56:46Z")

</div>

To top up the conversation, I sometimes have the same issue. Turns out it’s because I redeploy the CR many times without cleaning up the PVC.