# CrashLoopBackOff when install psmdb-operator v1.15.0

**URL:** <https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137>\
**Category:** Percona Operator for MongoDB\
**Tags:** psmdb-operator\
**Created:** [October 24, 2023, 7:58pm UTC](https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137 "2023-10-24T19:58:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![psamagal](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/psamagal/32/13495_2.png) [@psamagal](https://forums.percona.com/u/psamagal)\
**Post date:** [October 24, 2023, 7:58pm UTC](https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137/1 "2023-10-24T19:58:49Z")

</div>

## Description:

Hi all!

I was trying to deploy the Percona Operator for MongoDB using the helm chart following the instructions on

> **[percona-helm-charts/charts/psmdb-operator at psmdb-operator-1.15.0 ·...](https://github.com/percona/percona-helm-charts/tree/psmdb-operator-1.15.0/charts/psmdb-operator)**
>
> Collection of Helm charts for Percona Kubernetes Operators. - percona/percona-helm-charts

Unfortunately I’m not able to get a stable operator working for long. After a few minutes of deploy the operator throws an error and restart.

## Steps to Reproduce:

The process I used for installation was

```auto
helm install mongodb-psmdb-operator percona/psmdb-operator -n mongodb-temp --version 1.15.0 -f percona.yml 

```

being the custom contents of the _percona.yml_

```auto
logLevel: DEBUG
resources:
  limits:
    cpu: 1
    memory: 1Gi
  requests:
    cpu: 1
    memory: 1Gi
disableTelemetry: true

```

the remaining will be the defaults for this version of the chart 1.15.0

## Version:

v1.15.0  
Deployed into GKE 1.25

## Logs:

Logs of the operator till it crashes

```auto
2023-10-24T19:01:39.873Z INFO setup Manager starting up {"gitCommit": "ed2d8b4907c39beadfb020ce1cb555fee0ac682d", "gitBranch": "release-1-15-0", "goVersion": "go1.20.9", "os": "linux", "arch": "amd64"}
I1024 19:01:40.924853 1 request.go:697] Waited for 1.034285823s due to client-side throttling, not priority and fairness, request: GET:https://10.145.32.1:443/apis/constraints.gatekeeper.sh/v1alpha1?timeout=32s
2023-10-24T19:01:43.435Z INFO server version {"platform": "kubernetes", "version": "v1.25.12-gke.500"}
2023-10-24T19:01:43.445Z INFO starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-10-24T19:01:43.445Z INFO controller-runtime.metrics Starting metrics server
I1024 19:01:43.445368 1 leaderelection.go:250] attempting to acquire leader lease mongodb-temp/08db0feb.percona.com...
2023-10-24T19:01:43.445Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
I1024 19:02:01.263394 1 leaderelection.go:260] successfully acquired lease mongodb-temp/08db0feb.percona.com
2023-10-24T19:02:01.263Z DEBUG events mongodb-psmdb-operator-84849858b8-mp2h9_ba6e5c4f-850a-4dac-a73b-4062eaf3861d became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"mongodb-temp","name":"08db0feb.percona.com","uid":"855ef146-fd90-41b9-8c86-cd0b82697425","apiVersion":"coordination.k8s.io/v1","resourceVersion":"2108411632"}, "reason": "LeaderElection"}
2023-10-24T19:02:01.263Z INFO Starting EventSource {"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2023-10-24T19:02:01.263Z INFO Starting Controller {"controller": "psmdb-controller"}
2023-10-24T19:02:01.263Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2023-10-24T19:02:01.263Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2023-10-24T19:02:01.263Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2023-10-24T19:02:01.263Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2023-10-24T19:02:01.263Z INFO Starting Controller {"controller": "psmdbbackup-controller"}
2023-10-24T19:02:01.263Z INFO Starting Controller {"controller": "psmdbrestore-controller"}
2023-10-24T19:02:01.397Z INFO Starting workers {"controller": "psmdb-controller", "worker count": 1}
2023-10-24T19:02:01.401Z INFO Starting workers {"controller": "psmdbrestore-controller", "worker count": 1}
2023-10-24T19:02:01.401Z INFO Starting workers {"controller": "psmdbbackup-controller", "worker count": 1}
E1024 19:03:27.804464 1 leaderelection.go:369] Failed to update lock: Put "https://10.145.32.1:443/apis/coordination.k8s.io/v1/namespaces/mongodb-temp/leases/08db0feb.percona.com": context deadline exceeded
I1024 19:03:27.804527 1 leaderelection.go:285] failed to renew lease mongodb-temp/08db0feb.percona.com: timed out waiting for the condition
2023-10-24T19:03:27.804Z DEBUG events mongodb-psmdb-operator-84849858b8-mp2h9_ba6e5c4f-850a-4dac-a73b-4062eaf3861d stopped leading {"type": "Normal", "object": {"kind":"Lease","namespace":"mongodb-temp","name":"08db0feb.percona.com","uid":"855ef146-fd90-41b9-8c86-cd0b82697425","apiVersion":"coordination.k8s.io/v1","resourceVersion":"2108413524"}, "reason": "LeaderElection"}
2023-10-24T19:03:27.804Z INFO Stopping and waiting for non leader election runnables
2023-10-24T19:03:27.806Z INFO Stopping and waiting for leader election runnables
2023-10-24T19:03:27.806Z INFO Stopping and waiting for caches
2023-10-24T19:03:27.804Z ERROR setup problem running manager {"error": "leader election lost"}
main.main
        /go/src/github.com/percona/percona-server-mongodb-operator/cmd/manager/main.go:161
runtime.main
        /usr/local/go/src/runtime/proc.go:250

```

## Expected Result:

To the operator pod not to go into **CrashLoopBackOff**

Do you have any idea, what could be the cause of this, or how can I fix it?

---

<div class="post-metadata">

**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\
**Post date:** [October 25, 2023, 6:01pm UTC](https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137/2 "2023-10-25T18:01:13Z")

</div>

Hi, the operator requires a minimum of 2 GB of ram and 2 CPU threads per node. See this for more info: [System requirements - Percona Operator for MongoDB](https://docs.percona.com/percona-operator-for-mongodb/System-Requirements.html#resource-limits)

---

<div class="post-metadata">

**Author:** ![psamagal](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/psamagal/32/13495_2.png) [@psamagal](https://forums.percona.com/u/psamagal)\
**Post date:** [October 26, 2023, 7:41am UTC](https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137/3 "2023-10-26T07:41:36Z")

</div>

Hi Ivan, Thanks for the quick feedback

Following your suggestion I did increase the resources allocated to the operator from 1 to 4 CPU and from 1 to 4Gi of memory and redeployed the psmdb-operator. But the same situation keeps happening

```auto
E1026 06:49:40.838717 1 leaderelection.go:369] Failed to update lock: Put "https://10.145.32.1:443/apis/coordination.k8s.io/v1/namespaces/mongodb-temp/leases/08db0feb.percona.com": context deadline exceeded
I1026 06:49:40.838758 1 leaderelection.go:285] failed to renew lease mongodb-temp/08db0feb.percona.com: timed out waiting for the condition
2023-10-26T06:49:40.838Z ERROR setup problem running manager {"error": "leader election lost"}
main.main
        /go/src/github.com/percona/percona-server-mongodb-operator/cmd/manager/main.go:161
runtime.main
        /usr/local/go/src/runtime/proc.go:250

```

I do not know the internals of the operator, but this appears to be some kind of timeout communicating with GKE API. Is there any way o tweak this timeout values?

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [October 26, 2023, 7:58am UTC](https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137/4 "2023-10-26T07:58:52Z")

</div>

@psamagal I deployed GKE 1.25 and used the same values YAML that you provided. Operator started just fine.

What is the size of the nodes in your kubernetes cluster?  
Were there any other parameters tuned in GKE?

---

<div class="post-metadata">

**Author:** ![psamagal](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/psamagal/32/13495_2.png) [@psamagal](https://forums.percona.com/u/psamagal)\
**Post date:** [October 26, 2023, 8:26am UTC](https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137/5 "2023-10-26T08:26:12Z")

</div>

At the moment it’s running in nodes with the following characteristics

```auto
Image type: Container-optimised OS with containerd (cos_containerd)
Machine type: n1-standard-8
Boot disk type: Standard persistent disk
Boot disk size: 100 GB
Provisioning model: Spot

```

No other parameters were tunned on the operator deployment and similar for the nodes, they are pretty much default configuration.  
Is there anything in particular that I should look for?

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [October 26, 2023, 11:28am UTC](https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137/6 "2023-10-26T11:28:42Z")

</div>

@psamagal please show the following:

1. `kubectl get pods -n mongodb-temp`
2. `kubectl describe pod <OPERATOR_POD> -n mongodb-temp`

---

<div class="post-metadata">

**Author:** ![psamagal](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/psamagal/32/13495_2.png) [@psamagal](https://forums.percona.com/u/psamagal)\
**Post date:** [October 26, 2023, 12:55pm UTC](https://forums.percona.com/t/crashloopbackoff-when-install-psmdb-operator-v1-15-0/26137/7 "2023-10-26T12:55:35Z")

</div>

Here you have the get pods. Currently the operator is the only thing running in this namespace

```auto
$ kubectl get pods -n mongodb-temp
NAME READY STATUS RESTARTS AGE
mongodb-psmdb-operator-75d5bbbd4-fcqss 0/1 CrashLoopBackOff 5 (28s ago) 16m

```

for the describe pod

```auto
$ kubectl describe pod mongodb-psmdb-operator-75d5bbbd4-fcqss -n mongodb-temp
Name: mongodb-psmdb-operator-75d5bbbd4-fcqss
Namespace: mongodb-temp
Priority: 0
Node: gke-<redacted>--pool-a-n1-std-8-f8aaf4b2-njm7/10.145.8.3
Start Time: Thu, 26 Oct 2023 14:32:31 +0200
Labels: app.kubernetes.io/instance=mongodb-psmdb-operator
              app.kubernetes.io/name=psmdb-operator
              pod-template-hash=75d5bbbd4
Annotations: <none>
Status: Running
IP: 10.145.20.130
IPs:
  IP: 10.145.20.130
Controlled By: ReplicaSet/mongodb-psmdb-operator-75d5bbbd4
Containers:
  psmdb-operator:
    Container ID: containerd://2be8442c255ae11adc4f67583b3d04012f4b6da56fa1f839ccd301e267a03f14
    Image: percona/percona-server-mongodb-operator:1.15.0
    Image ID: docker.io/percona/percona-server-mongodb-operator@sha256:d8a5b33db1938d42769cb5a87d34a128332a2d0302eaa6d7c860e7c4667ea3b6
    Port: 60000/TCP
    Host Port: 0/TCP
    Command:
      percona-server-mongodb-operator
    State: Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason: Error
      Exit Code: 1
      Started: Thu, 26 Oct 2023 14:46:58 +0200
      Finished: Thu, 26 Oct 2023 14:47:43 +0200
    Ready: False
    Restart Count: 5
    Limits:
      cpu: 4
      memory: 4Gi
    Requests:
      cpu: 4
      memory: 4Gi
    Environment:
      LOG_STRUCTURED: false
      LOG_LEVEL: DEBUG
      WATCH_NAMESPACE: mongodb-temp
      POD_NAME: mongodb-psmdb-operator-75d5bbbd4-fcqss (v1:metadata.name)
      OPERATOR_NAME: percona-server-mongodb-operator
      RESYNC_PERIOD: 5s
      DISABLE_TELEMETRY: true
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w94fn (ro)
Conditions:
  Type Status
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  kube-api-access-w94fn:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional: <nil>
    DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 15m default-scheduler Successfully assigned mongodb-temp/mongodb-psmdb-operator-75d5bbbd4-fcqss to gke-<redacted>--pool-a-n1-std-8-f8aaf4b2-njm7
  Normal Pulling 15m kubelet Pulling image "percona/percona-server-mongodb-operator:1.15.0"
  Normal Pulled 15m kubelet Successfully pulled image "percona/percona-server-mongodb-operator:1.15.0" in 4.916799598s (13.610398918s including waiting)
  Normal Created 3m51s (x5 over 15m) kubelet Created container psmdb-operator
  Normal Started 3m51s (x5 over 15m) kubelet Started container psmdb-operator
  Normal Pulled 3m51s (x4 over 14m) kubelet Container image "percona/percona-server-mongodb-operator:1.15.0" already present on machine
  Warning BackOff 34s (x13 over 12m) kubelet Back-off restarting failed container

```