Mongodb container is not coming up and psmdb is showing stopping state

Mongodb container is not coming up and psmdb is showing stopping state

Version:

1.15
kubectl get pods --all-namespaces | grep mongo
mongodb-operator1 my-cluster-name-cfg-0 1/2 CrashLoopBackOff 8 (3m38s ago) 20m
mongodb-operator1 my-cluster-name-rs0-0 1/2 CrashLoopBackOff 8 (4m1s ago) 20m
mongodb-operator1 percona-server-mongodb-operator-788df88f8f-7m2nj 1/1 Running 0 21m
onap dev-multicloud-k8s-mongo-0 1/1 Running 0 34d
onap dev-nbi-mongo-0 1/1 Running 0 34d
user@bastion:~$ kubectl describe pod my-cluster-name-cfg-0 -n mongodb-operator1
Name: my-cluster-name-cfg-0
Namespace: mongodb-operator1
Priority: 0
Service Account: default
Node: worker-node-1/10.151.27.42
Start Time: Wed, 21 Feb 2024 13:52:21 +0000
Labels: app.kubernetes.io/component=cfg
app.kubernetes.io/instance=my-cluster-name
app.kubernetes.io/managed-by=percona-server-mongodb-operator
app.kubernetes.io/name=percona-server-mongodb
app.kubernetes.io/part-of=percona-server-mongodb
app.kubernetes.io/replset=cfg
controller-revision-hash=my-cluster-name-cfg-6856b5f69
statefulset.kubernetes.io/pod-name=my-cluster-name-cfg-0
Annotations: cni.projectcalico.org/containerID: 921dce507240ce5c381324d0dfa3b42c2ebeb798ada4e9e6e019e6f2be97066d
cni.projectcalico.org/podIP: 10.233.84.247/32
cni.projectcalico.org/podIPs: 10.233.84.247/32
Open Source Database Software Support & Services | Percona bbd128646ab2b7452c36477970c0db24
Open Source Database Software Support & Services | Percona a8ed1c9dc435430943376a322973e396
Status: Running
IP: 10.233.84.247
IPs:
IP: 10.233.84.247
Controlled By: StatefulSet/my-cluster-name-cfg
Init Containers:
mongo-init:
Container ID: containerd://bd9e2b26ffe99ac2e3297296f6dc8b3894c046620913d1029369abb159c8bcf3
Image: perconalab/percona-server-mongodb-operator:main
Image ID: docker.io/perconalab/percona-server-mongodb-operator@sha256:d8b2da3f4a1f6c7a890d1083e682e51a289bf066e1c7dfdfba81fc70e0136587
Port:
Host Port:
Command:
/init-entrypoint.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 21 Feb 2024 13:52:29 +0000
Finished: Wed, 21 Feb 2024 13:52:29 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 300m
memory: 500M
Requests:
cpu: 300m
memory: 500M
Environment:
Mounts:
/data/db from mongod-data (rw)
/opt/percona from bin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c64k4 (ro)
Containers:
mongod:
Container ID: containerd://659d11caa87f850d6b48b2606553d166a689fa1372e530220265c40aaef65aeb
Image: perconalab/percona-server-mongodb-operator:main-mongod6.0
Image ID: docker.io/perconalab/percona-server-mongodb-operator@sha256:758cbcc09eb01ed362bce8adfccb05351c02fd7985f7903f53746cc1c0c94595
Port: 27017/TCP
Host Port: 0/TCP
Command:
/opt/percona/ps-entry.sh
Args:
–bind_ip_all
–auth
–dbpath=/data/db
–port=27017
–replSet=cfg
–storageEngine=wiredTiger
–relaxPermChecks
–sslAllowInvalidCertificates
–clusterAuthMode=x509
–configsvr
–enableEncryption
–encryptionKeyFile=/etc/mongodb-encryption/encryption-key
–wiredTigerCacheSizeGB=0.25
–wiredTigerIndexPrefixCompression=true
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 132
Started: Wed, 21 Feb 2024 14:09:00 +0000
Finished: Wed, 21 Feb 2024 14:09:00 +0000
Ready: False
Restart Count: 8
Limits:
cpu: 300m
memory: 500M
Requests:
cpu: 300m
memory: 500M
Liveness: exec [/opt/percona/mongodb-healthcheck k8s liveness --ssl --sslInsecure --sslCAFile /etc/mongodb-ssl/ca.crt --sslPEMKeyFile /tmp/tls.pem --startupDelaySeconds 7200] delay=60s timeout=10s period=30s #success=1 #failure=4
Readiness: exec [/opt/percona/mongodb-healthcheck k8s readiness --component mongod] delay=10s timeout=2s period=3s #success=1 #failure=3
Environment Variables from:
internal-my-cluster-name-users Secret Optional: false

Hi Team,
I tried with mongod debug image as well to ge the container logs but still iam not getting any mongod container logs
Name: my-cluster-name-rs0-0
Namespace: mongodb-operator
Priority: 0
Service Account: default
Node: worker-node-2/10.151.27.43
Start Time: Thu, 22 Feb 2024 10:29:59 +0000
Labels: app.kubernetes.io/component=mongod
app.kubernetes.io/instance=my-cluster-name
app.kubernetes.io/managed-by=percona-server-mongodb-operator
app.kubernetes.io/name=percona-server-mongodb
app.kubernetes.io/part-of=percona-server-mongodb
app.kubernetes.io/replset=rs0
controller-revision-hash=my-cluster-name-rs0-df9b47794
statefulset.kubernetes.io/pod-name=my-cluster-name-rs0-0
Annotations: cni.projectcalico.org/containerID: 9b2f062055df661e25b32acc1156cb2777fabe4a5251924c0b3e756c5c6d59dc
cni.projectcalico.org/podIP: 10.233.83.188/32
cni.projectcalico.org/podIPs: 10.233.83.188/32
percona.com/ssl-hash: 252d35abcde3bcabb68773c0e683895d
Open Source Database Software Support & Services | Percona 838348b60eccbb1b223af7e530c44679
Status: Running
IP: 10.233.83.188
IPs:
IP: 10.233.83.188
Controlled By: StatefulSet/my-cluster-name-rs0
Init Containers:
mongo-init:
Container ID: containerd://b6ea84f0013b70633cae4540ad8ea2855cd55f1915077817bcdaff2597dca0eb
Image: percona/percona-server-mongodb-operator:1.15.0
Image ID: docker.io/percona/percona-server-mongodb-operator@sha256:d8a5b33db1938d42769cb5a87d34a128332a2d0302eaa6d7c860e7c4667ea3b6
Port:
Host Port:
Command:
/init-entrypoint.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 22 Feb 2024 10:30:01 +0000
Finished: Thu, 22 Feb 2024 10:30:01 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 300m
memory: 500M
Requests:
cpu: 300m
memory: 500M
Environment:
Mounts:
/data/db from mongod-data (rw)
/opt/percona from bin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bxbnt (ro)
Containers:
mongod:
Container ID: containerd://62288ff41313397682a306d38bbb718960da7424ba4547cf4a97b00464514d57
Image: percona/percona-server-mongodb:6.0.9-7-debug
Image ID: docker.io/percona/percona-server-mongodb@sha256:ef2c66575de74bbfeffb24122083e38d02a12867d414f1128f5dfa8fb381f85c
Port: 27017/TCP
Host Port: 0/TCP
Command:
/opt/percona/ps-entry.sh
Args:
–bind_ip_all
–auth
–dbpath=/data/db
–port=27017
–replSet=rs0
–storageEngine=wiredTiger
–relaxPermChecks
–sslAllowInvalidCertificates
–clusterAuthMode=x509
–shardsvr
–enableEncryption
–encryptionKeyFile=/etc/mongodb-encryption/encryption-key
–wiredTigerCacheSizeGB=0.25
–wiredTigerIndexPrefixCompression=true
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 132
Started: Thu, 22 Feb 2024 10:32:57 +0000
Finished: Thu, 22 Feb 2024 10:32:57 +0000
Ready: False
Restart Count: 3

Hi,

It seems to the liveness probe error. Kindly share the output of -

  1. kubectl get events for the namespace.
  2. Kindly provide cr.yaml configuration.
  3. If possible try to get logs of the container.

Regards,
Parag

Hi Parag,
I go the solution for the mongd Container, I have downgarded the percona-server image from 6.0.9 to 4.0.20( CPU was not supporting for Mongo 6.0.9 image Version), Now its getting deployed but my mongs pods are not coming up.
Please find the requested logs.
kubecctl logs rso0.txt (41.5 KB)
kubectl get events .txt (14.1 KB)
kubectl logs .txt (39.6 KB)

Hi Parag Did you get the time to check the logs