Check storage connecion: unable to get storage: get config: get: mongo: no documents in result

MongoDB Percona Operator for MongoDB
21

No, it should not be added anywhere and should just work.

The error

couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout

indicates that on some replicaset the backup was not started. How many replicasets do you have? Could you check the logs of backup-agent container in them and see if something fails?

1 Like
22

I’ve 3 replicasets;
But sometimes there is always liveness and readiness probe failed in all the pods rs0, cfg and mongo-db-mongos set of pods and mongo-db-mongos pods will not be in Ready state sometimes and I’ve to restart the deployments;

1 Like
23

Do you know why probes are failing? Backup can fail if the cluster is unstable.

1 Like
24

@spronin

image
image735×152 4.15 KB

When I get events it shows
image
image726×174 6.18 KB

1 Like
25

Looking at this - something is clearly not right.
The probes are failing. You need to troubleshoot what is causing these failures:

  1. Check the k8s messages (start with kubectl describe pod XXX)
  2. Check pod/container logs
  3. Check if your k8s cluster is healthy in general (see if nodes are failing, monitoring, etc)
1 Like
26
  1. When I describe cfg and rs0 pods I get liveness probe failed :
Events:
  Type     Reason     Age                   From     Message
  ----     ------     ----                  ----     -------
  Warning  Unhealthy  5m37s (x207 over 9h)  kubelet  Liveness probe failed:

And for mongos I get:

Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  2m10s (x5925 over 9h)  kubelet  Readiness probe failed:
  1. When I check logs; There is no error logs in mongos container
    But in backup-agent container; It shows
2021-06-07T03:41:19.000+0000 E [agentCheckup] check node connecion: server selection error: server selection timeout
current topology: Type: Single

logs of mongo-init container


++ id -u
++ id -g
+ install -o 99 -g 99 -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh
++ id -u
++ id -g
+ install -o 99 -g 99 -m 0755 -D /mongodb-healthcheck /data/db/mongodb-healthcheck
  1. And k8s cluster is healthy in general; All the nodes in the k8s cluster are in ready state
1 Like
27

You can run minio without https by setting requestAutoCert to false.

certificate:
      ## Use this field to provide one or more external CA certificates. This is used by MinIO
      ## to verify TLS connections with other applications:
      ## https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret
      externalCaCertSecret: {}
      ## Use this field to provide a list of Secrets with external certificates. This can be used to to configure
      ## TLS for MinIO Tenant pods. Create secrets as explained here:
      ## https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret
      externalCertSecret: {}
      ## Enable automatic Kubernetes based certificate generation and signing as explained in
      ## https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster
      requestAutoCert: true
      ## This field is used only when "requestAutoCert" is set to true. Use this field to set CommonName
      ## for the auto-generated certificate. Internal DNS name for the pod will be used if CommonName is
      ## not provided. DNS name format is *.minio.default.svc.cluster.local
      certConfig: {}
1 Like