Cluster with backup enabled CrashLoopBackoff only rs0-1

MongoDB Percona Operator for MongoDB
1

Hello.
I don’t know if this is a bug or I’m missing something but every mongodb cluster I make backup container failed with “Unable to reach primary for set rs0”.
cluster size 3
arbiter.enabled false
shard false

All the nodes of the mongo cluster are working correctly but only rs0-1 backup container log:

connect@src/mongo/shell/mongo.js:353:17
@(connect):3:6
2021-02-16T12:44:05.298+0000 F - [main] exception: connect failed
2021-02-16T12:44:05.298+0000 E - [main] exiting with code 1
+ exit_status=0
++ grep -E ‘^([0-9]+)$’ /tmp/tmp.exfxQxes0t
+ rs_size=
+ [[ 0 == 0 ]]
+ [[ ‘’ -ge 1 ]]
+ sleep 40
+ for i in ‘{1…10}’
+ ‘[’ ‘’ ‘]’
+ mongo ‘mongodb://backup:redacted@localhost:27017/?replicaSet=rs0’ ‘–eval=
(db.isMaster().hosts).length’ --quiet
+ tee /tmp/tmp.exfxQxes0t
2021-02-16T12:44:45.506+0000 I NETWORK [js] Starting new replica set monitor for
rs0/localhost:27017
2021-02-16T12:44:45.506+0000 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to
localhost:27017
2021-02-16T12:44:45.507+0000 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach
primary for set rs0
2021-02-16T12:44:45.507+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any
nodes for set rs0. Please check network connectivity and the status of the set. This has happened
for 1 checks in a row.

thanks in advance!

1 Like
2

Hello @gasenov ,

thank you for the request.
I have tried to do a backup and restore and it works fine for me, but could you please share more details about your installation and steps you perform?

  1. Please share the complete cr.yaml (it does not contain any sensitive data)
  2. How do you take the backup? Is it scheduled? If it is manual - please share the backup.yaml manifest.
  3. Is there anything specific about the installation? Kubernetes version, storage type, etc.?
1 Like
3

The issue is not during backup/restore. It just failed to start the backup container. Or maybe it start doing backups from the startup ??!?
Actually it starts but because of the backup container (in the replica pod) timeouts the pod is shat down periodically.
In a matter of fact after some time like the next day it was connected. When i watched it for hours trying to connect and couldn’t. Strange

1 Like
4

@gasenov yeah, this I cannot reproduce as well.
Could you please share your cr.yaml?

1 Like