# Backup failing after update to 2.0.4 **URL:** \ **Category:** Percona Backup for MongoDB\ **Tags:** closed-no-reply\ **Created:** [February 23, 2023, 8:50am UTC](https://forums.percona.com/t/backup-failing-after-update-to-2-0-4/20248 "2023-02-23T08:50:46Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![aranjith0](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/aranjith0/32/13012_2.png) [@aranjith0](https://forums.percona.com/u/aranjith0)\ **Post date:** [February 23, 2023, 8:50am UTC](https://forums.percona.com/t/backup-failing-after-update-to-2-0-4/20248/1 "2023-02-23T08:50:46Z")
# [mongod@ip-10-80-10-113 ~]$ pbm status Cluster: configReplSet: - configReplSet/pmgo-pl204.int.compumark.com:27019 [S]: pbm-agent v2.0.4 OK - configReplSet/pmgo-pl202.int.compumark.com:27019 [P]: pbm-agent v2.0.4 OK - configReplSet/pmgo-pl206.int.compumark.com:27029 [S]: pbm-agent v2.0.4 OK shard3ReplSet: - shard3ReplSet/pmgo-pl206.int.compumark.com:27018 [P]: pbm-agent v2.0.4 OK - shard3ReplSet/pmgo-pl205.int.compumark.com:27018 [S]: pbm-agent v2.0.4 OK - shard3ReplSet/pmgo-pl201.int.compumark.com:27028 [S]: pbm-agent v2.0.4 OK shard1ReplSet: - shard1ReplSet/pmgo-pl203.int.compumark.com:27018 [P]: pbm-agent v2.0.4 OK - shard1ReplSet/pmgo-pl201.int.compumark.com:27018 [S]: pbm-agent v2.0.4 OK - shard1ReplSet/pmgo-pl202.int.compumark.com:27028 [S]: pbm-agent v2.0.4 OK shard2ReplSet: - shard2ReplSet/pmgo-pl204.int.compumark.com:27018 [P]: pbm-agent v2.0.4 OK - shard2ReplSet/pmgo-pl202.int.compumark.com:27018 [S]: pbm-agent v2.0.4 OK - shard2ReplSet/pmgo-pl205.int.compumark.com:27028 [S]: pbm-agent v2.0.4 OK # PITR incremental backup: Status [OFF] # Currently running: (none) # Backups: S3 us-east-1 s3://cm-mongo-db-shared-prod-va/pbm/backup/ Snapshots: _**2023-02-23T01:02:02Z 0.00B [ERROR: couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout] [2023-02-23T01:02:38Z]**_ 2023-02-20T15:41:10Z 2.31TB [restore\_to\_time: 2023-02-21T03:47:57Z] 2023-02-15T12:06:25Z 2.30TB [restore\_to\_time: 2023-02-16T00:45:43Z] 2023-01-09T06:34:38Z 62.12KB \ [restore\_to\_time: 2023-01-09T06:34:44Z] 2023-01-06T13:04:46Z 14.88GB \ [restore\_to\_time: 2023-01-06T13:23:57Z] pbm logs: [mongod@ip-10-80-10-113 ~]$ **pbm logs -t 1000 -s D --event=backup/2023-02-23T01:02:02Z** 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] init backup meta 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination list for shard3ReplSet: [[[pmgo-pl201.int.compumark.com:27028](http://pmgo-pl201.int.compumark.com:27028)] [[pmgo-pl205.int.compumark.com:27018](http://pmgo-pl205.int.compumark.com:27018)] [[pmgo-pl206.int.compumark.com:27018](http://pmgo-pl206.int.compumark.com:27018)]] 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination list for configReplSet: [[[pmgo-pl204.int.compumark.com:27019](http://pmgo-pl204.int.compumark.com:27019) [pmgo-pl206.int.compumark.com:27029](http://pmgo-pl206.int.compumark.com:27029)] [[pmgo-pl202.int.compumark.com:27019](http://pmgo-pl202.int.compumark.com:27019)]] 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination list for shard1ReplSet: [[[pmgo-pl202.int.compumark.com:27028](http://pmgo-pl202.int.compumark.com:27028)] [[pmgo-pl201.int.compumark.com:27018](http://pmgo-pl201.int.compumark.com:27018)] [[pmgo-pl203.int.compumark.com:27018](http://pmgo-pl203.int.compumark.com:27018)]] 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination list for shard2ReplSet: [[[pmgo-pl205.int.compumark.com:27028](http://pmgo-pl205.int.compumark.com:27028)] [[pmgo-pl202.int.compumark.com:27018](http://pmgo-pl202.int.compumark.com:27018)] [[pmgo-pl204.int.compumark.com:27018](http://pmgo-pl204.int.compumark.com:27018)]] 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination configReplSet, set candidates [[pmgo-pl204.int.compumark.com:27019](http://pmgo-pl204.int.compumark.com:27019) [pmgo-pl206.int.compumark.com:27029](http://pmgo-pl206.int.compumark.com:27029)] 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination shard2ReplSet, set candidates [[pmgo-pl205.int.compumark.com:27028](http://pmgo-pl205.int.compumark.com:27028)] 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination shard3ReplSet, set candidates [[pmgo-pl201.int.compumark.com:27028](http://pmgo-pl201.int.compumark.com:27028)] 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination shard1ReplSet, set candidates [[pmgo-pl202.int.compumark.com:27028](http://pmgo-pl202.int.compumark.com:27028)] 2023-02-23T01:02:03Z I [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup started 2023-02-23T01:02:03Z D [configReplSet/pmgo-pl206.int.compumark.com:27029] [backup/2023-02-23T01:02:02Z] skip: lock not acquired 2023-02-23T01:02:03Z I [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] backup started 2023-02-23T01:02:03Z I [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup started 2023-02-23T01:02:03Z D [shard2ReplSet/pmgo-pl202.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node 2023-02-23T01:02:04Z I [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup started 2023-02-23T01:02:04Z D [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] waiting for balancer off 2023-02-23T01:02:04Z D [shard3ReplSet/pmgo-pl205.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node 2023-02-23T01:02:04Z D [shard2ReplSet/pmgo-pl204.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node 2023-02-23T01:02:04Z D [shard3ReplSet/pmgo-pl206.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node 2023-02-23T01:02:04Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node 2023-02-23T01:02:04Z D [shard1ReplSet/pmgo-pl201.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node 2023-02-23T01:02:04Z D [shard1ReplSet/pmgo-pl203.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node 2023-02-23T01:02:04Z D [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] balancer status: off 2023-02-23T01:02:08Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] bcp nomination: configReplSet won by [pmgo-pl204.int.compumark.com:27019](http://pmgo-pl204.int.compumark.com:27019) 2023-02-23T01:02:08Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] bcp nomination: shard2ReplSet won by [pmgo-pl205.int.compumark.com:27028](http://pmgo-pl205.int.compumark.com:27028) 2023-02-23T01:02:08Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] bcp nomination: shard3ReplSet won by [pmgo-pl201.int.compumark.com:27028](http://pmgo-pl201.int.compumark.com:27028) 2023-02-23T01:02:08Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] bcp nomination: shard1ReplSet won by [pmgo-pl202.int.compumark.com:27028](http://pmgo-pl202.int.compumark.com:27028) 2023-02-23T01:02:38Z I [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] mark RS as error `couldn't get response from all shards: convergeClusterWithTimeout: reached converge timeout`: 2023-02-23T01:02:38Z I [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] mark backup as error `couldn't get response from all shards: convergeClusterWithTimeout: reached converge timeout`: 2023-02-23T01:02:38Z D [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] set balancer on 2023-02-23T01:02:38Z E [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] backup: couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout 2023-02-23T01:02:38Z D [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] releasing lock 2023-02-23T01:02:39Z I [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] mark RS as error `waiting for running: cluster failed: `: 2023-02-23T01:02:39Z D [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] set balancer on 2023-02-23T01:02:39Z E [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup: waiting for running: cluster failed: 2023-02-23T01:02:39Z D [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] releasing lock 2023-02-23T01:02:39Z I [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] mark RS as error `waiting for running: cluster failed: `: 2023-02-23T01:02:39Z D [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] set balancer on 2023-02-23T01:02:39Z E [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup: waiting for running: cluster failed: 2023-02-23T01:02:39Z D [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] releasing lock 2023-02-23T01:03:02Z I [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] mark RS as error `waiting for running: cluster failed: `: 2023-02-23T01:03:02Z D [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] set balancer on 2023-02-23T01:03:02Z E [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup: waiting for running: cluster failed: 2023-02-23T01:03:02Z D [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] releasing lock Please help me to troubleshoot this issue.