[mongod@ip-10-80-10-113 ~]$ pbm status
Cluster:
configReplSet:
- configReplSet/pmgo-pl204.int.compumark.com:27019 [S]: pbm-agent v2.0.4 OK
- configReplSet/pmgo-pl202.int.compumark.com:27019 [P]: pbm-agent v2.0.4 OK
- configReplSet/pmgo-pl206.int.compumark.com:27029 [S]: pbm-agent v2.0.4 OK
shard3ReplSet: - shard3ReplSet/pmgo-pl206.int.compumark.com:27018 [P]: pbm-agent v2.0.4 OK
- shard3ReplSet/pmgo-pl205.int.compumark.com:27018 [S]: pbm-agent v2.0.4 OK
- shard3ReplSet/pmgo-pl201.int.compumark.com:27028 [S]: pbm-agent v2.0.4 OK
shard1ReplSet: - shard1ReplSet/pmgo-pl203.int.compumark.com:27018 [P]: pbm-agent v2.0.4 OK
- shard1ReplSet/pmgo-pl201.int.compumark.com:27018 [S]: pbm-agent v2.0.4 OK
- shard1ReplSet/pmgo-pl202.int.compumark.com:27028 [S]: pbm-agent v2.0.4 OK
shard2ReplSet: - shard2ReplSet/pmgo-pl204.int.compumark.com:27018 [P]: pbm-agent v2.0.4 OK
- shard2ReplSet/pmgo-pl202.int.compumark.com:27018 [S]: pbm-agent v2.0.4 OK
- shard2ReplSet/pmgo-pl205.int.compumark.com:27028 [S]: pbm-agent v2.0.4 OK
PITR incremental backup:
Status [OFF]
Currently running:
(none)
Backups:
S3 us-east-1 s3://cm-mongo-db-shared-prod-va/pbm/backup/
Snapshots:
2023-02-23T01:02:02Z 0.00B [ERROR: couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout] [2023-02-23T01:02:38Z]
2023-02-20T15:41:10Z 2.31TB [restore_to_time: 2023-02-21T03:47:57Z]
2023-02-15T12:06:25Z 2.30TB [restore_to_time: 2023-02-16T00:45:43Z]
2023-01-09T06:34:38Z 62.12KB <logical, selective> [restore_to_time: 2023-01-09T06:34:44Z]
2023-01-06T13:04:46Z 14.88GB <logical, selective> [restore_to_time: 2023-01-06T13:23:57Z]
pbm logs:
[mongod@ip-10-80-10-113 ~]$ pbm logs -t 1000 -s D --event=backup/2023-02-23T01:02:02Z
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] init backup meta
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination list for shard3ReplSet: [[pmgo-pl201.int.compumark.com:27028] [pmgo-pl205.int.compumark.com:27018] [pmgo-pl206.int.compumark.com:27018]]
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination list for configReplSet: [[pmgo-pl204.int.compumark.com:27019 pmgo-pl206.int.compumark.com:27029] [pmgo-pl202.int.compumark.com:27019]]
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination list for shard1ReplSet: [[pmgo-pl202.int.compumark.com:27028] [pmgo-pl201.int.compumark.com:27018] [pmgo-pl203.int.compumark.com:27018]]
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination list for shard2ReplSet: [[pmgo-pl205.int.compumark.com:27028] [pmgo-pl202.int.compumark.com:27018] [pmgo-pl204.int.compumark.com:27018]]
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination configReplSet, set candidates [pmgo-pl204.int.compumark.com:27019 pmgo-pl206.int.compumark.com:27029]
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination shard2ReplSet, set candidates [pmgo-pl205.int.compumark.com:27028]
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination shard3ReplSet, set candidates [pmgo-pl201.int.compumark.com:27028]
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] nomination shard1ReplSet, set candidates [pmgo-pl202.int.compumark.com:27028]
2023-02-23T01:02:03Z I [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup started
2023-02-23T01:02:03Z D [configReplSet/pmgo-pl206.int.compumark.com:27029] [backup/2023-02-23T01:02:02Z] skip: lock not acquired
2023-02-23T01:02:03Z I [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] backup started
2023-02-23T01:02:03Z I [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup started
2023-02-23T01:02:03Z D [shard2ReplSet/pmgo-pl202.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node
2023-02-23T01:02:04Z I [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup started
2023-02-23T01:02:04Z D [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] waiting for balancer off
2023-02-23T01:02:04Z D [shard3ReplSet/pmgo-pl205.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node
2023-02-23T01:02:04Z D [shard2ReplSet/pmgo-pl204.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node
2023-02-23T01:02:04Z D [shard3ReplSet/pmgo-pl206.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node
2023-02-23T01:02:04Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node
2023-02-23T01:02:04Z D [shard1ReplSet/pmgo-pl201.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node
2023-02-23T01:02:04Z D [shard1ReplSet/pmgo-pl203.int.compumark.com:27018] [backup/2023-02-23T01:02:02Z] skip after nomination, probably started by another node
2023-02-23T01:02:04Z D [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] balancer status: off
2023-02-23T01:02:08Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] bcp nomination: configReplSet won by pmgo-pl204.int.compumark.com:27019
2023-02-23T01:02:08Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] bcp nomination: shard2ReplSet won by pmgo-pl205.int.compumark.com:27028
2023-02-23T01:02:08Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] bcp nomination: shard3ReplSet won by pmgo-pl201.int.compumark.com:27028
2023-02-23T01:02:08Z D [configReplSet/pmgo-pl202.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] bcp nomination: shard1ReplSet won by pmgo-pl202.int.compumark.com:27028
2023-02-23T01:02:38Z I [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] mark RS as error couldn't get response from all shards: convergeClusterWithTimeout: reached converge timeout:
2023-02-23T01:02:38Z I [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] mark backup as error couldn't get response from all shards: convergeClusterWithTimeout: reached converge timeout:
2023-02-23T01:02:38Z D [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] set balancer on
2023-02-23T01:02:38Z E [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] backup: couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout
2023-02-23T01:02:38Z D [configReplSet/pmgo-pl204.int.compumark.com:27019] [backup/2023-02-23T01:02:02Z] releasing lock
2023-02-23T01:02:39Z I [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] mark RS as error waiting for running: cluster failed: <nil>:
2023-02-23T01:02:39Z D [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] set balancer on
2023-02-23T01:02:39Z E [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup: waiting for running: cluster failed:
2023-02-23T01:02:39Z D [shard2ReplSet/pmgo-pl205.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] releasing lock
2023-02-23T01:02:39Z I [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] mark RS as error waiting for running: cluster failed: <nil>:
2023-02-23T01:02:39Z D [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] set balancer on
2023-02-23T01:02:39Z E [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup: waiting for running: cluster failed:
2023-02-23T01:02:39Z D [shard3ReplSet/pmgo-pl201.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] releasing lock
2023-02-23T01:03:02Z I [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] mark RS as error waiting for running: cluster failed: <nil>:
2023-02-23T01:03:02Z D [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] set balancer on
2023-02-23T01:03:02Z E [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] backup: waiting for running: cluster failed:
2023-02-23T01:03:02Z D [shard1ReplSet/pmgo-pl202.int.compumark.com:27028] [backup/2023-02-23T01:02:02Z] releasing lock
Please help me to troubleshoot this issue.