Not the answer you need?
Register and ask your own question!

Cannot take backup on sharded cluster

vvolvvol Current User Role Contributor
Cannot take backup via pbm on sharded mongodb cluster, works ok with single-node mongodb.

After trying to run pbm backup the following error occurred: Failed with "couldn't get response from all shards: convergeClusterWithTimeout: reached converge timeout". Mongodb uri is set to config set replicaset like 'mongodb://pbmuser:[email protected]:27019,cs2:27019,cs3:27019/?authSource=admin&replicaSet=csReplSet'. Credentials are valid, pbm-agent is present on all mongod nodes accross all shards + config set nodes.

In logs I see messages like:
Jun  4 08:46:43 pbm-agent[15215]: 2020/06/04 08:46:43 Got command resyncBcpList
Jun  4 08:46:43 pbm-agent[15215]: 2020/06/04 08:46:43 [INFO] resync_list: operation has been scheduled on another replset node
Jun  4 08:47:48 pbm-agent[15215]: 2020/06/04 08:47:48 Got command backup 2020-06-04T08:47:47Z
Jun  4 08:47:48 pbm-agent[15215]: 2020/06/04 08:47:48 Backup has been scheduled on another replset node
Jun  5 03:30:01 pbm-agent[15215]: 2020/06/05 03:30:01 Got command backup 2020-06-05T03:30:01Z
Jun  5 03:30:01 pbm-agent[15215]: 2020/06/05 03:30:01 Backup has been scheduled on another replset node

Jun  5 08:00:36 pbm-agent[18799]: 2020/06/05 08:00:36 Got command backup 2020-06-05T08:00:36Z
Jun  5 08:00:36 pbm-agent[18799]: 2020/06/05 08:00:36 Backup 2020-06-05T08:00:36Z started on node csReplSet/10.1.1.33:27019
Jun  5 08:00:52 pbm-agent[18799]: 2020/06/05 08:00:52 Mark backup as failed `couldn't get response from all shards: convergeClusterWithTimeout: reached converge timeout`: <nil>
Jun  5 08:00:52 pbm-agent[18799]: 2020/06/05 08:00:52 [ERROR] backup: couldn't get response from all shards: convergeClusterWithTimeout: reached converge timeout

How can I enable debug logs to see more? Or maybe you know the reason?

MongoDB version - 4.0.6
PBM version - 1.1.3

Thanks in advance.

Comments

  • AndrewPogrebnoiAndrewPogrebnoi Percona Staff Role
    Hi @vvol

    Each pbm-agent processes should connect to their localhost mongod with a standalone type of connection. So agent's URIs should be in format "mongodb://pbmuser:[email protected]cs1:27019", "mongodb://pbmuser:[email protected]cs2:27019", "mongodb://pbmuser:[email protected]rs0:27019" and so on.  Since each agent should serve only one node.

    Now we have a bit confusing example in the documentation. I think we will fix it.

    Cheers!
  • vvolvvol Current User Role Contributor
    Thanks for reply! Backup was successful after I had changed URI.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.