# PBM Restore failed **URL:** \ **Category:** Percona Backup for MongoDB\ **Created:** [May 13, 2023, 2:59pm UTC](https://forums.percona.com/t/pbm-restore-failed/22041 "2023-05-13T14:59:25Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![aranjith0](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/aranjith0/32/13012_2.png) [@aranjith0](https://forums.percona.com/u/aranjith0)\ **Post date:** [May 13, 2023, 2:59pm UTC](https://forums.percona.com/t/pbm-restore-failed/22041/1 "2023-05-13T14:59:25Z")
Team, PBM restore to a new cluster got failed multiple time with same message. “retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body” # [mongod@ip-10-80-11-253 ~]$ pbm status Cluster: shard1ReplSet: - shard1ReplSet/10.80.11.253:27038 [P]: pbm-agent v2.1.0 OK configReplSet: - configReplSet/10.80.11.253:27039 [P]: pbm-agent v2.1.0 OK shard3ReplSet: - shard3ReplSet/10.80.10.159:27038 [P]: pbm-agent v2.1.0 OK shard2ReplSet: - shard2ReplSet/10.80.10.186:27038 [P]: pbm-agent v2.1.0 OK # PITR incremental backup: Status [OFF] # Currently running: Snapshot restore “2023-05-09T14:56:30Z”, started at 2023-05-13T05:10:22Z. Status: error. [op id: 645f1bbdfe3cefd0df864ba6] # Backups: S3 us-east-1 s3://cm-mongo-db-shared-prod-va/pbm/backup/ Snapshots: 2023-05-09T14:56:30Z 2.68TB [restore\_to\_time: 2023-05-10T01:41:15Z] 2023-05-07T01:00:03Z 2.65TB [restore\_to\_time: 2023-05-07T11:20:39Z] -===================================================== pbm logs: 2023-05-12T17:16:39Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-12T17:16:39.016551478Z] restore started 2023-05-12T17:16:39Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-12T17:16:39.016551478Z] waiting for ‘starting’ status 2023-05-12T17:16:39Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-12T17:16:39.016551478Z] moving to state running 2023-05-12T17:16:39Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-12T17:16:39.016551478Z] moving to state running 2023-05-12T17:16:40Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] moving to state running 2023-05-12T17:16:40Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-12T17:16:39.016551478Z] moving to state running 2023-05-12T17:16:51Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-12T17:16:39.016551478Z] restoring users and roles 2023-05-12T17:16:51Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-12T17:16:39.016551478Z] moving to state dumpDone 2023-05-12T18:09:11Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-12T17:16:39.016551478Z] releasing lock 2023-05-12T18:09:11Z E [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-12T17:16:39.016551478Z] mongorestore: restore mongo dump (successes: 52340979 / fails: 0): [ctr.lineage.dom.com](http://ctr.lineage.dom.com): error restoring from archive on stdin: received termination signal 2023-05-12T18:09:12Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-12T17:16:39.016551478Z] releasing lock 2023-05-12T18:09:12Z E [configReplSet/10.80.11.253:27039] [restore/2023-05-12T17:16:39.016551478Z] check cluster for restore `dumpDone`: convergeCluster: restore on the shard shard2ReplSet failed with: mongorestore: restore mongo dump (successes: 52340979 / fails: 0): [ctr.lineage.dom.com](http://ctr.lineage.dom.com): error restoring from archive on stdin: received termination signal 2023-05-12T20:02:17Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-12T20:02:17Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] session recreated, resuming download 2023-05-12T20:13:29Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-12T20:13:29Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] session recreated, resuming download 2023-05-13T00:07:23Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-13T00:07:23Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] session recreated, resuming download 2023-05-13T00:11:12Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-13T00:11:12Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] session recreated, resuming download 2023-05-13T00:12:27Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-13T00:12:27Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] session recreated, resuming download 2023-05-13T00:17:16Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] retryChunk got copy: context deadline ---
**Author:** ![Dmytro\_Zghoba](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dmytro_zghoba/32/4721_2.png) [@Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba)\ **Post date:** [May 18, 2023, 6:34am UTC](https://forums.percona.com/t/pbm-restore-failed/22041/2 "2023-05-18T06:34:05Z")
Hi @aranjith0 , > [@aranjith0](#): > > 2023-05-13T00:11:12Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-12T17:16:39.016551478Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s It seems the `shard1ReplSet` had an issue with downloading the backup. Could you please test network performance from the node to storage? ---
**Author:** ![aranjith0](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/aranjith0/32/13012_2.png) [@aranjith0](https://forums.percona.com/u/aranjith0)\ **Post date:** [May 18, 2023, 9:50am UTC](https://forums.percona.com/t/pbm-restore-failed/22041/3 "2023-05-18T09:50:55Z")
HI @ [Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba), Random 10G file created on s3 bucket and tried to download on all 3 nodes. I don’t see any much difference in performance. node-1 (shard1ReplSet ): [mongod@ip-10-80-11-253 tmp]$ time aws s3 cp s3://cm-mongo-db-shared-prod-va/data.img1 /tmp/ download: s3://cm-mongo-db-shared-prod-va/data.img1 to ./data.img1 real 0m4.422s user 0m4.927s sys 0m6.647s * * * node-2 (shard2ReplSet ): [mongod@ip-10-80-10-186 ~]$ time aws s3 cp s3://cm-mongo-db-shared-prod-va/data.img1 /tmp/ download: s3://cm-mongo-db-shared-prod-va/data.img1 to …/…/tmp/data.img1 real 0m4.820s user 0m4.617s sys 0m6.915s * * * node-3 (shard3ReplSet ): [mongod@ip-10-80-10-159 ~]$ time aws s3 cp s3://cm-mongo-db-shared-prod-va/data.img1 /tmp/ download: s3://cm-mongo-db-shared-prod-va/data.img1 to …/…/tmp/data.img1 real 0m4.834s user 0m4.596s sys 0m6.879s ---
**Author:** ![Dmytro\_Zghoba](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dmytro_zghoba/32/4721_2.png) [@Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba)\ **Post date:** [May 18, 2023, 10:51am UTC](https://forums.percona.com/t/pbm-restore-failed/22041/4 "2023-05-18T10:51:46Z")
Does the restore always fail with such error? Did you manage to restore the backup successfully at least once? ---
**Author:** ![aranjith0](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/aranjith0/32/13012_2.png) [@aranjith0](https://forums.percona.com/u/aranjith0)\ **Post date:** [May 18, 2023, 10:57am UTC](https://forums.percona.com/t/pbm-restore-failed/22041/5 "2023-05-18T10:57:40Z")
Yes- I tried once again yesterday and it took 20 hrs for a FULL restore and seen same messages in that. I had done a restore successfully with version 1.8.1 without any issues ( it took 13 hrs). [mongod@ip-10-80-11-253 ~]$ pbm logs -sD -t0 -i 646469f41ff7942aef24ab9b 2023-05-17T05:45:24Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] backup: 2023-05-16T15:30:02Z 2023-05-17T05:45:24Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] restore started 2023-05-17T05:45:24Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] waiting for ‘starting’ status 2023-05-17T05:45:24Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] backup: 2023-05-16T15:30:02Z 2023-05-17T05:45:24Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] restore started 2023-05-17T05:45:24Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] waiting for ‘starting’ status 2023-05-17T05:45:24Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] backup: 2023-05-16T15:30:02Z 2023-05-17T05:45:24Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] restore started 2023-05-17T05:45:24Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] waiting for ‘starting’ status 2023-05-17T05:45:24Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] backup: 2023-05-16T15:30:02Z 2023-05-17T05:45:24Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] restore started 2023-05-17T05:45:24Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] waiting for ‘starting’ status 2023-05-17T05:45:25Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] moving to state running 2023-05-17T05:45:25Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] moving to state running 2023-05-17T05:45:25Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] moving to state running 2023-05-17T05:45:25Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] moving to state running 2023-05-17T05:45:36Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] restoring users and roles 2023-05-17T05:45:36Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] moving to state dumpDone 2023-05-17T13:38:59Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-17T13:38:59Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] session recreated, resuming download 2023-05-17T13:41:40Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-17T13:41:40Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] session recreated, resuming download 2023-05-17T19:19:01Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] restoring users and roles 2023-05-17T19:19:01Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] moving to state dumpDone 2023-05-17T21:40:22Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] restoring users and roles 2023-05-17T21:40:22Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] moving to state dumpDone 2023-05-18T01:22:46Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] starting oplog replay 2023-05-18T01:22:46Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] starting sharded txn sync 2023-05-18T01:22:46Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] + applying {shard3ReplSet 2023-05-16T15:30:02Z/shard3ReplSet/local.oplog.rs.bson.s2 s2 {1684251002 20} {1684287522 4} 0} 2023-05-18T01:22:46Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] restoring users and roles 2023-05-18T01:22:46Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] moving to state dumpDone 2023-05-18T01:22:46Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] starting oplog replay 2023-05-18T01:22:46Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] starting sharded txn sync 2023-05-18T01:22:46Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] + applying {shard2ReplSet 2023-05-16T15:30:02Z/shard2ReplSet/local.oplog.rs.bson.s2 s2 {1684251002 20} {1684287522 4} 0} 2023-05-18T01:22:47Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] starting oplog replay 2023-05-18T01:22:47Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] starting sharded txn sync 2023-05-18T01:22:47Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] + applying {shard1ReplSet 2023-05-16T15:30:02Z/shard1ReplSet/local.oplog.rs.bson.s2 s2 {1684251002 20} {1684287522 4} 0} 2023-05-18T01:22:47Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] starting oplog replay 2023-05-18T01:22:47Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] starting sharded txn sync 2023-05-18T01:22:47Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] + applying {configReplSet 2023-05-16T15:30:02Z/configReplSet/local.oplog.rs.bson.s2 s2 {1684251002 20} {1684287522 4} 0} 2023-05-18T01:22:49Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] oplog replay finished on {1684287522 1} 2023-05-18T01:22:49Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] exit waitingTxnChecker 2023-05-18T01:22:49Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] exit distTxnChecker 2023-05-18T01:24:17Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [prepare] {0 0} 2023-05-18T01:24:17Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [commit] {1684260886 543} 2023-05-18T01:24:22Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [prepare] {0 0} 2023-05-18T01:24:22Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [commit] {1684260886 543} 2023-05-18T01:27:00Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] 2023-05-18T01:27:01Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] 2023-05-18T01:27:01Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [prepare] {0 0} 2023-05-18T01:27:01Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [commit] {1684260886 543} 2023-05-18T01:27:02Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] 2023-05-18T01:27:02Z W [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-18T01:27:02Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] session recreated, resuming download 2023-05-18T01:27:03Z W [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-18T01:27:03Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] session recreated, resuming download 2023-05-18T01:27:35Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: \ [prepare] {0 0} 2023-05-18T01:27:35Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: \ [commit] {1684264714 115739} 2023-05-18T01:29:45Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: \ [prepare] {0 0} 2023-05-18T01:29:45Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: \ [commit] {1684264714 115739} 2023-05-18T01:29:55Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: \ [prepare] {0 0} 2023-05-18T01:29:55Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] \ 2023-05-18T01:29:55Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: \ [commit] {1684264714 115739} 2023-05-18T01:29:56Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] \ 2023-05-18T01:29:57Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] \ 2023-05-18T01:30:06Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [prepare] {0 0} 2023-05-18T01:30:06Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [commit] {1684264750 162} 2023-05-18T01:30:54Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [prepare] {0 0} 2023-05-18T01:30:54Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [commit] {1684264750 162} 2023-05-18T01:30:56Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [prepare] {0 0} 2023-05-18T01:30:56Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] found dist txn: [commit] {1684264750 162} 2023-05-18T01:30:56Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] 2023-05-18T01:30:57Z W [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-18T01:30:57Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] session recreated, resuming download 2023-05-18T01:30:57Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] 2023-05-18T01:30:58Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] txn converged to [commit] 2023-05-18T01:30:58Z W [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-18T01:30:58Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] session recreated, resuming download 2023-05-18T01:30:59Z W [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s 2023-05-18T01:30:59Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] session recreated, resuming download 2023-05-18T01:34:45Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] oplog replay finished on {1684287521 2} 2023-05-18T01:34:45Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] exit waitingTxnChecker 2023-05-18T01:34:45Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] exit distTxnChecker 2023-05-18T01:34:45Z I [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] restore finished successfully 2023-05-18T01:34:45Z D [shard3ReplSet/10.80.10.159:27038] [restore/2023-05-17T05:45:24.629271292Z] releasing lock 2023-05-18T01:35:16Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] oplog replay finished on {1684287522 1} 2023-05-18T01:35:16Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] exit waitingTxnChecker 2023-05-18T01:35:16Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] exit distTxnChecker 2023-05-18T01:35:16Z I [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] restore finished successfully 2023-05-18T01:35:16Z D [shard2ReplSet/10.80.10.186:27038] [restore/2023-05-17T05:45:24.629271292Z] releasing lock 2023-05-18T01:35:35Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] oplog replay finished on {1684287522 4} 2023-05-18T01:35:35Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] exit waitingTxnChecker 2023-05-18T01:35:35Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] exit distTxnChecker 2023-05-18T01:35:35Z I [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] restore finished successfully 2023-05-18T01:35:35Z D [shard1ReplSet/10.80.11.253:27038] [restore/2023-05-17T05:45:24.629271292Z] releasing lock 2023-05-18T01:35:35Z I [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] restore finished successfully 2023-05-18T01:35:35Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] epoch set to {1684373735 11} 2023-05-18T01:35:35Z D [configReplSet/10.80.11.253:27039] [restore/2023-05-17T05:45:24.629271292Z] releasing lock