retryChunk got copy: context deadline exceeded

MongoDB Percona Backup for MongoDB
1

Hi,
i am seeing below messages in the logs while restoring the backups to a new cluster on Version2.2.1
Also i think the full cluster restore took ~24hrs whereas restore took 18hrs in older versions (2.0.5)

2023-11-13T14:57:27Z W [shard1ReplSet/10.80.11.0:27038] [restore/2023-11-13T09:29:21.900315695Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s
2023-11-13T14:57:27Z I [shard1ReplSet/10.80.11.0:27038] [restore/2023-11-13T09:29:21.900315695Z] session recreated, resuming download

[mongod@ip-10-80-11-188 ~]$ pbm status
Cluster:
========
shard3ReplSet:
  - shard3ReplSet/10.80.11.188:27038 [P]: pbm-agent v2.2.1 OK
configReplSet:
  - configReplSet/10.80.11.0:27039 [P]: pbm-agent v2.2.1 OK
shard1ReplSet:
  - shard1ReplSet/10.80.11.0:27038 [P]: pbm-agent v2.2.1 OK
shard2ReplSet:
  - shard2ReplSet/10.80.11.40:27038 [P]: pbm-agent v2.2.1 OK


PITR incremental backup:
========================
Status [OFF]

Currently running:
==================
(none)

Backups:
========
S3 us-east-1 s3://cm-mongo-db-shared-prod-va/percona/backup/
  Snapshots:
    2023-11-11T01:00:02Z 2.24TB <logical> [restore_to_time: 2023-11-11T12:53:21Z]
2

@aranjith0

Is the issue repeating every time ? Did you tried running that again ?

2023-11-13T14:57:27Z W [shard1ReplSet/10.80.11.0:27038] [restore/2023-11-13T09:29:21.900315695Z] retryChunk got copy: context deadline exceeded (Client.Timeout or context cancellation while reading body), try to reconnect in 0s
2023-11-13T14:57:27Z I [shard1ReplSet/10.80.11.0:27038] [restore/2023-11-13T09:29:21.900315695Z] session recreated, resuming download

Was the network stable, and health of the target cluster was fine during the activity? Did you observe anything unusual in the MongoDB or system/kernel logs?

Still to expedite the PBM process you can tweak the parallel download depending on your hardware resources and database load. To do so you need to edit the PBM configuration file as below.

restore:
   numDownloadWorkers: <int>
   maxDownloadBufferMb: <int>
   downloadChunkMb: 32

  • numDownloadWorkers - the number of workers to download data from the storage. By default, it equals to the number of CPU cores

  • maxDownloadBufferMb - the maximum size of memory buffer to store the downloaded data chunks for decompression and ordering. It is calculated as numDownloadWorkers * downloadChunkMb * 16

  • downloadChunkMb is the size of the data chunk to download (by default, 32 MB)

Reference:- Restore a backup - Percona Backup for MongoDB

Can you please share the below details of PBM for a review ?

pbm config --list
pbm logs -t0 
pbm logs --event restore

Are you trying to restore backup taken in older PBM version (2.0) or using the same version 2.2.1 for both backup/restore process ?