I’m currently trying to restore backup to freshly installed mongodb instance and get the following error while restoring collection:
[ERROR] restore/2020-09-07T10:54:20Z: restore: restore mongo dump (successes: 26676221 / fails: 0): BetaProcessing.email_log: error restoring from archive on stdin: reading bson input: error demultiplexing archive; archive io error
Error happens in the middle of restore process (so some collections restored successfully) and each time with different collection.
PBM version is the latest - 1.3.1, already happened with both pbm 1.1.3 and 1.2.0.
MongoDB version is 4.0.6. It is replica set with single node in it.
Can you explain me what the problem is? I suspect it can be related to MongoDB itself but not sure.
Thanks in advance.
Any updates here?
The error message up to the end of “error restoring from archive on stdin: reading bson input:” shows the pbm-agent is just attempting to read the file from remote storage as it does the restore. That’s the where, when.
The “what” of the problem is “error demultiplexing archive; archive io error”, and my guess is it is an AWS S3 read error. I’m not sure of that, you haven’t stated what the remote storage you’re using is, its just the “demultiplexing” makes me think of how a file stream with that object store’s HTTP API is assembled from many different, discrete http packets.
Could you please let us know which remote storage you are using?
Yes, I’m using S3 remote storage.
Hi @vvol ,
The issue can’t be reproduced with psmdb 4.0.6 and pbm 1.3.1, so can you please inform whether error happens every time (maybe, you’ve tried to restore another DB or from different backup) or only during restore from one particular backup? Was at least one restore attempt successful? If the issue is permanent, please share the preliminary DB size you’re trying to restore.
Considering the error itself, it seems to be caused by environment rather than pbm - did you try to monitor memory consumption during restore (the similar error can be also thrown when service is running out of memory)? In addition to that, network issues can also result in unsuccessful restore, have noticed any issues during backup itself?
Hi @sandra ,
This is happening across different backups. For example, I see this periodically with dev backup (one instance, around 150gb of data) and prod backup (4 shards, more than 1tb of data). As I’ve stated before this happens on different stages of restore, so I suspect this can be memory or network issue. I’ll check this out and monitor closely.
In prod I also faced with issue https://jira.percona.com/browse/PBM-485 , but it seems to be resolved now, I’ll verify.
Hi, I am also facing a similar issue, while trying to do a restore for MongoDB operator in Google Cloud. Using GCS bucket for storing backups.
Attaching the logs from backup-agent container here:-
I am trying to restore 100 GB of data here. Have observed the same issue happening every time I do a restore operation.
I am using PBM 1.3.4, Percona mongodb operator version 1.4.0, mongodb 4.2.
Currently PSMDB operator ships with an older version of PBM (some 1.1.x). And judging on the log output - that is definitely not v1.3.4. Probably you have a 1.3.4 version of PBM command-line tool but PBM-agents in operator remains an older version.
An upcoming operator release would be shipped with the latest PBM version but I’m not sure about the release date though.