Not the answer you need?
Register and ask your own question!

Why restore failed with "error demultiplexing archive" error?

vvolvvol Current User Role Contributor
Hello,
I'm currently trying to restore backup to freshly installed mongodb instance and get the following error while restoring collection:

[ERROR] restore/2020-09-07T10:54:20Z: restore: restore mongo dump (successes: 26676221 / fails: 0): BetaProcessing.email_log: error restoring from archive on stdin: reading bson input: error demultiplexing archive; archive io error

Error happens in the middle of restore process (so some collections restored successfully) and each time with different collection.
PBM version is the latest - 1.3.1, already happened with both pbm 1.1.3 and 1.2.0.
MongoDB version is 4.0.6. It is replica set with single node in it.

Can you explain me what the problem is? I suspect it can be related to MongoDB itself but not sure.
Thanks in advance.

Answers

  • vvolvvol Current User Role Contributor

    Any updates here?

  • Akira KuroganeAkira Kurogane Percona Percona Staff Role

    Hi.

    The error message up to the end of "error restoring from archive on stdin: reading bson input:" shows the pbm-agent is just attempting to read the file from remote storage as it does the restore. That's the where, when.

    The "what" of the problem is "error demultiplexing archive; archive io error", and my guess is it is an AWS S3 read error. I'm not sure of that, you haven't stated what the remote storage you're using is, its just the "demultiplexing" makes me think of how a file stream with that object store's HTTP API is assembled from many different, discrete http packets.

    Could you please let us know which remote storage you are using?

  • vvolvvol Current User Role Contributor

    Yes, I'm using S3 remote storage.

  • sandrasandra Percona Staff Role

    Hi @vvol ,

    The issue can't be reproduced with psmdb 4.0.6 and pbm 1.3.1, so can you please inform whether error happens every time (maybe, you've tried to restore another DB or from different backup) or only during restore from one particular backup? Was at least one restore attempt successful? If the issue is permanent, please share the preliminary DB size you're trying to restore.

    Considering the error itself, it seems to be caused by environment rather than pbm - did you try to monitor memory consumption during restore (the similar error can be also thrown when service is running out of memory)? In addition to that, network issues can also result in unsuccessful restore, have noticed any issues during backup itself?

  • vvolvvol Current User Role Contributor
    edited September 24

    Hi @sandra ,

    This is happening across different backups. For example, I see this periodically with dev backup (one instance, around 150gb of data) and prod backup (4 shards, more than 1tb of data). As I've stated before this happens on different stages of restore, so I suspect this can be memory or network issue. I'll check this out and monitor closely.

    In prod I also faced with issue https://jira.percona.com/browse/PBM-485 , but it seems to be resolved now, I'll verify.

Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.