# Cannot restore incremetal backup

**URL:** <https://forums.percona.com/t/cannot-restore-incremetal-backup/25078>\
**Category:** Percona Backup for MongoDB\
**Tags:** closed-no-reply\
**Created:** [September 11, 2023, 7:07am UTC](https://forums.percona.com/t/cannot-restore-incremetal-backup/25078 "2023-09-11T07:07:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pjotz](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pjotz/32/12032_2.png) [@Pjotz](https://forums.percona.com/u/Pjotz)\
**Post date:** [September 11, 2023, 7:07am UTC](https://forums.percona.com/t/cannot-restore-incremetal-backup/25078/1 "2023-09-11T07:07:07Z")

</div>

Hi, im trying to restore an incremental backup, all instances are running in docker containers.

I added the pbm-agent & to the mongodb entrypoint, so it now starts when container starts up. But…

i do a incremental backup, everything works perfect. then i delete one of my collections, and will try and run a restore.  
The following below is the whole restore process, and logs, after it tried to restore, the config servers studently loosing connection, have no idea why, and when i check my db. the collection is not restored.

Hope you can see what happens from the logs 🫤

# [mongodb@mongodb\_content\_cfg\_server1 percona]$pbm status

Cluster:

mongo\_data\_rs2:

- mongo\_data\_rs2/mongodb\_content\_rs2n1:27017 [P]: pbm-agent v2.2.1 OK  
mongo\_data\_rs3:
- mongo\_data\_rs3/mongodb\_content\_rs3n1:27017 [P]: pbm-agent v2.2.1 OK  
mongo\_data\_rs1:
- mongo\_data\_rs1/mongodb\_content\_rs1n1:27017 [P]: pbm-agent v2.2.1 OK  
mongo\_conf:
- mongo\_conf/mongodb\_content\_cfg\_server1:27017 [P]: pbm-agent v2.2.1 OK
- mongo\_conf/mongodb\_content\_cfg\_server2:27017 [S]: pbm-agent v2.2.1 OK
- mongo\_conf/mongodb\_content\_cfg\_server3:27017 [S]: pbm-agent v2.2.1 OK

# PITR incremental backup:

Status [OFF]

# Currently running:

(none)

# Backups:

FS /opt/backup/  
Snapshots:  
2023-08-28T10:30:25Z 24.09MB [restore\_to\_time: 2023-08-28T10:30:28Z]  
2023-08-28T09:01:02Z 14.49MB \<incremental, base\> [restore\_to\_time: 2023-08-28T09:01:04Z]

[mongodb@mongodb\_content\_cfg\_server1 percona]$ pbm list  
Backup snapshots:  
2023-08-28T09:01:02Z \<incremental, base\> [restore\_to\_time: 2023-08-28T09:01:04Z]  
2023-08-28T10:30:25Z [restore\_to\_time: 2023-08-28T10:30:28Z]

PITR :

[mongodb@mongodb\_content\_cfg\_server1 percona]$ pbm restore 2023-08-28T10:30:25Z  
Starting restore 2023-08-28T10:55:42.716998345Z from ‘2023-08-28T10:30:25Z’…Restore of the snapshot from ‘2023-08-28T10:30:25Z’ has started.  
Check restore status with: pbm describe-restore 2023-08-28T10:55:42.716998345Z -c \</path/to/pbm.conf.yaml\>  
No other pbm command is available while the restore is running!

[mongodb@mongodb\_content\_cfg\_server1 percona]$ pbm restore 2023-08-28T10:30:25Z  
Starting restore 2023-08-28T10:55:42.716998345Z from ‘2023-08-28T10:30:25Z’…Restore of the snapshot from ‘2023-08-28T10:30:25Z’ has started.  
Check restore status with: pbm describe-restore 2023-08-28T10:55:42.716998345Z -c \</path/to/pbm.conf.yaml\>  
No other pbm command is available while the restore is running!

[mongodb@mongodb\_content\_cfg\_server1 percona]$ pbm describe-restore 2023-08-28T10:55:42.716998345Z -c /opt/percona/storage.yaml  
name: “2023-08-28T10:55:42.716998345Z”  
opid: “”  
backup: “”  
type: physical  
status: running  
last\_transition\_time: “2023-08-28T10:56:09Z”  
replsets:

- name: mongo\_data\_rs1  
status: running  
last\_transition\_time: “2023-08-28T10:55:59Z”  
nodes:
  - name: mongodb\_content\_rs1n1:27017  
status: running  
last\_transition\_time: “2023-08-28T10:55:54Z”

- name: mongo\_data\_rs2  
status: running  
last\_transition\_time: “2023-08-28T10:56:03Z”  
nodes:
  - name: mongodb\_content\_rs2n1:27017  
status: running  
last\_transition\_time: “2023-08-28T10:55:58Z”

- name: mongo\_data\_rs3  
status: running  
last\_transition\_time: “2023-08-28T10:56:03Z”  
nodes:
  - name: mongodb\_content\_rs3n1:27017  
status: running  
last\_transition\_time: “2023-08-28T10:55:58Z”

- name: mongo\_conf  
status: running  
last\_transition\_time: “2023-08-28T10:56:04Z”  
nodes:
  - name: mongodb\_content\_cfg\_server1:27017  
status: running  
last\_transition\_time: “2023-08-28T10:55:59Z”
  - name: mongodb\_content\_cfg\_server2:27017  
status: running  
last\_transition\_time: “2023-08-28T10:55:58Z”
  - name: mongodb\_content\_cfg\_server3:27017  
status: running  
last\_transition\_time: “2023-08-28T10:55:58Z”

# [mongodb@mongodb\_content\_cfg\_server1 percona]$ pbm status

Cluster:

mongo\_data\_rs1:

- mongo\_data\_rs1/mongodb\_content\_rs1n1:27017 [P]: pbm-agent v2.2.1 OK  
mongo\_data\_rs2:
- mongo\_data\_rs2/mongodb\_content\_rs2n1:27017 [P]: pbm-agent v2.2.1 OK  
mongo\_conf:
- mongo\_conf/mongodb\_content\_cfg\_server1:27017 [P]: pbm-agent FAILED status:

> ERROR with ERROR: lost agent, last heartbeat: 1693220172

- mongo\_conf/mongodb\_content\_cfg\_server2:27017 [S]: pbm-agent FAILED status:

> ERROR with ERROR: lost agent, last heartbeat: 1693220171

- mongo\_conf/mongodb\_content\_cfg\_server3:27017 [S]: pbm-agent FAILED status:

> ERROR with ERROR: lost agent, last heartbeat: 1693220172  
> mongo\_data\_rs3:

- mongo\_data\_rs3/mongodb\_content\_rs3n1:27017 [P]: pbm-agent v2.2.1 OK

# PITR incremental backup:

Status [OFF]

# Currently running:

(none)

# Backups:

FS /opt/backup/  
Snapshots:  
2023-08-28T10:30:25Z 24.09MB [restore\_to\_time: 2023-08-28T10:30:28Z]  
2023-08-28T09:01:02Z 14.49MB \<incremental, base\> [restore\_to\_time: 2023-08-28T09:01:04Z]

[mongodb@mongodb\_content\_cfg\_server1 percona]$ pbm list --restore  
Restores history:

[mongodb@mongodb\_content\_cfg\_server1 percona]$ pbm logs  
2023-08-28T10:55:43Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-08-28T10:55:42.716998345Z] waiting for `starting` status in rs map[.pbm.restore/2023-08-28T10:55:42.716998345Z/rs.mongo\_conf/node.mongodb\_content\_cfg\_server1:27017:{} .pbm.restore/2023-08-28T10:55:42.716998345Z/rs.mongo\_conf/node.mongodb\_content\_cfg\_server2:27017:{} .pbm.restore/2023-08-28T10:55:42.716998345Z/rs.mongo\_conf/node.mongodb\_content\_cfg\_server3:27017:{}]  
2023-08-28T10:55:44Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-08-28T10:55:42.716998345Z] moving to state starting  
2023-08-28T10:55:44Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-08-28T10:55:42.716998345Z] waiting for `starting` status in rs map[.pbm.restore/2023-08-28T10:55:42.716998345Z/rs.mongo\_data\_rs1/node.mongodb\_content\_rs1n1:27017:{}]  
2023-08-28T10:55:48Z I [mongo\_data\_rs3/mongodb\_content\_rs3n1:27017] [restore/2023-08-28T10:55:42.716998345Z] waiting for cluster  
2023-08-28T10:55:48Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-08-28T10:55:42.716998345Z] waiting for cluster  
2023-08-28T10:55:48Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-08-28T10:55:42.716998345Z] waiting for shards map[.pbm.restore/2023-08-28T10:55:42.716998345Z/rs.mongo\_conf/rs:{} .pbm.restore/2023-08-28T10:55:42.716998345Z/rs.mongo\_data\_rs1/rs:{} .pbm.restore/2023-08-28T10:55:42.716998345Z/rs.mongo\_data\_rs2/rs:{} .pbm.restore/2023-08-28T10:55:42.716998345Z/rs.mongo\_data\_rs3/rs:{}]  
2023-08-28T10:55:49Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-08-28T10:55:42.716998345Z] waiting for cluster  
2023-08-28T10:55:54Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-08-28T10:55:42.716998345Z] waiting for cluster  
2023-08-28T10:56:23Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] pbm-agent:  
Version: 2.2.1  
Platform: linux/amd64  
GitCommit: 5d2c6eeb81be87a21ccf8d3ead6cba762e6bf70f  
GitBranch: release-2.2.1  
BuildTime: 2023-07-25\_13:42\_UTC  
GoVersion: go1.19  
2023-08-28T10:56:23Z

DataNode logs:

[mongodb@mongodb\_content\_rs2n1 /]$ pbm logs  
2023-08-28T10:51:23Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] got epoch {1693219863 8}  
2023-08-28T10:51:23Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-08-28T10:51:23.296062775Z] backup: 2023-08-28T10:30:25Z  
2023-08-28T10:51:23Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-08-28T10:51:23.296062775Z] recovery started  
2023-08-28T10:51:23Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] got command restore [name: 2023-08-28T10:51:23.296062775Z, snapshot: 2023-08-28T10:30:25Z] \<ts: 1693219883\>  
2023-08-28T10:51:23Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] got epoch {1693219863 8}  
2023-08-28T10:51:23Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] got command restore [name: 2023-08-28T10:51:23.296062775Z, snapshot: 2023-08-28T10:30:25Z] \<ts: 1693219883\>  
2023-08-28T10:51:23Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-08-28T10:51:23.296062775Z] backup: 2023-08-28T10:30:25Z  
2023-08-28T10:51:23Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-08-28T10:51:23.296062775Z] recovery started  
2023-08-28T10:51:23Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] got epoch {1693219863 8}  
2023-08-28T10:51:24Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-08-28T10:51:23.296062775Z] moving to state starting  
2023-08-28T10:51:24Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-08-28T10:51:23.296062775Z] backup: 2023-08-28T10:30:25Z  
2023-08-28T10:51:24Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-08-28T10:51:23.296062775Z] recovery started  
2023-08-28T10:51:24Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-08-28T10:51:23.296062775Z] waiting for `starting` status in rs map[.pbm.restore/2023-08-28T10:51:23.296062775Z/rs.mongo\_conf/node.mongodb\_content\_cfg\_server1:27017:{} .pbm.restore/2023-08-28T10:51:23.296062775Z/rs.mongo\_conf/node.mongodb\_content\_cfg\_server2:27017:{} .pbm.restore/2023-08-28T10:51:23.296062775Z/rs.mongo\_conf/node.mongodb\_content\_cfg\_server3:27017:{}]  
2023-08-28T10:51:24Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-08-28T10:51:23.296062775Z] moving to state starting  
2023-08-28T10:51:24Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-08-28T10:51:23.296062775Z] waiting for `starting` status in rs map[.pbm.restore/2023-08-28T10:51:23.296062775Z/rs.mongo\_data\_rs2/node.mongodb\_content\_rs2n1:27017:{}]  
2023-08-28T10:51:24Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-08-28T10:51:23.296062775Z] moving to state starting  
2023-08-28T10:51:24Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-08-28T10:51:23.296062775Z] waiting for `starting` status in rs map[.pbm.restore/2023-08-28T10:51:23.296062775Z/rs.mongo\_data\_rs1/node.mongodb\_content\_rs1n1:27017:{}]  
2023-08-28T10:51:28Z I [mongo\_data\_rs3/mongodb\_content\_rs3n1:27017] [restore/2023-08-28T10:51:23.296062775Z] waiting for cluster  
2023-08-28T10:51:29Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-08-28T10:51:23.296062775Z] waiting for cluster  
2023-08-28T10:51:29Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-08-28T10:51:23.296062775Z] waiting for cluster