# Cant execute incremental backup
**URL:** https://forums.percona.com/t/cant-execute-incremental-backup/25702
**Category:** Percona Backup for MongoDB
**Created:** [October 6, 2023, 6:34am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702 "2023-10-06T06:34:45Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Pjotz](https://forums.percona.com/u/Pjotz)
#### Post date: [October 6, 2023, 6:34am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/1 "2023-10-06T06:34:45Z")
Hi,
Got everything running, system is 3 nodes with 1 replica, got a backup container for every container. im able to run psyhical backup, but getting error below when trying to run incremental.
Note: the mongodb container is the mongo community editon, not perconas.
---
### Author:  [@Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba)
#### Post date: [October 6, 2023, 7:03am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/2 "2023-10-06T07:03:34Z")
Hi @Pjotz ,
The Physical, Incremental, and Snapshot-based backups are [only possible with Percona Server for MongoDB](https://docs.percona.com/percona-backup-mongodb/features/backup-types.html). PSMDB has [$backupCursor](https://docs.percona.com/percona-server-for-mongodb/6.0/backup-cursor.html), which is not available in Community Edition.
The error tells: no available **PBM agents** (or proper **mongod** ) for backups (in your case, incremental).
With MongoDB Community Edition, you can only have Full Logical, Selective Logical, and Point-in-Time recovery backups.
However, since [PBM 2.3.0](https://docs.percona.com/percona-backup-mongodb/release-notes/2.3.0.html#release-highlights), you can set up a mixed environment - replace one or more CE node(s) with PSMDB **mongod** in each replsets. In this case, PBM will try to do other backups. Later, you can restore with CE-only instances.
---
### Author:  [@Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba)
#### Post date: [October 6, 2023, 7:10am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/3 "2023-10-06T07:10:12Z")
PSMDB is fully compatible with CE (just with extra enterprise-level features available for everyone).
You can do this in one of two ways:
- replace existing node(s) with PSMDB
- add one **hidden** PSMDB node with PBM and one arbiter node (for voting balance; no agent is needed for the arbiter)
---
### Author:  [@Pjotz](https://forums.percona.com/u/Pjotz)
#### Post date: [October 6, 2023, 9:49am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/4 "2023-10-06T09:49:28Z")
Thanks for the reply, have gotten everything running, and gotten the first incremental backup done, when i try to restore i get this error message?
when i check the user on the node:
---
### Author:  [@Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba)
#### Post date: [October 6, 2023, 10:07am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/5 "2023-10-06T10:07:12Z")
Are you running PBM as a sidecar (not inside the same mongod container)?
For physical/incremental restore, PBM agent must have ability to execute **mongod** binary (it can be available by **$PATH** env var or customized by [restore.mongodLocation](https://docs.percona.com/percona-backup-mongodb/reference/restore-options.html#mongodlocation) / [restore.mongodLocationMap](https://docs.percona.com/percona-backup-mongodb/reference/restore-options.html#mongodlocationmap) config.
PBM Agents perform important MongoDB nodes and cluster config [preparations](https://www.percona.com/blog/physical-backup-support-in-percona-backup-for-mongodb/) during restore from physical backups.
Additionally to the blog article, take a look at [PBM docs: Physical backups and restores](https://docs.percona.com/percona-backup-mongodb/features/physical.html#physical-backups-and-restores)
---
### Author:  [@Pjotz](https://forums.percona.com/u/Pjotz)
#### Post date: [October 6, 2023, 10:11am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/6 "2023-10-06T10:11:24Z")
Hi, yeah running as a sidecar, didnt know the option for running the pbm agent inside the percona-mongodb-server was an option? because the pbm-agent is not installed there by default as i can see.
i would rather run both in same container if you got the recipy for that.
---
### Author:  [@Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba)
#### Post date: [October 6, 2023, 10:17am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/7 "2023-10-06T10:17:53Z")
We do not provide docker images for PBM with PSMDB inside.
You will need to build your own image using [Percona Software Repositories](https://docs.percona.com/percona-software-repositories/index.html) to install PSDMB and PBM for the image.
---
### Author:  [@Pjotz](https://forums.percona.com/u/Pjotz)
#### Post date: [October 9, 2023, 10:31am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/8 "2023-10-09T10:31:36Z")
Hello again,
Tried to create the container with both service, with no luck, cannot get it to start both services. so I´m back to the sidecar. and tried the solution with the mongoLocation: and still getting an error. when i try to execute the binary inside the node container it can win.
* * *
configfile:
storage:
type: filesystem
filesystem:
path: /opt/backup/
restore:
mongodLocation: /usr/bin/mongod
## [mongodb@mongodb\_backup\_cfg\_server1 /]$ cat /opt/xci/percona/storage.yaml storage: type: filesystem filesystem: path: /opt/xci/backup/ restore: mongodLocation: /usr/bin/mongod
PBM LOGS:
## [mongodb@mongodb\_backup\_cfg\_server1 /]$ pbm logs 2023-10-09T10:25:17Z I [mongo\_conf/mongodb\_content\_cfg\_server3:27017] [restore/2023-10-09T10:25:17.375588822Z] backup: 2023-10-06T09:25:38Z 2023-10-09T10:25:17Z I [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-10-09T10:25:17.375588822Z] recovery started 2023-10-09T10:25:17Z I [mongo\_conf/mongodb\_content\_cfg\_server3:27017] [restore/2023-10-09T10:25:17.375588822Z] recovery started 2023-10-09T10:25:17Z I [mongo\_conf/mongodb\_content\_cfg\_server2:27017] got command restore [name: 2023-10-09T10:25:17.375588822Z, snapshot: 2023-10-06T09:25:38Z] \ 2023-10-09T10:25:17Z I [mongo\_conf/mongodb\_content\_cfg\_server2:27017] got epoch {1696832006 5} 2023-10-09T10:25:17Z E [mongo\_conf/mongodb\_content\_cfg\_server1:27017] [restore/2023-10-09T10:25:17.375588822Z] restore: check mongod binary: run: fork/exec /usr/bin/mongod: no such file or directory. stderr: 2023-10-09T10:25:17Z I [mongo\_conf/mongodb\_content\_cfg\_server2:27017] [restore/2023-10-09T10:25:17.375588822Z] backup: 2023-10-06T09:25:38Z 2023-10-09T10:25:17Z E [mongo\_conf/mongodb\_content\_cfg\_server3:27017] [restore/2023-10-09T10:25:17.375588822Z] restore: check mongod binary: run: fork/exec /usr/bin/mongod: no such file or directory. stderr: 2023-10-09T10:25:17Z I [mongo\_conf/mongodb\_content\_cfg\_server2:27017] [restore/2023-10-09T10:25:17.375588822Z] recovery started 2023-10-09T10:25:17Z E [mongo\_conf/mongodb\_content\_cfg\_server2:27017] [restore/2023-10-09T10:25:17.375588822Z] restore: check mongod binary: run: fork/exec /usr/bin/mongod: no such file or directory. stderr: 2023-10-09T10:25:17Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] got command restore [name: 2023-10-09T10:25:17.375588822Z, snapshot: 2023-10-06T09:25:38Z] \ 2023-10-09T10:25:17Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] got epoch {1696832006 5} 2023-10-09T10:25:17Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-10-09T10:25:17.375588822Z] backup: 2023-10-06T09:25:38Z 2023-10-09T10:25:17Z I [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-10-09T10:25:17.375588822Z] recovery started 2023-10-09T10:25:18Z E [mongo\_data\_rs1/mongodb\_content\_rs1n1:27017] [restore/2023-10-09T10:25:17.375588822Z] restore: check mongod binary: run: fork/exec /usr/bin/mongod: no such file or directory. stderr: 2023-10-09T10:25:18Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] got command restore [name: 2023-10-09T10:25:17.375588822Z, snapshot: 2023-10-06T09:25:38Z] \ 2023-10-09T10:25:18Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] got epoch {1696832006 5} 2023-10-09T10:25:18Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-10-09T10:25:17.375588822Z] backup: 2023-10-06T09:25:38Z 2023-10-09T10:25:18Z I [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-10-09T10:25:17.375588822Z] recovery started 2023-10-09T10:25:18Z E [mongo\_data\_rs2/mongodb\_content\_rs2n1:27017] [restore/2023-10-09T10:25:17.375588822Z] restore: check mongod binary: run: fork/exec /usr/bin/mongod: no such file or directory. stderr:
when running command in the node container:
[mongodb@mongodb\_content\_rs1n1 /]$ whoami
mongodb
[mongodb@mongodb\_content\_rs1n1 /]$ /usr/bin/mongod
{“t”:{“$date”:“2023-10-09T10:26:58.385+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“main”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}
{“t”:{“$date”:“2023-10-09T10:26:58.388+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4648601, “ctx”:“main”,“msg”:“Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:4615611, “ctx”:“initandlisten”,“msg”:“MongoDB starting”,“attr”:{“pid”:157,“port”:27017,“dbPath”:“/data/db”,“architecture”:“64-bit”,“host”:“mongodb\_content\_rs1n1”}}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“W”, “c”:“CONTROL”, “id”:20720, “ctx”:“initandlisten”,“msg”:“Available memory is less than system memory”,“attr”:{“availableMemSizeMB”:16384,“systemMemSizeMB”:96575}}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:{“buildInfo”:{“version”:“4.4.23-22”,“gitVersion”:“6dbe14d25e2a4ba0515610749b1afe4119b06c42”,“openSSLVersion”:“OpenSSL 1.1.1k FIPS 25 Mar 2021”,“modules”:,“allocator”:“tcmalloc”,“environment”:{“distarch”:“x86\_64”,“target\_arch”:“x86\_64”}}}}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:{“os”:{“name”:“Oracle Linux Server release 8.8”,“version”:“Kernel 5.4.0-139-generic”}}}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:{“options”:{}}}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“E”, “c”:“STORAGE”, “id”:20568, “ctx”:“initandlisten”,“msg”:“Error setting up listener”,“attr”:{“error”:{“code”:9001,“codeName”:“SocketException”,“errmsg”:“Address already in use”}}}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“REPL”, “id”:4784900, “ctx”:“initandlisten”,“msg”:“Stepping down the ReplicationCoordinator for shutdown”,“attr”:{“waitTimeMillis”:10000}}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“COMMAND”, “id”:4784901, “ctx”:“initandlisten”,“msg”:“Shutting down the MirrorMaestro”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“SHARDING”, “id”:4784902, “ctx”:“initandlisten”,“msg”:“Shutting down the WaitForMajorityService”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4784905, “ctx”:“initandlisten”,“msg”:“Shutting down the global connection pool”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4784918, “ctx”:“initandlisten”,“msg”:“Shutting down the ReplicaSetMonitor”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“SHARDING”, “id”:4784921, “ctx”:“initandlisten”,“msg”:“Shutting down the MigrationUtilExecutor”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:4784925, “ctx”:“initandlisten”,“msg”:“Shutting down free monitoring”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:4784927, “ctx”:“initandlisten”,“msg”:“Shutting down the HealthLog”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:4784929, “ctx”:“initandlisten”,“msg”:“Acquiring the global lock for shutdown”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“-”, “id”:4784931, “ctx”:“initandlisten”,“msg”:“Dropping the scope cache for shutdown”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“FTDC”, “id”:4784926, “ctx”:“initandlisten”,“msg”:“Shutting down full-time data capture”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:20565, “ctx”:“initandlisten”,“msg”:“Now exiting”}
{“t”:{“$date”:“2023-10-09T10:26:58.389+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23138, “ctx”:“initandlisten”,“msg”:“Shutting down”,“attr”:{“exitCode”:48}}
[mongodb@mongodb\_content\_rs1n1 /]$
what am i doing wrong?
---
### Author:  [@Pjotz](https://forums.percona.com/u/Pjotz)
#### Post date: [October 17, 2023, 5:53am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/9 "2023-10-17T05:53:40Z")
@Dmytro_Zghoba any ideas?
---
### Author:  [@Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba)
#### Post date: [October 17, 2023, 7:52am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/10 "2023-10-17T07:52:06Z")
the most simplest way I found
```auto
FROM mongo:6.0 as mongo
FROM percona/percona-backup-mongodb:2.3.0
COPY --from=mongo /bin/mongod /bin/
```
---
### Author:  [@Pjotz](https://forums.percona.com/u/Pjotz)
#### Post date: [October 17, 2023, 8:56am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/11 "2023-10-17T08:56:45Z")
Thank you for that, had missed that it was missing mongod inside the backup sidecar. now that is done. it acctully starts the restore without any errors. Tho, as you can follow on the screenshot below.
1. starts restore of incremental backup
2. stuck at waiting
3. logs stuck at waiting for cluster
4. pbm agent lost connection and not comming back up.
5. nothing is restored, but all docker containers are running.
I have tried to create a single container with both backup and mongo server, but without any success.
---
### Author:  [@Dmytro\_Zghoba](https://forums.percona.com/u/Dmytro_Zghoba)
#### Post date: [October 17, 2023, 9:07am UTC](https://forums.percona.com/t/cant-execute-incremental-backup/25702/12 "2023-10-17T09:07:35Z")
PBM shuts down the mongod process during physical restore. Docker Engine could start it again.
Physical/incremental backup is a copy of **–dbpth**. To restore, PBM has to shut down the mongod process and replace the **dbpath** (and do extra work using mongod binary).
The mongod process should not start during restore. (if it starts, the dbpath data will be broken.) Make sure the restart policy does not restart the mongod process/container.
Start cluster when PBM report about successful restore.
Also, all mongos instances should be down during restore period.