Environment:
PSMDB (mongod 7.0.30-16), 3-node replica set, WiredTiger with directoryForIndexes:true
Nodes: r5.16xlarge, data disk gp3 16000 GiB / 40000 IOPS / 1000 MB/s
RAM: 248 GB WT cache
Dataset: 11 TB on-disk, one large index (index/149-*.wt, 1.7 TB)
PBM version: 2.12.0
Description
I kept hitting a wall while trying to restore a multi TB replicaset (11TB) with PBM, on both pbm restore (physical) and pbm restore-finish (external/EBS). Both go through recoverStandaloneFromOplog(), which replays the oplog in a temporary standalone mongod and then shuts it down. That shutdown forces a WiredTiger checkpoint, and PBM waits for it in waitMgoShutdown() using a hardcoded mongodLockTimeout = 30 * time.Minute. There’s no config, env var, or CLI flag to change it.
Due to the huge data size / workload, the checkpoint needs more than 30 minutes to complete across a large number of WT index files, so the wait times out. PBM marks the restore error, and it tears down the data dir while the checkpoint thread is still writing causing WT_PANIC and leaves /data empty.
I confirmed the checkpoint isn’t stuck as I patched the pbm-agent locally and raised the constant to 6h locally, and the exact same ebs restore-finish phase completed successfully in about 1h45m.
It would be good if this mongodLockTimeout parameter can be made configurable or the default parameter raised more generously so datasets at this scale can be restored successfully.
see the code path here: percona-backup-mongodb/pbm/restore/physical.go at v2.12.0 · percona/percona-backup-mongodb · GitHub
Log snippets below:
timeout error
root@host-mongod-rs01-01:~# pbm describe-restore 2026-07-28T16:35:48.371868986Z -c /etc/pbm-storage.conf --mongodb-uri="mongodb://localhost:27018"
name: "2026-07-28T16:35:48.371868986Z"
opid: ""
backup: ""
type: physical
status: error
error: 'recover oplog as standalone: shutdown mongo: wait mongod shutdown: timeout
during waiting for lock file /data/mongod.lock'
start: "1970-01-01T00:00:00Z"
last_transition_time: "2026-07-28T20:43:10Z"
replsets:
- name: rs01
status: error
last_transition_time: "2026-07-28T20:43:10Z"
nodes:
- name: host-mongod-rs01-01.com:27018
status: error
error: 'recover oplog as standalone: shutdown mongo: wait mongod shutdown: timeout
during waiting for lock file /data/mongod.lock'
last_transition_time: "2026-07-28T20:43:09Z"
- name: host-mongod-rs01-02.com:27018
status: error
error: 'recover oplog as standalone: shutdown mongo: wait mongod shutdown: timeout
during waiting for lock file /data/mongod.lock'
last_transition_time: "2026-07-28T20:42:23Z"
- name: host-mongod-rs01-03.com:27018
status: error
error: 'recover oplog as standalone: shutdown mongo: wait mongod shutdown: timeout
during waiting for lock file /data/mongod.lock'
last_transition_time: "2026-07-28T20:43:05Z"
error: 'recover oplog as standalone: shutdown mongo: wait mongod shutdown: timeout
during waiting for lock file /data/mongod.lock
Physical restore failure, pbm.restore.log, ~11 TB, 2026-07-28 (exactly 30 min)
20:11:50 temp standalone mongod started on port 27411 (recoverFromOplogAsStandalone=true, takeUnstableCheckpointOnShutdown=true, dbPath:/data)
20:11:52 ...
20:13:08 Completed oplog application for recovery ... numOpsApplied: 388147 (oplog replay SUCCEEDED, startup complete)
20:13:08 client application.name: pbm-physical-restore connects, issues {shutdown:1, timeoutMillis:15000}
20:13:08 "Shutting down checkpoint thread" → takeUnstableCheckpointOnShutdown forces full WT checkpoint
20:28..20:32 Checkpoint has been running for 1181 seconds and wrote: 505000 pages (9041 MB)
(then NO further checkpoint progress logged for ~11 min)
~20:43:10 recover oplog as standalone: shutdown mongo: wait mongod shutdown: timeout during waiting for lock file /data/mongod.lock
→ restore marked error; PBM begins teardown
20:43:15 __directory_list_worker:46:/data/journal: opendir: No such file or directory
→ WT_PANIC → fassert → abort (PBM deleted dbpath while checkpoint thread still writing → /data empty afterward)
after patching pbm-agent, restore completes successfully
pbm-agent: locally rebuilt from v2.12.0 with mongodLockTimeout changed 30m → 6h
pbm restore --external → restore-finish
index/149's shutdown checkpoint runs PAST the old 30-min kill point and COMPLETES successfully
pbm describe-restore: status: done (all 3 nodes done, ~1h45m)