Percona Incremental backup - stuck at Waiting for the node to shutdown

MongoDB Percona Backup for MongoDB
1

Hi
Im trying to backup/restore a incremtental backup with percona. that backup runs like a charm, and if im running on a single node cluster, incremental restore works perfectly. but if i´m running in a sharded cluster, with 3x cfg servers, and 3 nodes, the restore starts but when getting to the step “waiting for the node to shutdown”, nothing happens, i can see the mongod service keeps running, but after around 30 sec, the pbm agent looses heartbeat… What is percona backup doing at that step? and what diffrent with single node and multinode in that step?

Logs:

Starting the restore job:
mongodb@mongodb_content_cfg_server2 /]$ pbm restore 2024-02-06T10:07:26Z -w
Starting restore 2024-02-07T06:47:59.304252431Z from ‘2024-02-06T10:07:26Z’…Started physical restore.
Waiting to finish…

Logs from node:
{“t”:{“$date”:“2024-02-07T06:47:55.698+00:00”},“s”:“I”, “c”:“CONNPOOL”, “id”:22567, “ctx”:“ShardRegistry”,“msg”:“Ending idle connection because the pool meets constraints”,“attr”:{“hostAndPort”:“mongodb_content_rs1n1:27017”,“numOpenConns”:1}}
2024-02-07T06:47:59.000+0000 I got command restore [name: 2024-02-07T06:47:59.304252431Z, snapshot: 2024-02-06T10:07:26Z] <ts: 1707288479>
2024-02-07T06:47:59.000+0000 I got epoch {1707288344 4}
2024-02-07T06:47:59.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] backup: 2024-02-06T10:07:26Z
2024-02-07T06:47:59.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] recovery started
2024-02-07T06:47:59.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] port: 27526
2024-02-07T06:48:00.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] mongod binary: /usr/bin/mongod, version: v5.0.24-21
2024-02-07T06:48:00.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] get src 2024-02-06T10:02:55Z
2024-02-07T06:48:00.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] moving to state starting
2024-02-07T06:48:00.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] waiting for starting status in rs map[.pbm.restore/2024-02-07T06:47:59.304252431Z/rs.mongo_data_rs2/node.mongodb_content_rs2n1:27017:{}]
{“t”:{“$date”:“2024-02-07T06:48:02.104+00:00”},“s”:“I”, “c”:“CONNPOOL”, “id”:22576, “ctx”:“ShardRegistry”,“msg”:“Connecting”,“attr”:{“hostAndPort”:“mongodb_content_cfg_server3:27017”}}
{“t”:{“$date”:“2024-02-07T06:48:02.282+00:00”},“s”:“I”, “c”:“CONNPOOL”, “id”:22576, “ctx”:“ShardRegistry”,“msg”:“Connecting”,“attr”:{“hostAndPort”:“mongodb_content_cfg_server1:27017”}}
2024-02-07T06:48:05.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] waiting for cluster
2024-02-07T06:48:15.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] converged to state starting
2024-02-07T06:48:15.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] starting
2024-02-07T06:48:15.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] moving to state running
2024-02-07T06:48:15.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] waiting for running status in rs map[.pbm.restore/2024-02-07T06:47:59.304252431Z/rs.mongo_data_rs2/node.mongodb_content_rs2n1:27017:{}]
2024-02-07T06:48:20.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] waiting for cluster
{“t”:{“$date”:“2024-02-07T06:48:29.643+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22430, “ctx”:“Checkpointer”,“msg”:“WiredTiger message”,“attr”:{“message”:“[1707288509:643870][7:0x7f3ed6126700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 7058, snapshot max: 7058 snapshot count: 0, oldest timestamp: (1707288202, 1) , meta checkpoint timestamp: (1707288502, 1) base write gen: 84”}}
2024-02-07T06:48:30.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] converged to state running
2024-02-07T06:48:30.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] send to stopAgent chan
2024-02-07T06:48:30.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] stop agents heartbeats
2024-02-07T06:48:30.000+0000 I [restore/2024-02-07T06:47:59.304252431Z] stopping mongod and flushing old data
2024-02-07T06:48:30.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] shutdown server
2024-02-07T06:48:30.000+0000 D [restore/2024-02-07T06:47:59.304252431Z] waiting for the node to shutdown

PBM status after 30 sec.
mongo_data_rs3:
mongo_data_rs3/mongodb_content_rs3n1:27017 [P]: pbm-agent FAILED status:
ERROR with ERROR: lost agent, last heartbeat: 1707288505
mongo_data_rs2:
mongo_data_rs2/mongodb_content_rs2n1:27017 [P]: pbm-agent FAILED status:
ERROR with ERROR: lost agent, last heartbeat: 1707288507
mongo_data_rs1:
mongo_data_rs1/mongodb_content_rs1n1:27017 [P]: pbm-agent FAILED status:
ERROR with ERROR: lost agent, last heartbeat: 1707288508
mongo_conf:
mongo_conf/mongodb_content_cfg_server1:27017 [S]: pbm-agent FAILED status:
ERROR with ERROR: lost agent, last heartbeat: 1707288506
mongo_conf/mongodb_content_cfg_server2:27017 [S]: pbm-agent FAILED status:
ERROR with ERROR: lost agent, last heartbeat: 1707288506
mongo_conf/mongodb_content_cfg_server3:27017 [P]: pbm-agent FAILED status:
ERROR with ERROR: lost agent, last heartbeat: 1707288506

2

I have tried to manually shutdown the mongod with /usr/bin/mongod --shutdown on all shards. then its starts to do the restore, but then stopping later on, and it seems like the restore is not fully successfull.

here are the logs:

2024-02-08T07:39:56.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] waiting for cluster
2024-02-08T07:40:06.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] converged to state running
2024-02-08T07:40:06.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] send to stopAgent chan
2024-02-08T07:40:07.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] stop agents heartbeats
2024-02-08T07:40:07.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] stopping mongod and flushing old data
2024-02-08T07:40:07.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] shutdown server
2024-02-08T07:40:07.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] waiting for the node to shutdown
{“t”:{“$date”:“2024-02-08T07:40:10.191+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23377, “ctx”:“SignalHandler”,“msg”:“Received signal”,“attr”:{“signal”:15,“error”:“Terminated”}}
{“t”:{“$date”:“2024-02-08T07:40:10.191+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23378, “ctx”:“SignalHandler”,“msg”:“Signal was sent by kill(2)”,“attr”:{“pid”:231,“uid”:1000}}
{“t”:{“$date”:“2024-02-08T07:40:10.191+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23381, “ctx”:“SignalHandler”,“msg”:“will terminate after current cmd ends”}
{“t”:{“$date”:“2024-02-08T07:40:10.191+00:00”},“s”:“I”, “c”:“REPL”, “id”:4784900, “ctx”:“SignalHandler”,“msg”:“Stepping down the ReplicationCoordinator for shutdown”,“attr”:{“waitTimeMillis”:15000}}
{“t”:{“$date”:“2024-02-08T07:40:10.191+00:00”},“s”:“I”, “c”:“REPL”, “id”:4794602, “ctx”:“SignalHandler”,“msg”:“Attempting to enter quiesce mode”}
{“t”:{“$date”:“2024-02-08T07:40:10.192+00:00”},“s”:“I”, “c”:“COMMAND”, “id”:4784901, “ctx”:“SignalHandler”,“msg”:“Shutting down the MirrorMaestro”}
… (removed because of to many chars in post) “ctx”:“SignalHandler”,“msg”:“shutdown: removing fs lock…”}
{“t”:{“$date”:“2024-02-08T07:40:10.382+00:00”},“s”:“I”, “c”:“-”, “id”:4784931, “ctx”:“SignalHandler”,“msg”:“Dropping the scope cache for shutdown”}
{“t”:{“$date”:“2024-02-08T07:40:10.382+00:00”},“s”:“I”, “c”:“FTDC”, “id”:4784926, “ctx”:“SignalHandler”,“msg”:“Shutting down full-time data capture”}
{“t”:{“$date”:“2024-02-08T07:40:10.382+00:00”},“s”:“I”, “c”:“FTDC”, “id”:20626, “ctx”:“SignalHandler”,“msg”:“Shutting down full-time diagnostic data capture”}
{“t”:{“$date”:“2024-02-08T07:40:10.388+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:20565, “ctx”:“SignalHandler”,“msg”:“Now exiting”}
{“t”:{“$date”:“2024-02-08T07:40:10.388+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23138, “ctx”:“SignalHandler”,“msg”:“Shutting down”,“attr”:{“exitCode”:0}}
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] revome old data
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-21-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-12-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-25–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-2–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-5-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-0–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-24-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/_mdb_catalog.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-26-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-15-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-16–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-37-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-12–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-18–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-21–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/sizeStorer.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-34–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/WiredTiger.wt
… (removed because of to many chars in post)
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-33-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/storage.bson
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-3-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-5–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-15–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-35-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-8–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/mongod.lock
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-17-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/journal
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-23-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-25-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-38-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-8-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-6–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-14-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-11-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-26–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/WiredTiger
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-39-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-18-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-9–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-41-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-42-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/WiredTiger.lock
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-13-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-7–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-11–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-17–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-43-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/diagnostic.data
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-1-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-13–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-1–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-14–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-24–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-6-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-29-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-27-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-29–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-31-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-19–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-34-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-4-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-2-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-35–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-7-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-0-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-22-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-22–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-10–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-28-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-40-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-3–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/index-19-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-4–3129342514983500851.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-36-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] remove /data/db/collection-9-8613182035066973332.wt
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copying backup data
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/journal/WiredTigerLog.0000000004.s2.0-104857600> to </data/db/journal/WiredTigerLog.0000000004>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/WiredTigerHS.wt.s2.0-24576> to </data/db/WiredTigerHS.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/_mdb_catalog.wt.s2.0-36864> to </data/db/_mdb_catalog.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-0–3129342514983500851.wt.s2.0-36864> to </data/db/collection-0–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-0-8613182035066973332.wt.s2.0-4096> to </data/db/collection-0-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-10–3129342514983500851.wt.s2.0-20480> to </data/db/collection-10–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-12–3129342514983500851.wt.s2.0-36864> to </data/db/collection-12–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-12-8613182035066973332.wt.s2.0-20480> to </data/db/collection-12-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-14–3129342514983500851.wt.s2.0-999424> to </data/db/collection-14–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-15–3129342514983500851.wt.s2.0-36864> to </data/db/collection-15–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-17–3129342514983500851.wt.s2.0-20480> to </data/db/collection-17–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-18-8613182035066973332.wt.s2.0-24576> to </data/db/collection-18-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-19–3129342514983500851.wt.s2.0-32768> to </data/db/collection-19–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-2–3129342514983500851.wt.s2.0-36864> to </data/db/collection-2–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-21–3129342514983500851.wt.s2.0-4096> to </data/db/collection-21–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-21-8613182035066973332.wt.s2.0-20480> to </data/db/collection-21-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-23–3129342514983500851.wt.s2.0-4096> to </data/db/collection-23–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-25–3129342514983500851.wt.s2.0-20480> to </data/db/collection-25–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-27–3129342514983500851.wt.s2.0-24576> to </data/db/collection-27–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-27-8613182035066973332.wt.s2.0-4096> to </data/db/collection-27-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-29–3129342514983500851.wt.s2.0-24576> to </data/db/collection-29–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-3-8613182035066973332.wt.s2.0-20480> to </data/db/collection-3-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-30-8613182035066973332.wt.s2.0-20480> to </data/db/collection-30-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-34–3129342514983500851.wt.s2.0-36864> to </data/db/collection-34–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-36-8613182035066973332.wt.s2.0-135168> to </data/db/collection-36-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-39-8613182035066973332.wt.s2.0-36864> to </data/db/collection-39-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-4–3129342514983500851.wt.s2.0-36864> to </data/db/collection-4–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-42-8613182035066973332.wt.s2.0-28672> to </data/db/collection-42-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-6–3129342514983500851.wt.s2.0-36864> to </data/db/collection-6–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-8–3129342514983500851.wt.s2.0-36864> to </data/db/collection-8–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/collection-9-8613182035066973332.wt.s2.0-4096> to </data/db/collection-9-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-1–3129342514983500851.wt.s2.0-36864> to </data/db/index-1–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-1-8613182035066973332.wt.s2.0-4096> to </data/db/index-1-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-10-8613182035066973332.wt.s2.0-4096> to </data/db/index-10-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-11–3129342514983500851.wt.s2.0-20480> to </data/db/index-11–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-11-8613182035066973332.wt.s2.0-4096> to </data/db/index-11-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-13–3129342514983500851.wt.s2.0-36864> to </data/db/index-13–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-13-8613182035066973332.wt.s2.0-20480> to </data/db/index-13-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-14-8613182035066973332.wt.s2.0-20480> to </data/db/index-14-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-15-8613182035066973332.wt.s2.0-4096> to </data/db/index-15-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-16–3129342514983500851.wt.s2.0-20480> to </data/db/index-16–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-16-8613182035066973332.wt.s2.0-4096> to </data/db/index-16-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-17-8613182035066973332.wt.s2.0-4096> to </data/db/index-17-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-18–3129342514983500851.wt.s2.0-20480> to </data/db/index-18–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-19-8613182035066973332.wt.s2.0-20480> to </data/db/index-19-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-2-8613182035066973332.wt.s2.0-4096> to </data/db/index-2-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-20–3129342514983500851.wt.s2.0-12288> to </data/db/index-20–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-20-8613182035066973332.wt.s2.0-20480> to </data/db/index-20-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-22–3129342514983500851.wt.s2.0-4096> to </data/db/index-22–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-22-8613182035066973332.wt.s2.0-20480> to </data/db/index-22-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-23-8613182035066973332.wt.s2.0-20480> to </data/db/index-23-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-24–3129342514983500851.wt.s2.0-4096> to </data/db/index-24–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-24-8613182035066973332.wt.s2.0-20480> to </data/db/index-24-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-25-8613182035066973332.wt.s2.0-20480> to </data/db/index-25-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-26–3129342514983500851.wt.s2.0-20480> to </data/db/index-26–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-26-8613182035066973332.wt.s2.0-20480> to </data/db/index-26-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-28–3129342514983500851.wt.s2.0-24576> to </data/db/index-28–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-28-8613182035066973332.wt.s2.0-4096> to </data/db/index-28-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-29-8613182035066973332.wt.s2.0-4096> to </data/db/index-29-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-3–3129342514983500851.wt.s2.0-20480> to </data/db/index-3–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-30–3129342514983500851.wt.s2.0-24576> to </data/db/index-30–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-31-8613182035066973332.wt.s2.0-20480> to </data/db/index-31-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-32-8613182035066973332.wt.s2.0-20480> to </data/db/index-32-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-33-8613182035066973332.wt.s2.0-4096> to </data/db/index-33-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-34-8613182035066973332.wt.s2.0-4096> to </data/db/index-34-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-35–3129342514983500851.wt.s2.0-36864> to </data/db/index-35–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-35-8613182035066973332.wt.s2.0-4096> to </data/db/index-35-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-37-8613182035066973332.wt.s2.0-94208> to </data/db/index-37-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-38-8613182035066973332.wt.s2.0-86016> to </data/db/index-38-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-4-8613182035066973332.wt.s2.0-20480> to </data/db/index-4-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-40-8613182035066973332.wt.s2.0-36864> to </data/db/index-40-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-41-8613182035066973332.wt.s2.0-36864> to </data/db/index-41-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-43-8613182035066973332.wt.s2.0-28672> to </data/db/index-43-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-5–3129342514983500851.wt.s2.0-20480> to </data/db/index-5–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-5-8613182035066973332.wt.s2.0-20480> to </data/db/index-5-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-6-8613182035066973332.wt.s2.0-4096> to </data/db/index-6-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-7–3129342514983500851.wt.s2.0-20480> to </data/db/index-7–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-7-8613182035066973332.wt.s2.0-4096> to </data/db/index-7-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-8-8613182035066973332.wt.s2.0-4096> to </data/db/index-8-8613182035066973332.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/index-9–3129342514983500851.wt.s2.0-20480> to </data/db/index-9–3129342514983500851.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/sizeStorer.wt.s2.0-36864> to </data/db/sizeStorer.wt>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/WiredTiger.backup.s2.0-153882> to </data/db/WiredTiger.backup>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/WiredTiger.s2.0-50> to </data/db/WiredTiger>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/storage.bson.s2.0-114> to </data/db/storage.bson>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] copy <2024-02-08T06:24:57Z/mongo_data_rs3/journal/WiredTigerLog.0000000005.s2> to </data/db/journal/WiredTigerLog.0000000005>
2024-02-08T07:40:11.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] preparing data
2024-02-08T07:40:13.000+0000 D [restore/2024-02-08T07:39:40.840706376Z] oplogTruncateAfterPoint: {1707373499 2}
2024-02-08T07:40:14.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] recovering oplog as standalone
2024-02-08T07:40:16.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] clean-up and reset replicaset config
2024-02-08T07:40:19.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] restore on node succeed
2024-02-08T07:40:19.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] moving to state done
2024-02-08T07:40:19.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] waiting for done status in rs map[.pbm.restore/2024-02-08T07:39:40.840706376Z/rs.mongo_data_rs3/node.mongodb_content_rs3n1:27017:{}]
2024-02-08T07:40:24.000+0000 I [restore/2024-02-08T07:39:40.840706376Z] waiting for shards map[.pbm.restore/2024-02-08T07:39:40.840706376Z/rs.mongo_conf/rs:{} .pbm.restore/2024-02-08T07:39:40.840706376Z/rs.mongo_data_rs1/rs:{} .pbm.restore/2024-02-08T07:39:40.840706376Z/rs.mongo_data_rs2/rs:{} .pbm.restore/2024-02-08T07:39:40.840706376Z/rs.mongo_data_rs3/rs:{}]

then its stuck here.
any idea whats going wrong? i will paste my pbm_config.yaml also

storage:
type: filesystem
filesystem:
path: /opt/XXX/backup/
restore:
mongodLocation: /usr/bin/mongod
mongodLocationMap:
“mongodb_content_cfg_server1:27017”: /usr/bin/mongod
“mongodb_content_cfg_server2:27017”: /usr/bin/mongod
“mongodb_content_cfg_server3:27017”: /usr/bin/mongod
“mongodb_content_rs1n1:27017”: /usr/bin/mongod
“mongodb_content_rs2n1:27017”: /usr/bin/mongod
“mongodb_content_rs3n1:27017”: /usr/bin/mongod

3

update:
I dont have any auth on my mongodb, is that an issue? i can see i cannot remotely run the db.runcommand({shutdown: 1}) because no auth is enabled, but the question is. Are the pbm-agent running the shutdown command from localhost, or is it running the command remotely from the cfg server where the restore is executed?

can you please help out here? :slight_smile:

4

update:
I have now enabled auth on my mongodb, and the restore is now running successfull. so it seems like it connects from the cfg server to the other hosts and running the shutdown, is this the correct assumption?

so question is now… is there anyway to get this running without authentication?

/Morten

5

Update again…

Found out that my pbm-uri had my hostname and not localhost, after changing to localhost, everything now works…
Hopefully this thread can be helpfull for others, trying to run mongodb and pbm in same docker container.