Pbm restore is failing

1

Hi Experts,

Facing below error while restoring from backup using pbm. I am executing this restore command by setting config cluster connection string in PBM_MONGODB_URI

[mongoadmin@test1gg ~]$ pbm restore 2024-03-28T10:54:38Z
Starting restore 2024-03-28T11:16:30.75472608Z from ‘2024-03-28T10:54:38Z’…Error: waiting for start: cluster failed: waiting for start: cluster failed: failed to ensure snapshot file 2024-03-28T10:54:38Z/shardr2/metadata.json: no such file

  • Restore on replicaset “shardr2” in state: error: failed to ensure snapshot file 2024-03-28T10:54:38Z/shardr2/metadata.json: no such file

Below are the detail of my cluster

[mongoadmin@test1gg ~]$ pbm status
Cluster:

shard1:

  • shard1/10.161.171.49:27017 [P]: pbm-agent v2.4.1 OK
  • shard1/10.161.171.50:27017 [S]: pbm-agent v2.4.1 OK
  • shard1/10.161.171.51:27017 [S]: pbm-agent v2.4.1 OK
    shardr2:
  • shardr2/10.161.171.49:27018 [S]: pbm-agent v2.4.1 OK
  • shardr2/10.161.171.50:27018 [P]: pbm-agent v2.4.1 OK
  • shardr2/10.161.171.51:27018 [S]: pbm-agent v2.4.1 OK
    confset:
  • confset/10.161.171.49:26000 [S]: pbm-agent v2.4.1 OK
  • confset/10.161.171.50:26001 [P]: pbm-agent v2.4.1 OK
  • confset/10.161.171.51:26002 [S]: pbm-agent v2.4.1 OK

PITR incremental backup:

Status [OFF]

Currently running:

(none)

Backups:

FS /backup/shard1
Snapshots:
2024-03-28T10:54:38Z 187.81KB [restore_to_time: 2024-03-28T10:54:42Z]

Backup files

confset backup is on 10.161.171.49

[mongoadmin@test1gg confset]$ pwd
/backup/shard1/2024-03-28T10:54:38Z/confset
[mongoadmin@test1gg confset]$ ls -ltr
total 176
-rwxrwxrwx. 1 mongod mongod 854 Mar 28 16:24 admin.system.users.s2
-rwxrwxrwx. 1 mongod mongod 155 Mar 28 16:24 admin.system.roles.s2
-rwxrwxrwx. 1 mongod mongod 171 Mar 28 16:24 admin.system.version.s2
-rwxrwxrwx. 1 mongod mongod 854 Mar 28 16:24 admin.pbmRUsers.s2
-rwxrwxrwx. 1 mongod mongod 155 Mar 28 16:24 admin.pbmRRoles.s2
-rwxrwxrwx. 1 mongod mongod 1140 Mar 28 16:24 admin.pbmBackups.s2
-rwxrwxrwx. 1 mongod mongod 1730 Mar 28 16:24 admin.pbmOpLog.s2
-rwxrwxrwx. 1 mongod mongod 42972 Mar 28 16:24 admin.pbmLog.s2
-rwxrwxrwx. 1 mongod mongod 505 Mar 28 16:24 admin.pbmConfig.s2
-rwxrwxrwx. 1 mongod mongod 27103 Mar 28 16:24 admin.system.profile.s2
-rwxrwxrwx. 1 mongod mongod 260 Mar 28 16:24 admin.pbmLock.s2
-rwxrwxrwx. 1 mongod mongod 1018 Mar 28 16:24 admin.pbmCmd.s2
-rwxrwxrwx. 1 mongod mongod 2755 Mar 28 16:24 admin.pbmRestores.s2
-rwxrwxrwx. 1 mongod mongod 511 Mar 28 16:24 admin.pbmAgents.s2
-rwxrwxrwx. 1 mongod mongod 258 Mar 28 16:24 config.collections.s2
-rwxrwxrwx. 1 mongod mongod 163 Mar 28 16:24 config.shards.s2
-rwxrwxrwx. 1 mongod mongod 20738 Mar 28 16:24 config.chunks.s2
-rwxrwxrwx. 1 mongod mongod 101 Mar 28 16:24 config.version.s2
-rwxrwxrwx. 1 mongod mongod 99 Mar 28 16:24 config.settings.s2
-rwxrwxrwx. 1 mongod mongod 143 Mar 28 16:24 config.databases.s2
-rwxrwxrwx. 1 mongod mongod 12240 Mar 28 16:24 metadata.json
drwxrwxrwx. 2 mongod mongod 51 Mar 28 16:24 oplog

shard1 and shardr2 backup is on 10.161.171.51

[root@test3gg shard1]# pwd
/backup/shard1/2024-03-28T10:54:38Z/shard1
[root@test3gg shard1]# ls -ltr
total 28
-rwxrwxrwx. 1 mongod mongod 853 Mar 28 16:24 admin.system.users.s2
-rwxrwxrwx. 1 mongod mongod 155 Mar 28 16:24 admin.system.roles.s2
-rwxrwxrwx. 1 mongod mongod 340 Mar 28 16:24 admin.system.version.s2
-rwxrwxrwx. 1 mongod mongod 853 Mar 28 16:24 admin.pbmRUsers.s2
-rwxrwxrwx. 1 mongod mongod 3665 Mar 28 16:24 admin.system.profile.s2
-rwxrwxrwx. 1 mongod mongod 155 Mar 28 16:24 admin.pbmRRoles.s2
-rwxrwxrwx. 1 mongod mongod 2994 Mar 28 16:24 metadata.json
drwxrwxrwx. 2 mongod mongod 50 Mar 28 16:24 oplog

[root@test3gg shardr2]# pwd
/backup/shard1/2024-03-28T10:54:38Z/shardr2
[root@test3gg shardr2]# ls -ltr
total 56
-rwxrwxrwx. 1 mongod mongod 850 Mar 28 16:24 admin.system.users.s2
-rwxrwxrwx. 1 mongod mongod 155 Mar 28 16:24 admin.system.roles.s2
-rwxrwxrwx. 1 mongod mongod 375 Mar 28 16:24 admin.system.version.s2
-rwxrwxrwx. 1 mongod mongod 850 Mar 28 16:24 admin.pbmRUsers.s2
-rwxrwxrwx. 1 mongod mongod 4653 Mar 28 16:24 admin.system.profile.s2
-rwxrwxrwx. 1 mongod mongod 155 Mar 28 16:24 admin.pbmRRoles.s2
-rwxrwxrwx. 1 mongod mongod 21245 Mar 28 16:24 gg.shardedtable.s2
-rwxrwxrwx. 1 mongod mongod 3518 Mar 28 16:24 metadata.json
drwxrwxrwx. 2 mongod mongod 50 Mar 28 16:24 oplog