PBM physical type backup failed when getting backupCursor with error of file not exist, please advice

MongoDB Percona Backup for MongoDB
#1

I am trying to verify the physical backup functionality of pbm according to doc (Experimental Feature: $backupCursorExtend in Percona Server for MongoDB - Percona Database Performance Blog) and with no luck.
I met with problem when exec “pbm backup -t physical”. This error could also been activated by exec “db.aggregate([{$backupCursor: {}}])” in the mongo shell and always get a exception like this:

m1:PRIMARY> db.aggregate([{$backupCursor: {}}])
uncaught exception: Error: command failed: {
“operationTime” : Timestamp(1655812554, 4),
“ok” : 0,
“errmsg” : “Failed to get a file’s size. Filename: /data/db/key.db/journal/WiredTigerLog.0000000004 Error: No such file or directory”,
“code” : 31403,
“codeName” : “Location31403”,
“$clusterTime” : {
“clusterTime” : Timestamp(1655812554, 4),
“signature” : {
“hash” : BinData(0,“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”),
“keyId” : NumberLong(0)
}
}
} : aggregate failed :

In the db folder, this folder “/data/db/key.db/journal/” do not even exist. But still got the same error if I create this folder in advance. The mongo server version is v4.4.6-8. Does anyone have some advice on this ? Thank you.

1 Like
#2

After some digging, I found out this error is because pbm can not find the fixed journal folder. I saw some discuss about make journal folder configurable a year ago. While on current master branch, journal folder is fixed in /data/db/journal. So I add some softlink to bypass this error and can make physical backup succeed. Wooha~

However, while restoring, the cluster state can not be ready anymore. digging…

1 Like
#3

Hello @Xiaolu,

I assume you have dbpath assigned to /data/db. Then PSMDB will use /data/db/key.db only if you enabled data at rest encryption. Thus we have situation when data at rest encryption is enabled but /data/db/key.db does not exist.
I have no idea how this happened on you machine. key.db subdir is created when you start PSMDB for the first time with empty dbpath and is never deleted by PSMDB.
Probably you enabled encryption when your dbpath dir was already initialized? This scenario is not supported because existing unencrypted instance cannot be switched to encrypted mode.

1 Like
#4

Hello, @Igor_Solodovnikov
You are right that my instance enabled data at rest encryption by default. My situation is /data/db/key.db exist while /data/db/key.db/journal does not exist. Since physical backup will copy journal file from /data/db/key.db/journal with data at rest encryption enabled, I have to add a soft link to bypass this error.

Blockquote cd /data/db/key.db && ln -s …/journal journal

1 Like
#5

Hi @Xiaolu
Has the original issue happened before any pbm restore? I mean it was the first time you’ve tried to run “pbm backup -t physical” that lead to the error, right?

1 Like
#6

That’s right, @Andrew_Pogrebnoi
Any idea about this error?

1 Like
#7

Not yet @Xiaolu. I wanted to ensure first that the pbm restore doesn’t cause it.

1 Like
#8

Hi @Xiaolu

I believe this issue is fixed by PSMDB-1119. That is it is fixed in these releases of Percona server for MonfoDB: 4.2.22-22, 4.4.16-16, 5.0.11-10

1 Like
#9

Hi, Igor

Thank you. I will try that fix.

1 Like