PBM 2.6.0 Wrong backup mongodb_version

MongoDB Percona Backup for MongoDB
,
1

Hey guys,
Looks like PBM 2.6.0 stores wrong mongodb_version in backup metadata and refuses to restore it
We have pdmdb 7.0.12-7 with FCV 7.0 and PBM 2.6.0:

pbm describe-backup 2024-09-11T13:00:02Z
name: “2024-09-11T13:00:02Z”
opid: 66e19453d889f208094ced1b
type: physical
last_write_time: “2024-09-11T13:00:05Z”
last_transition_time: “2024-09-11T13:15:20Z”
mongodb_version: 6.0.16-13
fcv: “7.0”
pbm_version: 2.6.0
status: done
size_h: 101.3 GiB
…
As you can see, here we have “mongodb_version: 6.0.16-13”, and we never had that version.

And the error is:
Starting restore 2024-09-11T13:55:32.953809398Z from ‘2024-09-11T13:00:02Z’.Error: backup’s Mongo version (6.0.16-13) is not compatible with Mongo 7.0.12-7

We downgraded PBM to 2.5.0 and made a fresh backup:

pbm describe-backup 2024-09-11T14:19:56Z
name: “2024-09-11T14:19:56Z”
opid: 66e1a70c504dc024da1f6e6a
type: physical
last_write_time: “2024-09-11T14:19:59Z”
last_transition_time: “2024-09-11T14:35:03Z”
mongodb_version: 7.0.12-7
fcv: “7.0”
pbm_version: 2.5.0
status: done
size_h: 101.3 GiB
…
Here we have correct mongodb_version
Looks like there is some bug in GetMongoVersion or smth like.

2

Hi,

Can you please provide the following info from your DB?
db.runCommand({ buildInfo: 1 })

3

Hello.

We have replicaset with 3 members and all of them identical:

test> db.runCommand({ buildInfo: 1 })
{
version: ‘7.0.12-7’,
psmdbVersion: ‘7.0.12-7’,
gitVersion: ‘87737e473e09ac7c02486ba75e784bc52ddc45e4’,
modules: ,
proFeatures: ,
allocator: ‘tcmalloc’,
javascriptEngine: ‘mozjs’,
sysInfo: ‘deprecated’,
versionArray: [ 7, 0, 13, -100 ],
openssl: {
running: ‘OpenSSL 3.0.2 15 Mar 2022’,
compiled: ‘OpenSSL 3.0.2 15 Mar 2022’
},
buildEnvironment: {
distmod: ‘’,
distarch: ‘x86_64’,
cc: ‘/opt/mongodbtoolchain/v4/bin/gcc: gcc (GCC) 11.3.0’,
ccflags: ‘-include mongo/platform/basic.h -ffp-contract=off -fasynchronous-unwind-tables -g2 -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -fno-omit-frame-pointer -fno-strict-aliasing -O2 -march=sandybridge -mtune=generic -mprefer-vector-width=128 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-const-variable -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -gdwarf32 -fdebug-types-section -Wa,–nocompress-debug-sections -fno-builtin-memcmp -Wimplicit-fallthrough=5’,
cxx: ‘/opt/mongodbtoolchain/v4/bin/g++: g++ (GCC) 11.3.0’,
cxxflags: ‘-Woverloaded-virtual -Wpessimizing-move -Wno-maybe-uninitialized -fsized-deallocation -Wno-deprecated -std=c++20’,
linkflags: ‘-Wl,–build-id=sha1 -B/opt/mongodbtoolchain/v4/bin -pthread -Wl,-z,now -fuse-ld=lld -fstack-protector-strong -gdwarf32 -fdebug-types-section -Wl,–build-id=sha1 -Wl,–hash-style=gnu -Wl,-z,noexecstack -Wl,–warn-execstack -Wl,-z,relro -Wl,–compress-debug-sections=none -Wl,-z,origin -Wl,–enable-new-dtags’,
target_arch: ‘x86_64’,
target_os: ‘linux’,
cppdefines: ‘SAFEINT_USE_INTRINSICS 0 PCRE2_STATIC PERCONA_AUDIT_ENABLED NDEBUG _XOPEN_SOURCE 700 _GNU_SOURCE _FORTIFY_SOURCE 2 ABSL_FORCE_ALIGNED_ACCESS BOOST_ENABLE_ASSERT_DEBUG_HANDLER BOOST_FILESYSTEM_NO_CXX20_ATOMIC_REF BOOST_LOG_NO_SHORTHAND_NAMES BOOST_LOG_USE_NATIVE_SYSLOG BOOST_LOG_WITHOUT_THREAD_ATTR BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS BOOST_SYSTEM_NO_DEPRECATED BOOST_THREAD_USES_DATETIME BOOST_THREAD_VERSION 5’
},
bits: 64,
debug: false,
maxBsonObjectSize: 16777216,
storageEngines: [ ‘devnull’, ‘inMemory’, ‘wiredTiger’ ],
ok: 1,
‘$clusterTime’: {
clusterTime: Timestamp({ t: 1726129770, i: 3 }),
signature: {
hash: Binary.createFromBase64(‘KItv1/IvWxxhwAd2erYQF6aV6ZQ=’, 0),
keyId: Long(‘7356931927510614018’)
}
},
operationTime: Timestamp({ t: 1726129770, i: 3 })
}

4

It’s not possible to reproduce the issue on our side, it works OK with PSMDB 7.0.12 and PBM 2.6.0.

mongodb_version: 7.0.12-7
fcv: "7.0"
pbm_version: 2.6.0

Also PBM can’t get some random PSMDB version unless it’s set inside database, so can you please provide the result of this output?

rs.status().members.forEach(function (member) {
    print(`[${member._id}]: ${member.name}`);
    
    if (member.self) {
        var buildInfo = db.serverBuildInfo();
        printjson({ psmdbVersion: buildInfo.psmdbVersion || "N/A", version: buildInfo.version || "N/A" });
    } else {
        try {
            var mdb = new Mongo(member.name + "/admin");
            var adminDB = mdb.getDB('admin');
            
            // !!!!!! Use your DBA user and password here !!!!!!
            var authResult = adminDB.auth('USER', 'PASSWORD');
            if (authResult) {
                var buildInfo = adminDB.runCommand({ buildInfo: 1 });
                if (buildInfo.ok) {
                    printjson({
                        psmdbVersion: buildInfo.psmdbVersion || "N/A",
                        version: buildInfo.version || "N/A"
                    });
                } else {
                    print("Failed to get build info: " + JSON.stringify(buildInfo));
                }
            } else {
                print("Authentication failed for node: " + member.name);
            }
        } catch (error) {
            print("Failed to connect or execute command on node: " + member.name + " - " + JSON.stringify(error));
        }
    }
    
    print("");
});

The output should like below:

[1]: testdb:27017
{
  psmdbVersion: '7.0.12-7',
  version: '7.0.12-7'
}

[2]: testdb:27018
{
  psmdbVersion: '7.0.12-7',
  version: '7.0.12-7'
}

[3]: testdb:27019
{
  psmdbVersion: '7.0.12-7',
  version: '7.0.12-7'
}

In addition, can you please inform in what way PSMDB was installed (from which repo or tarball or custom build)? Thanks.

5

Installed package is
https://repo.percona.com/pdmdb-7.0.12/apt/pool/main/p/percona-server-mongodb/percona-server-mongodb_7.0.12-7.jammy_amd64.deb

Output:

[16]: (redacted):27018
{
  psmdbVersion: '7.0.12-7',
  version: '7.0.12-7'
}

[17]: (redacted):27018
{
  psmdbVersion: '7.0.12-7',
  version: '7.0.12-7'
}

[18]: (redacted):27018
{
  psmdbVersion: '7.0.12-7',
  version: '7.0.12-7'
}

Well if it can’t be reproduced, we’ll keep digging from our side. Maybe will wait for 2.6.1

Thank you anyway.

6

Unfortunately, newer PBM version won’t bring any changes unless the issue is rectified.

Each backup on storage has metadata in json format, for instance, for backup 2024-09-12T05:59:08Z the filename would be 2024-09-12T05_59_08Z.pbm.json. It holds MongoDB and PBM versions for each node inside RS when backup was created, as well as node name, can you please download this file and provide us versions saved during backup and also ensure that this backup was taken from the same RS as one in question (node name match)?

F.e.

grep -B1 mongo_version 2024-09-12T05_59_08Z.pbm.json 
			"pbm_version": "2.6.0",
			"mongo_version": "7.0.12-7"
--
			"pbm_version": "2.6.0",
			"mongo_version": "7.0.12-7"
--
			"pbm_version": "2.6.0",
			"mongo_version": "7.0.12-7"
7

This is the particular backup I was asking about:

grep -B1 mongo_version 2024-09-11T13_00_02Z.pbm.json
			"pbm_version": "2.6.0",
			"mongo_version": "6.0.16-13"

But this time I additionally checked backup from the day before aaand:

grep -B1 mongo_version 2024-09-10T13_00_02Z.pbm.json
			"pbm_version": "2.6.0",
			"mongo_version": "7.0.12-7"

pdbdm and pbm version haven’t changed from 9.10 to 9.11 and yet we have two different mongo_versions.
Very strange. Some kind of anomaly.
Let’s close this thread for now, we need more info. I’ll come back if it’s still an issue.

Thank you for your time!

1 Like