PBM-agent on docker running into "storage is not initialized"

MongoDB Percona Backup for MongoDB
,
1

Hi,

I’ve tried to go trough docs, but didn’t find correct information. I’ve selected to build a image on which I will be able to run PBM backups.

  1. Guide in Run in Docker - Percona Backup for MongoDB, contains wrong information about the images & packages.
    The “example” doesn’t contain required packages:
    pbm: /lib64/libc.so.6: version GLIBC_2.32' not found (required by pbm) pbm: /lib64/libc.so.6: version GLIBC_2.34’ not found (required by pbm)

So I decided to build image using following docker file

FROM percona/percona-server-mongodb:latest AS mdb
FROM percona/percona-backup-mongodb:latest AS pbm

FROM debian:bookworm-slim

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    libgssapi-krb5-2 \
    libkrb5-3 \
    libcurl4 \
    ca-certificates \
    libldap-common \
    ldap-utils \
    && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /data/db /data/configdb

COPY --from=mdb /usr/bin/mongod /usr/bin/
COPY --from=pbm /usr/bin/pbm* /usr/bin/

CMD ["tail", "-f", "/dev/null"]
  1. Container with the image deployed, PBM started using pbm config --file /etc/percona-backup/pbm_config.yaml`

This is the what file contains:
storage:
type: filesystem
filesystem:
path: /backups
I see that PBM started OK, but now I need to perform backup.
Started 3 processes for all MongoDB nodes on the same docker image using:
'pbm-agent --mongodb-uri “mongodb://XXX:XXX@mongodb-0.mongodb:27017/?authSource=admin&replicaSet=rs0”

Unfortunately, pbm-agent returns:
2025-04-08T12:30:53Z W [rs0/mongodb-0.mongodb:27017] [agentCheckup] storage is not initialized

pbm-agent doesn’t offer --config flag, are you able to explain why I am getting " storage is not initialized"?

2

Are you sure you have started you MongoDB nodes in the ReplicaSet topology?

3

Yes, MongoDB server works in ReplicaSet.

4

Hi, the Example docker image uses oraclelinux:8 image which includes the dependencies so it does not contain wrong information. In your case you have configured a slim image which comes with minimum packages so it is expected you are missing some dependencies.
Now related to your error, PBM requires a shared storage (all nodes need to see the same volume or bucket) please check the steps to add a remote storage for more details.

5

Percona images are based on redhat/ubi9-minimal. Could you replace debian image by oraclelinux:9 ?

Screenshot 2025-04-08 at 16.59.21
Screenshot 2025-04-08 at 16.59.211387×677 78.7 KB

6

My MakeFile:

FROM percona/percona-server-mongodb:latest AS mdb
FROM percona/percona-backup-mongodb:latest AS pbm

FROM oraclelinux:9

RUN mkdir -p /data/db /data/configdb

COPY --from=mdb /usr/bin/mongod /usr/bin/
COPY --from=pbm /usr/bin/pbm* /usr/bin/

CMD ["tail", "-f", "/dev/null"]

My pbm_config.yaml → 

storage:
  type: filesystem
  filesystem:
    path: /backups

Note: /backups is a directory, mounted as PV → ~10Gi

pbm status command result:

[root@percona-backup-mongodb-6c47d4555c-bh2gn /]# pbm status
Cluster:
========
rs0:
  - mongodb-0.mongodb.svc.cluster.local.:27017 [P]: pbm-agent [v2.9.1] OK
  - mongodb-1.mongodb.svc.cluster.local.:27017 [S]: pbm-agent [v2.9.1] OK
  - mongodb-2.mongodb.svc.cluster.local.:27017 [S]: pbm-agent [v2.9.1] OK


PITR incremental backup:
========================
Status [OFF]

Currently running:
==================
(none)

Backups:
========
FS  /backups
  Snapshots:
    2025-04-09T06:13:41Z 0.00B <physical> [ERROR: Backup stuck at `starting` stage, last beat ts: 1744179222] [2025-04-09T06:13:42Z]

pbm config:

[root@percona-backup-mongodb-6c47d4555c-bh2gn /]# pbm config
storage:
  type: filesystem
  filesystem:
    path: /backups
pitr:
  enabled: false
  compression: s2
backup:
  oplogSpanMin: 0
  compression: s2
restore: {}

pbm-agent logs from the backup attempt:

2025-04-09T13:22:00.000+0000 I got command backup [name: 2025-04-09T13:21:59Z, compression: s2 (level: default)] <ts: 1744204919>, opid: 67f674775adaf96af16f5d86

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got epoch {1744179172 5}

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 E [backup/2025-04-09T13:21:59Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I [backup/2025-04-09T13:21:59Z] node is not suitable for backup

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 D [backup/2025-04-09T13:21:59Z] init backup meta

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 D [backup/2025-04-09T13:21:59Z] nomination list for rs0: []

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got command backup [name: 2025-04-09T13:21:59Z, compression: s2 (level: default)] <ts: 1744204919>, opid: 67f674775adaf96af16f5d86

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got command backup [name: 2025-04-09T13:21:59Z, compression: s2 (level: default)] <ts: 1744204919>, opid: 67f674775adaf96af16f5d86

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got epoch {1744179172 5}

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got epoch {1744179172 5}

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 E [backup/2025-04-09T13:21:59Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 E [backup/2025-04-09T13:21:59Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:22:09 pm2025-04-09T13:22:09.000+0000 W [agentCheckup] storage is not initialized

Wed, Apr 9 2025 3:22:14 pm2025-04-09T13:22:14.000+0000 W [agentCheckup] storage is not initialized

Wed, Apr 9 2025 3:22:14 pm2025-04-09T13:22:14.000+0000 W [agentCheckup] storage is not initialized

Wed, Apr 9 2025 3:22:15 pm2025-04-09T13:22:15.000+0000 D [backup/2025-04-09T13:21:59Z] nomination timeout

Wed, Apr 9 2025 3:22:15 pm2025-04-09T13:22:15.000+0000 D [backup/2025-04-09T13:21:59Z] skip after nomination, probably started by another node

Wed, Apr 9 2025 3:22:29 pm2025-04-09T13:22:29.000+0000 W [agentCheckup] storage is not initialized

Wed, Apr 9 2025 3:22:29 pm2025-04-09T13:22:29.000+0000 W [agentCheckup] storage is not initialized

Note: MongoDB runs in replicaSet, version of Mongodb: v7.0.5

7

It looks for me that pbm-agent either lacks pbm-agent --config (same as pbm) or lacks access to the storage.

But I think you could introduce pbm-agent config to check what storage agent has configured.
Or it’s some different issue (my configuration problem) which is causing this?

8

I’ve solved problem with storage.

There must be a delay between setting pbm config file and starting pbm-agents. Here is a script snip:

              echo "Setting up storage"
              pbm config --file /etc/percona-backup/pbm_config.yaml
              echo "Waiting 30 seconds before starting pbm-agents..."
              sleep 30
              echo "Setting up pbm-agent 1"

Thanks to that, there is no longer an error about storage.

But, I’m unable to perform backup:

2025-04-09T13:36:41.000+0000 I got command backup [name: 2025-04-09T13:36:41Z, compression: s2 (level: default)] <ts: 1744205801>, opid: 67f677e9d56c927f4c86be26

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 I got command backup [name: 2025-04-09T13:36:41Z, compression: s2 (level: default)] <ts: 1744205801>, opid: 67f677e9d56c927f4c86be26

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 I got epoch {1744205727 3} 2025-04-09T13:36:41.177142031Z 2025-04-09T13:36:41.000+0000 I got epoch {1744205727 3}

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 E [backup/2025-04-09T13:36:41Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 E [backup/2025-04-09T13:36:41Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 I [backup/2025-04-09T13:36:41Z] node is not suitable for backup

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 D [backup/2025-04-09T13:36:41Z] init backup meta

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 D [backup/2025-04-09T13:36:41Z] nomination list for rs0: []

Wed, Apr 9 2025 3:36:42 pm2025-04-09T13:36:42.000+0000 I got command backup [name: 2025-04-09T13:36:41Z, compression: s2 (level: default)] <ts: 1744205801>, opid: 67f677e9d56c927f4c86be26

Wed, Apr 9 2025 3:36:42 pm2025-04-09T13:36:42.000+0000 I got epoch {1744205727 3}

Wed, Apr 9 2025 3:36:42 pm2025-04-09T13:36:42.000+0000 E [backup/2025-04-09T13:36:41Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:36:56 pm2025-04-09T13:36:56.000+0000 D [backup/2025-04-09T13:36:41Z] nomination timeout

Wed, Apr 9 2025 3:36:56 pm2025-04-09T13:36:56.000+0000 D [backup/2025-04-09T13:36:41Z] skip after nomination, probably started by another node

Does it mean that I’ve to setup Percona Server on cluster, click in the Percona Server UI to perform backup with specific I don’t know pod/container/server agent and then I should be able to perform backups?

9

And - I’ve solved final problem.

So, in my setup pbm backup --type=logical works.
But for pbm backup --type=physical → I’m getting error Wed, Apr 9 2025 3:36:42 pm2025-04-09T13:36:42.000+0000 E [backup/2025-04-09T13:36:41Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Which tells me that I need Percona Server to perform physical backup. I think it’s missing in your docs.

10

Seems PBM cannot retrieve the version from MongoDB. Most likely gets empty string and interprets as the old version. Can you check with mongosh:

db.runCommand(
   {
     buildInfo: 1
   }
)
11

Nothing special in return, but still - logical backup works, while physical doesn’t.

{
  version: '7.0.8',
  gitVersion: 'c5d33e55ba38d98e2f48765ec4e55338d67a4a64',
  modules: [],
  allocator: 'tcmalloc',
  javascriptEngine: 'mozjs',
  sysInfo: 'deprecated',
  versionArray: [ 7, 0, 8, 0 ],
  openssl: {
    running: 'OpenSSL 3.0.11 19 Sep 2023',
    compiled: 'OpenSSL 3.0.9 30 May 2023'
  },
  buildEnvironment: {
    distmod: 'debian12',
    distarch: 'x86_64',
    cc: '/opt/mongodbtoolchain/v4/bin/gcc: gcc (GCC) 11.3.0',
    ccflags: '-Werror -include mongo/platform/basic.h -ffp-contract=off -fasynchronous-unwind-tables -g2 -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -gdwarf-5 -fno-omit-frame-pointer -fno-strict-aliasing -O2 -march=sandybridge -mtune=generic -mprefer-vector-width=128 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-const-variable -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -gdwarf64 -Wa,--nocompress-debug-sections -fno-builtin-memcmp -Wimplicit-fallthrough=5',
    cxx: '/opt/mongodbtoolchain/v4/bin/g++: g++ (GCC) 11.3.0',
    cxxflags: '-Woverloaded-virtual -Wpessimizing-move -Wno-maybe-uninitialized -fsized-deallocation -Wno-deprecated -std=c++20',
    linkflags: '-Wl,--fatal-warnings -B/opt/mongodbtoolchain/v4/bin -gdwarf-5 -pthread -Wl,-z,now -fuse-ld=lld -fstack-protector-strong -gdwarf64 -Wl,--build-id -Wl,--hash-style=gnu -Wl,-z,noexecstack -Wl,--warn-execstack -Wl,-z,relro -Wl,--compress-debug-sections=none -Wl,-z,origin -Wl,--enable-new-dtags',
    target_arch: 'x86_64',
    target_os: 'linux',
    cppdefines: 'SAFEINT_USE_INTRINSICS 0 PCRE2_STATIC NDEBUG _XOPEN_SOURCE 700 _GNU_SOURCE _FORTIFY_SOURCE 2 ABSL_FORCE_ALIGNED_ACCESS BOOST_ENABLE_ASSERT_DEBUG_HANDLER BOOST_FILESYSTEM_NO_CXX20_ATOMIC_REF BOOST_LOG_NO_SHORTHAND_NAMES BOOST_LOG_USE_NATIVE_SYSLOG BOOST_LOG_WITHOUT_THREAD_ATTR BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS BOOST_SYSTEM_NO_DEPRECATED BOOST_THREAD_USES_DATETIME BOOST_THREAD_VERSION 5'
  },
  bits: 64,
  debug: false,
  maxBsonObjectSize: 16777216,
  storageEngines: [ 'devnull', 'wiredTiger' ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1744970998, i: 3 }),
    signature: {
      hash: Binary.createFromBase64('M00heSbr3iAsV0QJXgPdGT38oh4=', 0),
      keyId: Long('7482819781188386822')
    }
  },
  operationTime: Timestamp({ t: 1744970998, i: 3 })
}

[root@percona-backup-mongodb-c44dbcdcf-cmb4r /]# pbm backup --type=logical
Starting backup '2025-04-18T10:05:45Z'....Backup '2025-04-18T10:05:45Z' to remote store '/backups/'
[root@percona-backup-mongodb-c44dbcdcf-cmb4r /]# pbm backup --type=physical
Starting backup '2025-04-18T10:08:52Z'.................................Error: get backup metadata: get: context deadline exceeded

2025-04-18T10:08:53.000+0000 I got command backup [name: 2025-04-18T10:08:52Z, compression: s2 (level: default)] <ts: 1744970932>, opid: 680224b495032062a1d65208
2025-04-18T10:08:53.000+0000 I got epoch {1744970633 5}
2025-04-18T10:08:53.000+0000 I got command backup [name: 2025-04-18T10:08:52Z, compression: s2 (level: default)] <ts: 1744970932>, opid: 680224b495032062a1d65208
2025-04-18T10:08:53.000+0000 I got epoch {1744970633 5}
2025-04-18T10:08:53.085898089Z 2025-04-18T10:08:53.000+0000 E [backup/2025-04-18T10:08:52Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6
2025-04-18T10:08:53.000+0000 I [backup/2025-04-18T10:08:52Z] node is not suitable for backup
2025-04-18T10:08:53.000+0000 E [backup/2025-04-18T10:08:52Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6
2025-04-18T10:08:53.000+0000 I got command backup [name: 2025-04-18T10:08:52Z, compression: s2 (level: default)] <ts: 1744970932>, opid: 680224b495032062a1d65208
2025-04-18T10:08:53.000+0000 D [backup/2025-04-18T10:08:52Z] init backup meta
2025-04-18T10:08:53.000+0000 I got epoch {1744970633 5}
2025-04-18T10:08:53.000+0000 E [backup/2025-04-18T10:08:52Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6
2025-04-18T10:08:53.000+0000 D [backup/2025-04-18T10:08:52Z] nomination list for rs0: []
2025-04-18T10:09:08.000+0000 D [backup/2025-04-18T10:08:52Z] nomination timeout
2025-04-18T10:09:08.000+0000 D [backup/2025-04-18T10:08:52Z] skip after nomination, probably started by another node