PBM-agent on docker running into "storage is not initialized"

Hi,

I’ve tried to go trough docs, but didn’t find correct information. I’ve selected to build a image on which I will be able to run PBM backups.

  1. Guide in Run in Docker - Percona Backup for MongoDB, contains wrong information about the images & packages.
    The “example” doesn’t contain required packages:
    pbm: /lib64/libc.so.6: version GLIBC_2.32' not found (required by pbm) pbm: /lib64/libc.so.6: version GLIBC_2.34’ not found (required by pbm)

So I decided to build image using following docker file

FROM percona/percona-server-mongodb:latest AS mdb
FROM percona/percona-backup-mongodb:latest AS pbm

FROM debian:bookworm-slim

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    libgssapi-krb5-2 \
    libkrb5-3 \
    libcurl4 \
    ca-certificates \
    libldap-common \
    ldap-utils \
    && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /data/db /data/configdb

COPY --from=mdb /usr/bin/mongod /usr/bin/
COPY --from=pbm /usr/bin/pbm* /usr/bin/

CMD ["tail", "-f", "/dev/null"]
  1. Container with the image deployed, PBM started using pbm config --file /etc/percona-backup/pbm_config.yaml`

This is the what file contains:
storage:
type: filesystem
filesystem:
path: /backups
I see that PBM started OK, but now I need to perform backup.
Started 3 processes for all MongoDB nodes on the same docker image using:
'pbm-agent --mongodb-uri “mongodb://XXX:XXX@mongodb-0.mongodb:27017/?authSource=admin&replicaSet=rs0”

Unfortunately, pbm-agent returns:
2025-04-08T12:30:53Z W [rs0/mongodb-0.mongodb:27017] [agentCheckup] storage is not initialized

pbm-agent doesn’t offer --config flag, are you able to explain why I am getting " storage is not initialized"?

Are you sure you have started you MongoDB nodes in the ReplicaSet topology?

Yes, MongoDB server works in ReplicaSet.

Hi, the Example docker image uses oraclelinux:8 image which includes the dependencies so it does not contain wrong information. In your case you have configured a slim image which comes with minimum packages so it is expected you are missing some dependencies.
Now related to your error, PBM requires a shared storage (all nodes need to see the same volume or bucket) please check the steps to add a remote storage for more details.

Percona images are based on redhat/ubi9-minimal. Could you replace debian image by oraclelinux:9 ?

My MakeFile:

FROM percona/percona-server-mongodb:latest AS mdb
FROM percona/percona-backup-mongodb:latest AS pbm

FROM oraclelinux:9

RUN mkdir -p /data/db /data/configdb

COPY --from=mdb /usr/bin/mongod /usr/bin/
COPY --from=pbm /usr/bin/pbm* /usr/bin/

CMD ["tail", "-f", "/dev/null"]

My pbm_config.yaml →

storage:
  type: filesystem
  filesystem:
    path: /backups

Note: /backups is a directory, mounted as PV → ~10Gi

pbm status command result:

[root@percona-backup-mongodb-6c47d4555c-bh2gn /]# pbm status
Cluster:
========
rs0:
  - mongodb-0.mongodb.svc.cluster.local.:27017 [P]: pbm-agent [v2.9.1] OK
  - mongodb-1.mongodb.svc.cluster.local.:27017 [S]: pbm-agent [v2.9.1] OK
  - mongodb-2.mongodb.svc.cluster.local.:27017 [S]: pbm-agent [v2.9.1] OK


PITR incremental backup:
========================
Status [OFF]

Currently running:
==================
(none)

Backups:
========
FS  /backups
  Snapshots:
    2025-04-09T06:13:41Z 0.00B <physical> [ERROR: Backup stuck at `starting` stage, last beat ts: 1744179222] [2025-04-09T06:13:42Z]

pbm config:

[root@percona-backup-mongodb-6c47d4555c-bh2gn /]# pbm config
storage:
  type: filesystem
  filesystem:
    path: /backups
pitr:
  enabled: false
  compression: s2
backup:
  oplogSpanMin: 0
  compression: s2
restore: {}

pbm-agent logs from the backup attempt:

2025-04-09T13:22:00.000+0000 I got command backup [name: 2025-04-09T13:21:59Z, compression: s2 (level: default)] <ts: 1744204919>, opid: 67f674775adaf96af16f5d86

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got epoch {1744179172 5}

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 E [backup/2025-04-09T13:21:59Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I [backup/2025-04-09T13:21:59Z] node is not suitable for backup

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 D [backup/2025-04-09T13:21:59Z] init backup meta

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 D [backup/2025-04-09T13:21:59Z] nomination list for rs0: []

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got command backup [name: 2025-04-09T13:21:59Z, compression: s2 (level: default)] <ts: 1744204919>, opid: 67f674775adaf96af16f5d86

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got command backup [name: 2025-04-09T13:21:59Z, compression: s2 (level: default)] <ts: 1744204919>, opid: 67f674775adaf96af16f5d86

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got epoch {1744179172 5}

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 I got epoch {1744179172 5}

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 E [backup/2025-04-09T13:21:59Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:22:00 pm2025-04-09T13:22:00.000+0000 E [backup/2025-04-09T13:21:59Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:22:09 pm2025-04-09T13:22:09.000+0000 W [agentCheckup] storage is not initialized

Wed, Apr 9 2025 3:22:14 pm2025-04-09T13:22:14.000+0000 W [agentCheckup] storage is not initialized

Wed, Apr 9 2025 3:22:14 pm2025-04-09T13:22:14.000+0000 W [agentCheckup] storage is not initialized

Wed, Apr 9 2025 3:22:15 pm2025-04-09T13:22:15.000+0000 D [backup/2025-04-09T13:21:59Z] nomination timeout

Wed, Apr 9 2025 3:22:15 pm2025-04-09T13:22:15.000+0000 D [backup/2025-04-09T13:21:59Z] skip after nomination, probably started by another node

Wed, Apr 9 2025 3:22:29 pm2025-04-09T13:22:29.000+0000 W [agentCheckup] storage is not initialized

Wed, Apr 9 2025 3:22:29 pm2025-04-09T13:22:29.000+0000 W [agentCheckup] storage is not initialized

Note: MongoDB runs in replicaSet, version of Mongodb: v7.0.5

It looks for me that pbm-agent either lacks pbm-agent --config (same as pbm) or lacks access to the storage.

But I think you could introduce pbm-agent config to check what storage agent has configured.
Or it’s some different issue (my configuration problem) which is causing this?

I’ve solved problem with storage.

There must be a delay between setting pbm config file and starting pbm-agents. Here is a script snip:

              echo "Setting up storage"
              pbm config --file /etc/percona-backup/pbm_config.yaml
              echo "Waiting 30 seconds before starting pbm-agents..."
              sleep 30
              echo "Setting up pbm-agent 1"

Thanks to that, there is no longer an error about storage.

But, I’m unable to perform backup:

2025-04-09T13:36:41.000+0000 I got command backup [name: 2025-04-09T13:36:41Z, compression: s2 (level: default)] <ts: 1744205801>, opid: 67f677e9d56c927f4c86be26

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 I got command backup [name: 2025-04-09T13:36:41Z, compression: s2 (level: default)] <ts: 1744205801>, opid: 67f677e9d56c927f4c86be26

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 I got epoch {1744205727 3} 2025-04-09T13:36:41.177142031Z 2025-04-09T13:36:41.000+0000 I got epoch {1744205727 3}

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 E [backup/2025-04-09T13:36:41Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 E [backup/2025-04-09T13:36:41Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 I [backup/2025-04-09T13:36:41Z] node is not suitable for backup

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 D [backup/2025-04-09T13:36:41Z] init backup meta

Wed, Apr 9 2025 3:36:41 pm2025-04-09T13:36:41.000+0000 D [backup/2025-04-09T13:36:41Z] nomination list for rs0: []

Wed, Apr 9 2025 3:36:42 pm2025-04-09T13:36:42.000+0000 I got command backup [name: 2025-04-09T13:36:41Z, compression: s2 (level: default)] <ts: 1744205801>, opid: 67f677e9d56c927f4c86be26

Wed, Apr 9 2025 3:36:42 pm2025-04-09T13:36:42.000+0000 I got epoch {1744205727 3}

Wed, Apr 9 2025 3:36:42 pm2025-04-09T13:36:42.000+0000 E [backup/2025-04-09T13:36:41Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Wed, Apr 9 2025 3:36:56 pm2025-04-09T13:36:56.000+0000 D [backup/2025-04-09T13:36:41Z] nomination timeout

Wed, Apr 9 2025 3:36:56 pm2025-04-09T13:36:56.000+0000 D [backup/2025-04-09T13:36:41Z] skip after nomination, probably started by another node

Does it mean that I’ve to setup Percona Server on cluster, click in the Percona Server UI to perform backup with specific I don’t know pod/container/server agent and then I should be able to perform backups?

And - I’ve solved final problem.

So, in my setup pbm backup --type=logical works.
But for pbm backup --type=physical → I’m getting error Wed, Apr 9 2025 3:36:42 pm2025-04-09T13:36:42.000+0000 E [backup/2025-04-09T13:36:41Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6

Which tells me that I need Percona Server to perform physical backup. I think it’s missing in your docs.

Seems PBM cannot retrieve the version from MongoDB. Most likely gets empty string and interprets as the old version. Can you check with mongosh:

db.runCommand(
   {
     buildInfo: 1
   }
)

Nothing special in return, but still - logical backup works, while physical doesn’t.

{
  version: '7.0.8',
  gitVersion: 'c5d33e55ba38d98e2f48765ec4e55338d67a4a64',
  modules: [],
  allocator: 'tcmalloc',
  javascriptEngine: 'mozjs',
  sysInfo: 'deprecated',
  versionArray: [ 7, 0, 8, 0 ],
  openssl: {
    running: 'OpenSSL 3.0.11 19 Sep 2023',
    compiled: 'OpenSSL 3.0.9 30 May 2023'
  },
  buildEnvironment: {
    distmod: 'debian12',
    distarch: 'x86_64',
    cc: '/opt/mongodbtoolchain/v4/bin/gcc: gcc (GCC) 11.3.0',
    ccflags: '-Werror -include mongo/platform/basic.h -ffp-contract=off -fasynchronous-unwind-tables -g2 -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -gdwarf-5 -fno-omit-frame-pointer -fno-strict-aliasing -O2 -march=sandybridge -mtune=generic -mprefer-vector-width=128 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-const-variable -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -gdwarf64 -Wa,--nocompress-debug-sections -fno-builtin-memcmp -Wimplicit-fallthrough=5',
    cxx: '/opt/mongodbtoolchain/v4/bin/g++: g++ (GCC) 11.3.0',
    cxxflags: '-Woverloaded-virtual -Wpessimizing-move -Wno-maybe-uninitialized -fsized-deallocation -Wno-deprecated -std=c++20',
    linkflags: '-Wl,--fatal-warnings -B/opt/mongodbtoolchain/v4/bin -gdwarf-5 -pthread -Wl,-z,now -fuse-ld=lld -fstack-protector-strong -gdwarf64 -Wl,--build-id -Wl,--hash-style=gnu -Wl,-z,noexecstack -Wl,--warn-execstack -Wl,-z,relro -Wl,--compress-debug-sections=none -Wl,-z,origin -Wl,--enable-new-dtags',
    target_arch: 'x86_64',
    target_os: 'linux',
    cppdefines: 'SAFEINT_USE_INTRINSICS 0 PCRE2_STATIC NDEBUG _XOPEN_SOURCE 700 _GNU_SOURCE _FORTIFY_SOURCE 2 ABSL_FORCE_ALIGNED_ACCESS BOOST_ENABLE_ASSERT_DEBUG_HANDLER BOOST_FILESYSTEM_NO_CXX20_ATOMIC_REF BOOST_LOG_NO_SHORTHAND_NAMES BOOST_LOG_USE_NATIVE_SYSLOG BOOST_LOG_WITHOUT_THREAD_ATTR BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS BOOST_SYSTEM_NO_DEPRECATED BOOST_THREAD_USES_DATETIME BOOST_THREAD_VERSION 5'
  },
  bits: 64,
  debug: false,
  maxBsonObjectSize: 16777216,
  storageEngines: [ 'devnull', 'wiredTiger' ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1744970998, i: 3 }),
    signature: {
      hash: Binary.createFromBase64('M00heSbr3iAsV0QJXgPdGT38oh4=', 0),
      keyId: Long('7482819781188386822')
    }
  },
  operationTime: Timestamp({ t: 1744970998, i: 3 })
}
[root@percona-backup-mongodb-c44dbcdcf-cmb4r /]# pbm backup --type=logical
Starting backup '2025-04-18T10:05:45Z'....Backup '2025-04-18T10:05:45Z' to remote store '/backups/'
[root@percona-backup-mongodb-c44dbcdcf-cmb4r /]# pbm backup --type=physical
Starting backup '2025-04-18T10:08:52Z'.................................Error: get backup metadata: get: context deadline exceeded
2025-04-18T10:08:53.000+0000 I got command backup [name: 2025-04-18T10:08:52Z, compression: s2 (level: default)] <ts: 1744970932>, opid: 680224b495032062a1d65208
2025-04-18T10:08:53.000+0000 I got epoch {1744970633 5}
2025-04-18T10:08:53.000+0000 I got command backup [name: 2025-04-18T10:08:52Z, compression: s2 (level: default)] <ts: 1744970932>, opid: 680224b495032062a1d65208
2025-04-18T10:08:53.000+0000 I got epoch {1744970633 5}
2025-04-18T10:08:53.085898089Z 2025-04-18T10:08:53.000+0000 E [backup/2025-04-18T10:08:52Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6
2025-04-18T10:08:53.000+0000 I [backup/2025-04-18T10:08:52Z] node is not suitable for backup
2025-04-18T10:08:53.000+0000 E [backup/2025-04-18T10:08:52Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6
2025-04-18T10:08:53.000+0000 I got command backup [name: 2025-04-18T10:08:52Z, compression: s2 (level: default)] <ts: 1744970932>, opid: 680224b495032062a1d65208
2025-04-18T10:08:53.000+0000 D [backup/2025-04-18T10:08:52Z] init backup meta
2025-04-18T10:08:53.000+0000 I got epoch {1744970633 5}
2025-04-18T10:08:53.000+0000 E [backup/2025-04-18T10:08:52Z] node check: full physical backup works since Percona Server for MongoDB 4.2.15, 4.4.6
2025-04-18T10:08:53.000+0000 D [backup/2025-04-18T10:08:52Z] nomination list for rs0: []
2025-04-18T10:09:08.000+0000 D [backup/2025-04-18T10:08:52Z] nomination timeout
2025-04-18T10:09:08.000+0000 D [backup/2025-04-18T10:08:52Z] skip after nomination, probably started by another node