[feature requested] MongoDB/S3 beginner - backup fails - NoSuchKey?

[Zody]

Hey everyone,

actually taking some steps with MongoDB cause we need it for some applications - so in this case i’ve setup a 3 node replica set with pbm installed and configured to a s3 storage.

After the startup of pbm-agent there where this warning:

[agentCheckup] storage is not initialized

But i do configured within each node in ~/pbm_config.yaml the s3:

storage:
  type: s3
  s3:
    endpointUrl: "the-correct-url.com"
    region: fsn1
    bucket: mdb-prod-rs-pbm
    credentials:
      access-key-id: xxx
      secret-access-key: xxx

and applied it with:

pbm config --file pbm_config.yaml

Which returned with zero-return code.

I just tried to create a logical backup and thats the journal from the node where i run the command:

Dec 11 10:21:42 mdb-1 pbm-agent[13431]: 2024-12-11T10:21:42.000+0000 I got command backup [name: 2024-12-11T10:21:42Z, compression: s2 (level: default)] <ts: 1733912502>, opid: 675967b65e1919b46c0610d9
Dec 11 10:21:42 mdb-1 pbm-agent[13431]: 2024-12-11T10:21:42.000+0000 I got epoch {1733909575 2}
Dec 11 10:21:42 mdb-1 pbm-agent[13431]: 2024-12-11T10:21:42.000+0000 D [backup/2024-12-11T10:21:42Z] init backup meta
Dec 11 10:21:42 mdb-1 pbm-agent[13431]: 2024-12-11T10:21:42.000+0000 D [backup/2024-12-11T10:21:42Z] nomination list for mdb-prod-rs: [[mdb-2:27017 mdb-3:27017] [mdb-1:27017]]
Dec 11 10:21:42 mdb-1 pbm-agent[13431]: 2024-12-11T10:21:42.000+0000 D [backup/2024-12-11T10:21:42Z] nomination mdb-prod-rs, set candidates [mdb-2:27017 mdb-3:27017]
Dec 11 10:21:43 mdb-1 pbm-agent[13431]: 2024-12-11T10:21:43.000+0000 D [backup/2024-12-11T10:21:42Z] skip after nomination, probably started by another node
Dec 11 10:21:47 mdb-1 pbm-agent[13431]: 2024-12-11T10:21:47.000+0000 D [backup/2024-12-11T10:21:42Z] bcp nomination: mdb-prod-rs won by mdb-2:27017

while the journal from the mdb-2 shows:

Dec 11 10:21:42 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:42.000+0000 I got command backup [name: 2024-12-11T10:21:42Z, compression: s2 (level: default)] <ts: 1733912502>, opid: 675967b65e1919b46c0610d9
Dec 11 10:21:42 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:42.000+0000 I got epoch {1733909575 2}
Dec 11 10:21:43 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:43.000+0000 I [backup/2024-12-11T10:21:42Z] backup started
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.000+0000 D [backup/2024-12-11T10:21:42Z] wait for tmp users {1733912506 3}
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.000+0000 D [backup/2024-12-11T10:21:42Z] dumping up to 1 collections in parallel
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.228+0000        Setting num cpus to 2
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.246+0000        enqueued collection 'admin.pbmConfig'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.247+0000        enqueued collection 'admin.system.users'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.247+0000        enqueued collection 'admin.system.roles'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.248+0000        enqueued collection 'admin.pbmOpLog'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.248+0000        enqueued collection 'admin.pbmAgents'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.248+0000        enqueued collection 'admin.pbmPITRChunks'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.249+0000        enqueued collection 'admin.pbmLockOp'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.249+0000        enqueued collection 'admin.pbmRestores'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.250+0000        enqueued collection 'admin.pbmLock'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.250+0000        enqueued collection 'admin.pbmCmd'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.250+0000        enqueued collection 'admin.pbmRRoles'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.251+0000        enqueued collection 'admin.pbmPITR'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.251+0000        enqueued collection 'admin.pbmRUsers'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.252+0000        enqueued collection 'admin.pbmBackups'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.252+0000        enqueued collection 'admin.system.version'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.252+0000        enqueued collection 'admin.pbmLog'
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmOpLog
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmPITRChunks
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmLog
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmConfig
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmLockOp
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmRestores
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmLock
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmCmd
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmRRoles
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.260+0000        archive prelude admin.pbmPITR
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.261+0000        archive prelude admin.pbmRUsers
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.261+0000        archive prelude admin.pbmAgents
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.261+0000        archive prelude admin.pbmBackups
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.261+0000        archive prelude admin.system.users
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.261+0000        archive prelude admin.system.roles
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.261+0000        archive prelude admin.system.version
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmOpLog
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmPITRChunks
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmLog
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmConfig
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmLockOp
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmRestores
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmLock
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmCmd
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmRRoles
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmPITR
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmRUsers
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.278+0000        archive prelude admin.pbmAgents
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.279+0000        archive prelude admin.pbmBackups
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.279+0000        archive prelude admin.system.users
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.279+0000        archive prelude admin.system.roles
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.279+0000        archive prelude admin.system.version
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.291+0000        writing admin.system.users to archive on stdout
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.319+0000        Mux open namespace admin.system.users
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.320+0000        counted 2 documents in admin.system.users
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.321+0000        done dumping admin.system.users (2 documents)
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.321+0000        writing admin.system.roles to archive on stdout
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.system.users.s2" [size hint: 36864 (36.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.401+0000        Mux close namespace admin.system.users
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.401+0000        Mux open namespace admin.system.roles
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.402+0000        counted 1 document in admin.system.roles
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.404+0000        done dumping admin.system.roles (1 document)
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.408+0000        writing admin.system.version to archive on stdout
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.system.roles.s2" [size hint: 20480 (20.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.484+0000        Mux close namespace admin.system.roles
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.484+0000        Mux open namespace admin.system.version
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.485+0000        counted 2 documents in admin.system.version
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.490+0000        done dumping admin.system.version (2 documents)
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.490+0000        dumping up to 1 collections in parallel
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.494+0000        writing admin.pbmConfig to archive on stdout
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.system.version.s2" [size hint: 36864 (36.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.686+0000        Mux close namespace admin.system.version
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.686+0000        Mux open namespace admin.pbmConfig
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.687+0000        counted 1 document in admin.pbmConfig
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.688+0000        done dumping admin.pbmConfig (1 document)
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.691+0000        writing admin.pbmOpLog to archive on stdout
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmConfig.s2" [size hint: 36864 (36.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.984+0000        Mux close namespace admin.pbmConfig
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.984+0000        Mux open namespace admin.pbmOpLog
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.985+0000        counted 3 documents in admin.pbmOpLog
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.988+0000        done dumping admin.pbmOpLog (3 documents)
Dec 11 10:21:46 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.989+0000        writing admin.pbmAgents to archive on stdout
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:46.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmOpLog.s2" [size hint: 36864 (36.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.309+0000        Mux close namespace admin.pbmOpLog
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.309+0000        Mux open namespace admin.pbmAgents
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.310+0000        counted 3 documents in admin.pbmAgents
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.312+0000        done dumping admin.pbmAgents (3 documents)
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.312+0000        writing admin.pbmPITRChunks to archive on stdout
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmAgents.s2" [size hint: 36864 (36.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.832+0000        Mux close namespace admin.pbmAgents
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.832+0000        Mux open namespace admin.pbmPITRChunks
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.833+0000        counted 0 documents in admin.pbmPITRChunks
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.835+0000        done dumping admin.pbmPITRChunks (0 documents)
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.835+0000        Mux close namespace admin.pbmPITRChunks
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.836+0000        writing admin.pbmLockOp to archive on stdout
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.867+0000        Mux open namespace admin.pbmLockOp
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.868+0000        counted 0 documents in admin.pbmLockOp
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.868+0000        Mux close namespace admin.pbmLockOp
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.868+0000        done dumping admin.pbmLockOp (0 documents)
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.868+0000        writing admin.pbmRestores to archive on stdout
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.878+0000        Mux open namespace admin.pbmRestores
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.879+0000        counted 0 documents in admin.pbmRestores
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.879+0000        Mux close namespace admin.pbmRestores
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.879+0000        done dumping admin.pbmRestores (0 documents)
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.880+0000        writing admin.pbmLock to archive on stdout
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.888+0000        counted 1 document in admin.pbmLock
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.889+0000        Mux open namespace admin.pbmLock
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.889+0000        done dumping admin.pbmLock (1 document)
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.890+0000        writing admin.pbmCmd to archive on stdout
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmLock.s2" [size hint: 32768 (32.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.943+0000        Mux close namespace admin.pbmLock
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.943+0000        Mux open namespace admin.pbmCmd
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.944+0000        counted 4 documents in admin.pbmCmd
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.947+0000        done dumping admin.pbmCmd (4 documents)
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.948+0000        writing admin.pbmRRoles to archive on stdout
Dec 11 10:21:47 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:47.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmCmd.s2" [size hint: 36864 (36.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.040+0000        Mux close namespace admin.pbmCmd
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.040+0000        Mux open namespace admin.pbmRRoles
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.041+0000        counted 1 document in admin.pbmRRoles
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.045+0000        done dumping admin.pbmRRoles (1 document)
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.048+0000        writing admin.pbmPITR to archive on stdout
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmRRoles.s2" [size hint: 0 (0.00B); part size: 10485760 (10.00MB)]
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.549+0000        Mux close namespace admin.pbmRRoles
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.549+0000        Mux open namespace admin.pbmPITR
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.550+0000        counted 0 documents in admin.pbmPITR
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.551+0000        Mux close namespace admin.pbmPITR
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.552+0000        done dumping admin.pbmPITR (0 documents)
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.553+0000        writing admin.pbmRUsers to archive on stdout
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.560+0000        Mux open namespace admin.pbmRUsers
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.561+0000        counted 2 documents in admin.pbmRUsers
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.563+0000        done dumping admin.pbmRUsers (2 documents)
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.564+0000        writing admin.pbmBackups to archive on stdout
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmRUsers.s2" [size hint: 0 (0.00B); part size: 10485760 (10.00MB)]
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.627+0000        Mux close namespace admin.pbmRUsers
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.627+0000        Mux open namespace admin.pbmBackups
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.628+0000        counted 3 documents in admin.pbmBackups
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.630+0000        done dumping admin.pbmBackups (3 documents)
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.631+0000        writing admin.pbmLog to archive on stdout
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmBackups.s2" [size hint: 36864 (36.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.861+0000        Mux close namespace admin.pbmBackups
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.861+0000        Mux open namespace admin.pbmLog
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.862+0000        counted 241 documents in admin.pbmLog
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.867+0000        done dumping admin.pbmLog (241 documents)
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.867+0000        dump phase III: the oplog
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.867+0000        finishing dump
Dec 11 10:21:48 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:48.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/admin.pbmLog.s2" [size hint: 49152 (48.00KB); part size: 10485760 (10.00MB)]
Dec 11 10:21:49 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:49.230+0000        Mux close namespace admin.pbmLog
Dec 11 10:21:49 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:49.230+0000        Mux finish
Dec 11 10:21:49 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:49.230+0000        mux completed successfully
Dec 11 10:21:49 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:49.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/metadata.json" [size hint: 0 (0.00B); part size: 10485760 (10.00MB)]
Dec 11 10:21:49 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:49.000+0000 I [backup/2024-12-11T10:21:42Z] mongodump finished, waiting for the oplog
Dec 11 10:21:50 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:50.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z/mdb-prod-rs/oplog/20241211102143-4.20241211102150-1.s2" [size hint: -1 (unknown); part size: 10485760 (10.00MB)]
Dec 11 10:21:50 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:50.000+0000 I [backup/2024-12-11T10:21:42Z] created chunk 2024-12-11T10:21:43 - 2024-12-11T10:21:50
Dec 11 10:21:50 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:50.000+0000 I [backup/2024-12-11T10:21:42Z] dropping tmp collections
Dec 11 10:21:51 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:51.000+0000 D [backup/2024-12-11T10:21:42Z] uploading "2024-12-11T10:21:42Z.pbm.json" [size hint: -1 (unknown); part size: 10485760 (10.00MB)]
Dec 11 10:21:51 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:51.000+0000 E [backup/2024-12-11T10:21:42Z] Failed to delete leftover files for canceled backup "2024-12-11T10:21:42Z"
Dec 11 10:21:52 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:52.000+0000 I [backup/2024-12-11T10:21:42Z] mark RS as error `check backup files: list: NoSuchKey:
Dec 11 10:21:52 mdb-2 pbm-agent[12465]:         status code: 404, request id: tx00000dd8a2ceb3ef0f24e-00675967bf-87b470c-fsn1-prod1-ceph3, host id: `: <nil>
Dec 11 10:21:52 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:52.000+0000 I [backup/2024-12-11T10:21:42Z] mark backup as error `check backup files: list: NoSuchKey:
Dec 11 10:21:52 mdb-2 pbm-agent[12465]:         status code: 404, request id: tx00000dd8a2ceb3ef0f24e-00675967bf-87b470c-fsn1-prod1-ceph3, host id: `: <nil>
Dec 11 10:21:52 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:52.000+0000 E [backup/2024-12-11T10:21:42Z] backup: check backup files: list: NoSuchKey:
Dec 11 10:21:52 mdb-2 pbm-agent[12465]:         status code: 404, request id: tx00000dd8a2ceb3ef0f24e-00675967bf-87b470c-fsn1-prod1-ceph3, host id:
Dec 11 10:21:52 mdb-2 pbm-agent[12465]: 2024-12-11T10:21:52.000+0000 D [backup/2024-12-11T10:21:42Z] releasing lock

So i’ve checked the files within the bucket from provider ui and seeing this:

bucket mdb-prod-rs-pbm/mdb-prod-rs-pbm/

Name	Size
2024-12-11T09%3A35%3A45Z	-
2024-12-11T10%3A08%3A03Z	-
2024-12-11T10%3A21%3A42Z	-
.pbm.init	5 B
2024-12-11T09:35:45Z.pbm.json	1,77 KB
2024-12-11T10:08:03Z.pbm.json	1,77 KB
2024-12-11T10:21:42Z.pbm.json	1,76 KB

Unfortunately i cant find much to backup: check backup files: list: NoSuchKey: but isn’t it weird, that he reports Failed to delete leftover files for canceled backup "2024-12-11T10:21:42Z" but that seems to be named as 2024-12-11T09%3A35%3A45Z within the providers ui? I dont think it’s a bug in the ui while converting the name ^^…

As I am new to the subject, I am fishing in the dark and would appreciate some advice in debugging this

//Edit

S3 giving me NoSuchKey error even when the key exists

Gives me more and more the feeling the : within the backup name is the problem and now looking for a configuration option to change the name timestamp format without special chars.

//Edit2

How to specify backup name using Percona Backup Manager on bare metal servers

Hi there, We are revising our backup strategy, where we want to take backups start of every month, weekly backups and daily backup. Is there any way we can specify the backup name? i.e daily_{DATE}, weekly_{DATE}, monthly_{DATE} We want to keep last 4 weeks backup and last 6 monthly backups, if we happen to use pbm cleanup command, it will remove all backups older than specific date. Thanks.

I found some feature requests, but they never had the reason while there where special chars. So here it is:
https://perconadev.atlassian.net/browse/PBM-1462

Best regards

[Ivan_Groenewold]

Hi, indeed this seems to be a problem with your storage provider converting some characters. Thanks for the FR, dev team will look into this.

[Zody]

For completeness, I would like to briefly mention that the provider reported back that special characters, especially the colon, should NOT cause any problems. The provider’s FAQ also confirms this.

From now on I’ll continue writing in the FR.