Backup failed in 2.3.1 version

, ,
1

I use percona packup mongodb 2.3.1. This is my cluster:

*pbm status*
*Cluster:*
*========*
*cfg:*
*  - cfg/mongo-gridfs-config-1:27019 [S]: pbm-agent v2.3.1 OK*
*  - cfg/mongo-gridfs-config-2:27019 [P]: pbm-agent v2.3.1 OK*
*  - cfg/mongo-gridfs-config-3:27019 [S]: pbm-agent v2.3.1 OK*
*  - cfg/gorodec-mongo-gridfs-config-1:27019 [S]: pbm-agent v2.3.1 OK*
*  - cfg/gorodec-mongo-gridfs-config-2:27019 [S]: pbm-agent v2.3.1 OK*
*  - cfg/gorodec-mongo-gridfs-config-3:27019 [S]: pbm-agent v2.3.1 OK*
*rs0:*
*  - rs0/mongo-gridfs-1:27017 [S]: pbm-agent v2.3.1 OK*
*  - rs0/mongo-gridfs-2:27017 [S]: pbm-agent v2.3.1 OK*
*  - rs0/mongo-gridfs-3:27017 [P]: pbm-agent v2.3.1 OK*
*  - rs0/gorodec-mongo-gridfs-1:27017 [S]: pbm-agent v2.3.1 OK*
*  - rs0/gorodec-mongo-gridfs-2:27017 [S]: pbm-agent v2.3.1 OK*
*  - rs0/gorodec-mongo-gridfs-3:27017 [S]: pbm-agent v2.3.1 OK*
*  *
*PITR incremental backup:*
*========================*
*Status [ON]*

My config:

storage:
  type: filesystem
  filesystem:
    path: /u01/backups
pitr:
  enabled: true
  oplogSpanMin: 10
backup:
  priority:
    "mongo-gridfs-1:27017": 5.0
    "mongo-gridfs-2:27017": 5.0
    "mongo-gridfs-3:27017": 5.0
    "mongo-gridfs-config-1:27019": 5.0
    "mongo-gridfs-config-2:27019": 5.0
    "mongo-gridfs-config-3:27019": 5.0
    "gorodec-mongo-gridfs-1:27017": 0.1
    "gorodec-mongo-gridfs-2:27017": 0.1
    "gorodec-mongo-gridfs-3:27017": 0.1
    "gorodec-mongo-gridfs-config-1:27019": 0.1
    "gorodec-mongo-gridfs-config-2:27019": 0.1
    "gorodec-mongo-gridfs-config-3:27019": 0.1

I run full backup once a week:

*sudo vim /etc/cleanup.bash*
*pbm cleanup --older-than=31d --yes*

*sudo vim /etc/cron.d/pbm-backup*

*0 0 * * sat root . /etc/sysconfig/pbm-cron; /usr/bin/pbm backup --compression=s2*
*0 1 * * * root . /etc/sysconfig/pbm-cron; bash /etc/cleanup.bash*

During backup process I was watching the process, I saw new files in backup direcory. Also there were successfull pbm-agent logs:

*Mar 02 12:33:59 mongo-gridfs-3 pbm-agent[118444]: 2024/03/02 12:33:59 [########################]  file-store.fs.chunks  44097919/44089756  (100.0%)*
*Mar 02 12:33:59 mongo-gridfs-3 pbm-agent[118444]: 2024-03-02T12:33:59.599+0300        done dumping file-store.fs.chunks (44097919 documents)*
*Mar 02 12:33:59 mongo-gridfs-3 pbm-agent[118444]: 2024-03-02T12:33:59.599+0300        Mux close namespace file-store.fs.chunks*
*Mar 02 12:33:59 mongo-gridfs-3 pbm-agent[118444]: 2024-03-02T12:33:59.603+0300        dump phase III: the oplog*
*Mar 02 12:33:59 mongo-gridfs-3 pbm-agent[118444]: 2024-03-02T12:33:59.603+0300        finishing dump*

After full backup there were only metadata.json in the backup directory:

*ll 2024-03-01T21:00:01Z/rs0*
*total 32*
*-rw-r--r-- 1 pbm cgred 32744 Mar  2 12:33 metadata.json*

Also I couldn’t check backup status:

*$ pbm status*
*Error: get status of running: get backup info: not found*

There are no information about this backup in admin.pbmBackups collection.
I found pbm logs:

 *pbm logs --tail=100 --event=backup/2024-03-01T21:00:01Z*
*2024-03-01T21:00:02Z I [cfg/mongo-gridfs-config-2:27019] [backup/2024-03-01T21:00:01Z] backup started*
*2024-03-01T21:00:02Z I [rs0/mongo-gridfs-3:27017] [backup/2024-03-01T21:00:01Z] backup started*
*2024-03-01T21:00:07Z I [cfg/mongo-gridfs-config-2:27019] [backup/2024-03-01T21:00:01Z] mongodump finished, waiting for the oplog*
*2024-03-01T22:00:25Z I [cfg/mongo-gridfs-config-2:27019] [backup/2024-03-01T21:00:01Z] dropping tmp collections*
*2024-03-01T22:00:25Z I [cfg/mongo-gridfs-config-2:27019] [backup/2024-03-01T21:00:01Z] mark RS as error `check cluster for dump done: convergeCluster: get backup metadata: not found`: <nil>*
*2024-03-01T22:00:26Z I [cfg/mongo-gridfs-config-2:27019] [backup/2024-03-01T21:00:01Z] mark backup as error `check cluster for dump done: convergeCluster: get backup metadata: not found`: <nil>*
*2024-03-01T22:00:26Z E [cfg/mongo-gridfs-config-2:27019] [backup/2024-03-01T21:00:01Z] backup: check cluster for dump done: convergeCluster: get backup metadata: not found*
*2024-03-02T09:33:59Z I [rs0/mongo-gridfs-3:27017] [backup/2024-03-01T21:00:01Z] mongodump finished, waiting for the oplog*

I fixed pbm status output after all pbm-agents restart, but I still cun’t backup data. Before upgrade I didn’t have these problems.