Incremental backup failed

test123124424214 · July 4, 2023, 1:33am

Hello everyone.

Try to do incremental backup with percona tools:

 percona/percona-backup-mongodb:2.1.0
  percona/percona-server-mongodb:6.0.6-5

running cluster into docker,
docker-compose up -d pdmdb_master pdmdb_slave pdmdb-init-replica

services:
  pdmdb_master:
    container_name: pdmdb_master
    image: percona/percona-server-mongodb:6.0.6-5
    ports:
      - 27022:27017
    restart: unless-stopped
    networks:
      - pdmdb-network
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "dbrs", "--oplogSize=2148" ]

  pdmdb_slave:
    container_name: pdmdb_slave
    image: percona/percona-server-mongodb:6.0.6-5
    ports:
      - 27023:27017
    restart: unless-stopped
    networks:
      - pdmdb-network
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "dbrs", "--oplogSize=2148" ]

  pdmdb-init-replica:
    container_name: pdmdb_replica
    image: percona/percona-server-mongodb:6.0.6-5
    volumes:
      - ./rs-init.sh:/scripts/rs-init.sh
    restart: "no"
    networks:
      - pdmdb-network
    depends_on:
      - pdmdb_master
      - pdmdb_slave
    command: >
      /scripts/rs-init.sh

  pdmdb-backup-master:
    container_name: pdmdb_backup_master
    image: percona/percona-backup-mongodb:2.1.0
    volumes:
      - ./mongodb_backup:/mongodb_backup
      - ./pbm_config.yaml:/etc/pbm_config.yaml
    networks:
      - pdmdb-network
    environment:
      PBM_MONGODB_URI: "mongodb://pdmdb_master:27017/?replicaSet=dbrs"
    depends_on:
      - pdmdb_master
      - pdmdb_slave
    user: root
    command: >
            bash -c "pbm config --file /etc/pbm_config.yaml &&
                     pbm-agent"

  pdmdb-backup-slave:
    container_name: pdmdb_backup_slave
    image: percona/percona-backup-mongodb:2.1.0
    volumes:
      - ./mongodb_backup:/mongodb_backup
      - ./pbm_config.yaml:/etc/pbm_config.yaml
    networks:
      - pdmdb-network
    environment:
      PBM_MONGODB_URI: "mongodb://pdmdb_slave:27017/?replicaSet=dbrs"
    depends_on:
      - pdmdb_master
      - pdmdb_slave
    user: root
    command: >
            bash -c "pbm config --file /etc/pbm_config.yaml &&
                     pbm-agent"

networks:
  pdmdb-network:
    driver: bridge

try to do incremental backup:
docker-compose run pdmdb-backup
pbm config --file /etc/pbm_config.yaml
pbm-agent 2> /pbm_log.txt &

Cluster:
========
dbrs:
  - dbrs/pdmdb_master:27017 [P]: pbm-agent v2.1.0 OK
  - dbrs/pdmdb_slave:27017 [S]: pbm-agent v2.1.0 OK


PITR incremental backup:
========================
Status [OFF]

Currently running:
==================
(none)

Backups:
========
FS  /mongodb_backup
  Snapshots:
    2023-07-04T02:19:57Z 0.00B <incremental, base> [ERROR: upload file `/data/db/journal/WiredTigerLog.0000000001`: get file stat: stat /data/db/journal/WiredTigerLog.0000000001: no such file or directory] [2023-07-04T02:20:03Z]

pbm backup --type incremental --base

my log is:

2023-07-04T01:19:36.000+0000 I starting PITR routine
2023-07-04T01:19:36.000+0000 I node: dbrs/pdmdb_master:27017
2023-07-04T01:19:36.000+0000 I listening for the commands
2023-07-04T01:19:41.000+0000 W [agentCheckup] get current storage status: query mongo: mongo: no documents in result
2023-07-04T01:19:43.000+0000 I got command backup [name: 2023-07-04T01:19:43Z, compression: s2 (level: default)] <ts: 1688433583>
2023-07-04T01:19:43.000+0000 I got epoch {1688433554 10}
2023-07-04T01:19:43.000+0000 D [backup/2023-07-04T01:19:43Z] init backup meta
2023-07-04T01:19:43.000+0000 D [backup/2023-07-04T01:19:43Z] nomination list for dbrs: [[pdmdb_master:27017]]
2023-07-04T01:19:43.000+0000 D [backup/2023-07-04T01:19:43Z] nomination dbrs, set candidates [pdmdb_master:27017]
2023-07-04T01:19:44.000+0000 I [backup/2023-07-04T01:19:43Z] backup started
2023-07-04T01:19:45.000+0000 D [backup/2023-07-04T01:19:43Z] flush incremental backup history
2023-07-04T01:19:45.000+0000 D [backup/2023-07-04T01:19:43Z] backup cursor id: b086c445-0e36-464e-96c2-ebdaf3519fe9
2023-07-04T01:19:48.000+0000 D [backup/2023-07-04T01:19:43Z] set journal up to {1688433585 2}
2023-07-04T01:19:48.000+0000 I [backup/2023-07-04T01:19:43Z] uploading data
2023-07-04T01:19:48.000+0000 D [backup/2023-07-04T01:19:43Z] stop cursor polling: <nil>, cursor err: <nil>
2023-07-04T01:19:48.000+0000 I [backup/2023-07-04T01:19:43Z] mark RS as error `upload file `/data/db/index-44-9164515937719912368.wt`: get file stat: stat /data/db/index-44-9164515937719912368.wt: no such file or directory`: <nil>
2023-07-04T01:19:48.000+0000 I [backup/2023-07-04T01:19:43Z] mark backup as error `upload file `/data/db/index-44-9164515937719912368.wt`: get file stat: stat /data/db/index-44-9164515937719912368.wt: no such file or directory`: <nil>
2023-07-04T01:19:48.000+0000 E [backup/2023-07-04T01:19:43Z] backup: upload file `/data/db/index-44-9164515937719912368.wt`: get file stat: stat /data/db/index-44-9164515937719912368.wt: no such file or directory
2023-07-04T01:19:48.000+0000 D [backup/2023-07-04T01:19:43Z] releasing lock

or

         pbm-agent'                                                                                                                                                                                                                                                                                                                                
pitr:                                                                                                                                                                                                                                                                                                                                              
  enabled: false                                                                                                                                                                                                                                                                                                                                   
  oplogSpanMin: 0                                                                                                                                                                                                                                                                                                                                  
  compression: s2                                                                                                                                                                                                                                                                                                                                  
storage:                                                                                                                                                                                                                                                                                                                                           
  type: filesystem                                                                                                                                                                                                                                                                                                                                 
  filesystem:                                                                                                                                                                                                                                                                                                                                      
    path: /mongodb_backup                                                                                                                                                                                                                                                                                                                          
backup:                                                                                                                                                                                                                                                                                                                                            
  compression: s2                                                                                                                                                                                                                                                                                                                                  
2023-07-04T02:18:04.000+0000 I pbm-agent:                                                                                                                                                                                                                                                                                                          
Version:   2.1.0                                                                                                                                                                                                                                                                                                                                   
Platform:  linux/amd64                                                                                                                                                                                                                                                                                                                             
GitCommit: 342b5b9c996e8bd17a678ea0cecbf2eaaf9704ab                                                                                                                                                                                                                                                                                                
GitBranch: release-2.1.0                                                                                                                                                                                                                                                                                                                           
BuildTime: 2023-04-14_12:23_UTC                                                                                                                                                                                                                                                                                                                    
GoVersion: go1.19
2023-07-04T02:18:04.000+0000 I node: dbrs/pdmdb_slave:27017
2023-07-04T02:18:04.000+0000 I starting PITR routine
2023-07-04T02:18:04.000+0000 I listening for the commands
2023-07-04T02:18:04.000+0000 I got command resync <ts: 1688437084>
2023-07-04T02:18:04.000+0000 I got epoch {1688437084 3}
2023-07-04T02:18:04.000+0000 D [resync] lock not acquired
2023-07-04T02:18:09.000+0000 W [agentCheckup] get current storage status: query mongo: mongo: no documents in result
2023-07-04T02:19:58.000+0000 I got command backup [name: 2023-07-04T02:19:57Z, compression: s2 (level: default)] <ts: 1688437197>
2023-07-04T02:19:58.000+0000 I got epoch {1688437084 24}
2023-07-04T02:19:59.000+0000 I [backup/2023-07-04T02:19:57Z] backup started
2023-07-04T02:20:00.000+0000 D [backup/2023-07-04T02:19:57Z] flush incremental backup history
2023-07-04T02:20:00.000+0000 D [backup/2023-07-04T02:19:57Z] backup cursor id: dcaae42e-2b68-4f45-a68d-c2628c8ea487
2023-07-04T02:20:03.000+0000 D [backup/2023-07-04T02:19:57Z] set journal up to {1688437200 2}
2023-07-04T02:20:03.000+0000 I [backup/2023-07-04T02:19:57Z] uploading data
2023-07-04T02:20:03.000+0000 D [backup/2023-07-04T02:19:57Z] stop cursor polling: <nil>, cursor err: <nil>
2023-07-04T02:20:03.000+0000 I [backup/2023-07-04T02:19:57Z] mark RS as error `upload file `/data/db/journal/WiredTigerLog.0000000001`: get file stat: stat /data/db/journal/WiredTigerLog.0000000001: no such file or directory`: <nil>
2023-07-04T02:20:03.000+0000 I [backup/2023-07-04T02:19:57Z] mark backup as error `upload file `/data/db/journal/WiredTigerLog.0000000001`: get file stat: stat /data/db/journal/WiredTigerLog.0000000001: no such file or directory`: <nil>
2023-07-04T02:20:03.000+0000 E [backup/2023-07-04T02:19:57Z] backup: upload file `/data/db/journal/WiredTigerLog.0000000001`: get file stat: stat /data/db/journal/WiredTigerLog.0000000001: no such file or directory
2023-07-04T02:20:03.000+0000 D [backup/2023-07-04T02:19:57Z] releasing lock

what is wrong i did?

test123124424214 · July 5, 2023, 10:11pm

========
FS  /data/mongodb_backup
  Snapshots:
    2023-07-05T22:08:38Z 0.00B <incremental, base> [ERROR: upload file `/datadb/index-45-236754668729706061.wt`: get file stat: stat /datadb/index-45-236754668729706061.wt: no such file or directory] [2023-07-05T22:08:43Z]
    2023-07-05T22:08:16Z 0.00B <incremental, base> [ERROR: upload file `/datadb/index-45-236754668729706061.wt`: get file stat: stat /datadb/index-45-236754668729706061.wt: no such file or directory] [2023-07-05T22:08:21Z]
    2023-07-05T22:07:34Z 0.00B <incremental, base> [ERROR: upload file `/datadb/index-45-236754668729706061.wt`: get file stat: stat /datadb/index-45-236754668729706061.wt: no such file or directory] [2023-07-05T22:07:39Z]
    2023-07-05T22:06:02Z 22.78KB <logical> [restore_to_time: 2023-07-05T22:06:07Z]

mongodb master data is:

collection-0--6847434795319775107.wt
collection-0--8347697731587832119.wt
collection-10--6847434795319775107.wt
collection-10--8347697731587832119.wt
collection-11--8347697731587832119.wt
collection-13--8347697731587832119.wt
collection-15--6847434795319775107.wt
collection-15--8347697731587832119.wt
collection-17--8347697731587832119.wt
collection-19--8347697731587832119.wt
collection-20--6847434795319775107.wt
collection-25--6847434795319775107.wt
collection-2--8347697731587832119.wt
collection-32--6847434795319775107.wt
collection-3--6847434795319775107.wt
collection-39--6847434795319775107.wt
collection-41--6847434795319775107.wt
collection-47--6847434795319775107.wt
collection-4--8347697731587832119.wt
collection-53--6847434795319775107.wt
collection-55--6847434795319775107.wt
collection-57--6847434795319775107.wt
collection-6--6847434795319775107.wt
collection-6--8347697731587832119.wt
collection-8--6847434795319775107.wt
collection-8--8347697731587832119.wt
diagnostic.data
index-11--6847434795319775107.wt
index-12--6847434795319775107.wt
index-12--8347697731587832119.wt
index-14--8347697731587832119.wt
index-16--6847434795319775107.wt
index-16--8347697731587832119.wt
index-1--6847434795319775107.wt
index-17--6847434795319775107.wt
index-1--8347697731587832119.wt
index-18--8347697731587832119.wt
index-20--8347697731587832119.wt
index-21--6847434795319775107.wt
index-22--6847434795319775107.wt
index-26--6847434795319775107.wt
index-2--6847434795319775107.wt
index-27--6847434795319775107.wt
index-30--6847434795319775107.wt
index-33--6847434795319775107.wt
index-34--6847434795319775107.wt
index-37--6847434795319775107.wt
index-3--8347697731587832119.wt
index-40--6847434795319775107.wt
index-42--6847434795319775107.wt
index-4--6847434795319775107.wt
index-48--6847434795319775107.wt
index-54--6847434795319775107.wt
index-56--6847434795319775107.wt
index-5--6847434795319775107.wt
index-5--8347697731587832119.wt
index-58--6847434795319775107.wt
index-59--6847434795319775107.wt
index-7--6847434795319775107.wt
index-7--8347697731587832119.wt
index-9--6847434795319775107.wt
index-9--8347697731587832119.wt
journal
_mdb_catalog.wt
mongod.lock
sizeStorer.wt
storage.bson
WiredTiger
WiredTigerLAS.wt
WiredTiger.lock
WiredTiger.turtle
WiredTiger.wt

mongodb slave data is:

collection-0-236754668729706061.wt
collection-10-236754668729706061.wt
collection-16-236754668729706061.wt
collection-17-236754668729706061.wt
collection-19-236754668729706061.wt
collection-21-236754668729706061.wt
collection-2-236754668729706061.wt
collection-23-236754668729706061.wt
collection-25-236754668729706061.wt
collection-30-236754668729706061.wt
collection-35-236754668729706061.wt
collection-37-236754668729706061.wt
collection-39-236754668729706061.wt
collection-4-236754668729706061.wt
collection-44-236754668729706061.wt
collection-49-236754668729706061.wt
collection-54-236754668729706061.wt
collection-61-236754668729706061.wt
collection-6-236754668729706061.wt
collection-68-236754668729706061.wt
collection-70-236754668729706061.wt
collection-76-236754668729706061.wt
collection-82-236754668729706061.wt
collection-8-236754668729706061.wt
collection-84-236754668729706061.wt
collection-86-236754668729706061.wt
diagnostic.data
index-11-236754668729706061.wt
index-1-236754668729706061.wt
index-18-236754668729706061.wt
index-20-236754668729706061.wt
index-22-236754668729706061.wt
index-24-236754668729706061.wt
index-26-236754668729706061.wt
index-27-236754668729706061.wt
index-31-236754668729706061.wt
index-32-236754668729706061.wt
index-3-236754668729706061.wt
index-36-236754668729706061.wt
index-38-236754668729706061.wt
index-40-236754668729706061.wt
index-41-236754668729706061.wt
index-45-236754668729706061.wt
index-46-236754668729706061.wt
index-50-236754668729706061.wt
index-51-236754668729706061.wt
index-5-236754668729706061.wt
index-55-236754668729706061.wt
index-56-236754668729706061.wt
index-59-236754668729706061.wt
index-62-236754668729706061.wt
index-63-236754668729706061.wt
index-66-236754668729706061.wt
index-69-236754668729706061.wt
index-71-236754668729706061.wt
index-7-236754668729706061.wt
index-77-236754668729706061.wt
index-83-236754668729706061.wt
index-85-236754668729706061.wt
index-87-236754668729706061.wt
index-88-236754668729706061.wt
index-9-236754668729706061.wt
journal
_mdb_catalog.wt
mongod.lock
sizeStorer.wt
storage.bson
WiredTiger
WiredTigerLAS.wt
WiredTiger.lock
WiredTiger.turtle
WiredTiger.wt

oleksandr.havryliak · July 7, 2023, 6:06am

Hi @test123124424214 !
For physical/incremental backups PBM must have RW access to mongodb datadir. Try to share datadir volume with pbm container and make sure that it has full permissions (define user option in compose-file
to be the same for all containers)

Topic		Replies	Views
Trouble setting up incremental backup in MongoDB Cluster - Docker Percona Backup for MongoDB	2	244	May 17, 2024
Cant execute incremental backup Percona Backup for MongoDB	11	1053	October 17, 2023
Cannot restore incremetal backup Percona Backup for MongoDB closed-no-reply	0	599	September 11, 2023
No support for Incremental Backup in 1.15.0 version? Percona Operator for MongoDB mongodb	11	775	October 26, 2023
PSMDB and PBM in separated containers Percona Backup for MongoDB percona , pbm	2	491	December 13, 2023

Incremental backup failed

Related topics