Unable to take logical backup on s3 using percona backup for mongodb

MongoDB Percona Backup for MongoDB
1

Hi,

I got the below error while taking the backup on s3 using “percona backup for mongodb”.

Error:

2020/09/03 08:40:08 [INFO] backup/2020-09-03T08:19:07Z: mark backup as error `mongodump: write data: upload to S3: MultipartUpload: upload multipart failed

upload id: XXXX

caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit.`:

My total mongoDB size is 1.5TB

Please assist on the same ASAP.

Thanks,

Surendra

1 Like
2

Anyone please answer this question.

Thanks,

Surendra Babu

1 Like
4

Hi Surendra.

There is an existing feature request for this: https://jira.percona.com/browse/PBM-485

We delayed work on it for several months to focus on point-on-time recovery functionality, but it’s back in the current development queue at the moment. We’re hoping to release it in 1.3.2, the next minor version.

Akira

1 Like
5

Thank you for the reply Akira, I am hoping/waiting for the minor release 1.3.2 with the fix soon.

1 Like
6

Hi. Just to give a heads-up v1.3.2 is expected to be released some day next week.

1 Like
7

Thank you Akira

1 Like
8

Hi Akira,

Initiated the percona backup for mongodb to s3,the db size is 1.6TB. We got the below issue after completing the backup to s3 (it took 7 hrs) .

Config command: pbm config --file /etc/mongodb/pbm_s3_store.yaml --mongodb-uri mongodb://xxxx:xxxx@127.0.0.1:27017 --set pitr.enabled=true

Backup Command: pbm backup --mongodb-uri mongodb://xxxx:xxxx@127.0.0.1:27017

List: pbm list --mongodb-uri mongodb://xxxx:xxxx@127.0.0.1:27017

Backup snapshots:

2020-10-16T08:33:33Z Failed with “oplog: read data: oplog has insufficient range, some records since the last saved ts {1602837232 4340} are missing. Run pbm backup to create a valid starting point for the PITR.”

Error: check PITR state for shard ‘zion’: PITR backup failed

PITR :

!Failed to run PITR backup. Agent logs:

zion: 2020-10-16T15:26:14.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found, a new backup is required to start PITR

Please assist on the same.

Thanks,

Surendra Babu

1 Like
9

Hi Surendra.

The first message is the key one; the second one about [failing to] defining a starting point for PITR is a secondary symptom.

“oplog: read data: oplog has insufficient range, some records since the last saved ts {1602837232 4340} are missing.”

The timestamp 1602837232 is presumably from 7hrs earlier when the backup started. It appears that the oplog collection is not large enough to store all the writes that happened in those 7hrs.

A backup snapshot has a first phase where all the collections are copied, with documents read by _id order in each collection. The versions of the documents will be whatever they are at the time the cursor passes by. In your case that means they will be inconsistent by up to 7hrs of clock time. To make all the collection documents as they were at one precise time (if/when they are restored) the oplog covering this period is also copied.

As your oplog wasn’t big enough (the oldest docs were more recent that the start time of the backup) it knew it couldn’t make an accurate backup this and showed the error message you see.

2 Likes
10

Thank you Akira,It worked after increasing the oplog size.

Thanks,

Surendra Babu

1 Like
11

Cool, good to hear.

1 Like
12

Hi Akira,

I am trying to restore(standalone node with replica set) the back up data but got the below error.

error:

2020-10-22T11:41:15.110+0000 demux finishing when there are still outs (14)

2020-10-22T11:41:15.000+0000 [ERROR] pitr/2020-10-20T13:32:59Z: restore: restore mongo dump (successes: 2091088089 / fails: 0): <db.collectionname>: error restoring from archive on stdin: reading bson input: error demultiplexing archive; archive io error

Out of 1.6TB data, one of the collection size is 900GB. My node RAM size is 120GB.

Is it possible to restore the same with pbm?

Thanks,

Surendra Babu

1 Like
13

Hi Surendra.

When the topic changes please make a new forum discussion. But in this case I’d say please look at this ticket instead: https://jira.percona.com/browse/PBM-555.

1 Like
15

I am having the same issue, but increasing oplog size did not solve it.
I’m using mongodb 5.0 on S3 compatible storage (backblaze b2), my database size is around 500G and takes 10 hours for snapshot backup, failing on oplog insufficient range:
“ERROR: oplog: read data: oplog has insufficient range, some records since the last saved ts {1635524899 25} are missing. Run pbm backup to create a valid starting point for the PITR.]”

I had already increased oplog size to 8G, 16G and it is currently with 25G. all giving the same message.

Any assistance is appreciated.

1 Like
16

please, can anyone help me? I believe that is the same first question Surendra did.

1 Like