Problem with PITR backups on non-Amazon S3 storage with new PBM version

MongoDB Percona Backup for MongoDB
1

Hello everyone! :slight_smile:

We are using PBM for a while and it works quite fine (version 2.1.0). Now on a new setup I’m using the newest version (2.4.1) and I have problems with the PITR backups:

Cluster:
========
pm0:
  - pm0/mongodb-1:27017 [P]: pbm-agent v2.4.1 OK
  - pm0/mongodb-2:27017 [S]: pbm-agent v2.4.1 OK
  - pm0/mongodb-3:27017 [S]: pbm-agent v2.4.1 OK


PITR incremental backup:
========================
Status [ON]
! ERROR while running PITR backup: 2024-03-28T12:19:30.000+0100 E [pm0/mongodb-2:27017] [pitr] copy oplog from "2024-03-28T11:17:10Z" backup:
storage copy: NotImplemented:
        status code: 501, request id: tx00000c3b32cefc1cce4f4-0066055242-e7c27ee-ce21, host id:

Currently running:
==================
(none)

Backups:
========
S3 nl-ams1 XXXX (I removed the URL here)
  Snapshots:
    2024-03-28T11:17:10Z 509.26MB <logical> [restore_to_time: 2024-03-28T11:19:21Z]
    2024-03-27T05:05:02Z 508.85MB <logical> [restore_to_time: 2024-03-27T05:07:09Z]
    2024-03-26T23:05:02Z 508.91MB <logical> [restore_to_time: 2024-03-26T23:07:37Z]
    2024-03-26T17:05:02Z 178.49KB <logical> [restore_to_time: 2024-03-26T17:05:08Z]
    2024-03-26T11:05:02Z 173.61KB <logical> [restore_to_time: 2024-03-26T11:05:08Z]
    2024-03-26T05:05:02Z 31.19KB <logical> [restore_to_time: 2024-03-26T05:05:09Z]
    2024-03-25T23:58:48Z 26.23KB <logical> [restore_to_time: 2024-03-25T23:58:54Z]
  PITR chunks [648.43MB]:
    2024-03-25T23:58:55Z - 2024-03-28T11:17:11Z

Can it be, that the S3 implementation (it’s not AWS) does not provide all necessary commands/is not following the standard, or is it a bug within PBM?

Thanks in advance,

Andreas

2

PS: I downgraded to version 2.2.1 and it works with the same. :man_shrugging:

3

Hi @Andreas_Filler

Thank you for the update, but that’s interesting.
PBM should work with other S3 compatible storages, but was only tested with the following ones:

Are you using anything different than that? Could you please provide more details on your storage along with the pbm config --list output?

Those would help us narrow down the issue here a bit more.

4

Hi @jenunes

Our S3 storage is not on the list. We are using the S3 provided by Gridscale Object Storage: Low-Cost, S3-Compatible & Fail-Safe Storage , which is a hoster in Germany, who is currently also expanding to other countries since the are not only a hoster, but also a provider of the complete software stack to provide professional hosting.

My configuration (some names slightly adjusted) for their storage in the Netherlands is:

pitr:
  enabled: true
  oplogSpanMin: 10
  compression: s2
storage:
  type: s3
  s3:
    provider: aws     (irrelevant for gridscale, just a placeholder)
    region: nl-ams1
    endpointUrl: https://ce21.gos3.io
    forcePathStyle: true
    bucket: xyz   (changed)
    prefix: percona
    credentials:
      access-key-id: '***'
      secret-access-key: '***'
    serverSideEncryption:
      sseAlgorithm: ""
      kmsKeyID: ""
      sseCustomerAlgorithm: AES256
      sseCustomerKey: '***'
    maxUploadParts: 10000
    storageClass: STANDARD
    insecureSkipTLSVerify: false
backup:
  compression: gzip

Thanks in advance,

Andreas