MongoDB backup failed "upload file: write data: upload to S3: SerializationError"

MongoDB Percona Backup for MongoDB
,
1

Greetings,

I can’t get MongoDB backup to work with Percona Operator on Kubernetes.
I’d be grateful for any help.

I’m getting these logs after applying on-demand backup:

>>> kubectl logs -n mongodb-percona pod/my-cluster-name-rs0-0 -c backup-agent
<Snip>
mark RS as error upload data files: upload file /data/db/key.db/WiredTigerLog.0000000010: upload file: write data: upload to S3: SerializationError: failed to decode REST XML response status code: 200, request id: caused by: XML syntax error on line 1: attribute name without = in element.:
<Snip>

full log.txt (3.5 KB)

Kubernetes is hosted on DigitalOcean and S3 bucket is also hosted there.

Steps to Reproduce:

I have this cr.yaml file:
cr.txt (4.1 KB)

And this is the backup I’m trying to do:
on-demand-backup.txt (212 Bytes)
The automated backup defined in cr.yaml also doesn’t work.

Additional info

image
image1326×61 73.6 KB

image
image850×142 53.5 KB

image
image913×67 30 KB

2

Hi @pepa!
Try to reconfigure the backup section with:

  1. endpointUrl: https://fra1.digitaloceanspaces.com
  2. region: us-east-1
3

Thanks for your reply.
I tried your solution on Civo cloud, it worked there.

But when I use it on DigitalOcean, it gives me 400 status code:

check storage connection: storage check failed with: get S3 object header: BadRequest: Bad Request
        status code: 400, request id: tx0000092fdbe2c55ac20ae-006665a7eb-fa4b23ec-fra1b, host id:

Btw DigitalOcean’s URL for bucket is: https://bucket-name.fra1.digitaloceanspaces.com

I could eventually go to Civo, but I would like to stay at DigitalOcean if possible.
Is there anything else I could try?

4

Hi @pepa
I would suggest trying to set forcepathstyle option to false if digitalocean requires bucket name in endpointUrl

5

Hi,
I have set it up like this and I’m no longer getting 400 error:

    storages:
      s3-appname:
        type: s3
        s3:
          endpointUrl: https://digitaloceanspaces.com
          bucket: "appname-mongodb-backups"
          credentialsSecret: my-cluster-name-backup-s3
          region: fra1
          forcePathStyle: false

I didn’t know about forcePathStyle, because it’s not documented in Custom Resource options - Percona Operator for MongoDB.

But the original problem remain - I’m still facing the serialization error. So I’ll be storing backups on Civo S3 (and maybe moving Kubernetes there as well) if this error can’t be resolved as it seems to only happen on DigitalOcean.

6

Hi @pepa!
I meant your settings should be like:

        s3:
          endpointUrl: https://fra1.digitaloceanspaces.com
          bucket: appname-mongodb-backups
          region: us-east-1
          forcePathStyle: true

or

        s3:
          endpointUrl: https://appname-mongodb-backups.fra1.digitaloceanspaces.com
          region: us-east-1
          forcePathStyle: false
7

Hi,
When I use the first settings I get 400 error.

In the second settings the bucket is required. After applying cr.yml I get:
The PerconaServerMongoDB "my-cluster-name" is invalid: spec.backup.storages.s3-appname.s3.bucket: Required value