Backup issues with operator 1.15

MongoDB Percona Operator for MongoDB
1

Hi,

I was asked to create a second MongoDB instance on a production cluster which had a working backup config, installed using v1.13 of the Percona operator. After successful testing on a small 4 node cluster, using v1.15 for both copies of the database, a second production instance, in a standard 3/3/3 config, was created, using SSD storage, into a separate namespace, using the updated operator.

Steps to Reproduce:

After adding the correct secret file to the new namespace, an existing backup config was modified slightly, the main change being the addition of a value for prefix, to allow greater control of the backup location within S3

Version:

Operator version 1.15

Logs:

root@kube-1:~# cat s3-ms-mdb-backup-prod.yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
finalizers:

  • delete-backup
    name: mongodb-clu1
    namespace: mongodb
    storages:
    s3-eu-west:
    s3:
    bucket: oabackups.viewranger.com
    credentialsSecret: outdooractive-backup-s3-base64
    insecureSkipTLSVerify: false
    maxUploadParts: 10000
    prefix: “second-mongo”
    region: eu-west-2
    storageClass: STANDARD
    uploadPartSize: 10485760
    type: s3
    tasks:
    • compressionType: gzip
      enabled: true
      keep: 30
      name: daily-s3-eu-west
      schedule: 30 2 * * *
      storageName: s3-eu-west

The only change was to add the prefix section, and to change the name. Removing the prefix section didn’t help.

When run in this form, the following error occurs.

root@kube-1:~# kubectl -n mongodb apply -f s3-ms-mdb-backup-prod.yaml
Error from server (BadRequest): error when creating “s3-ms-mdb-backup-prod.yaml”: PerconaServerMongoDB in version “v1” cannot be handled as a PerconaServerMongoDB: strict decoding error: unknown field “metadata.storages”

I believe I’ve seen some reference to an update to the apiVersion in the release notes, so have tried altering that to v1beta1 - error message below

root@kube-1:~# kubectl -n mongodb apply -f s3-ms-mdb-backup-prod.yaml
error: resource mapping not found for name: “mongodb-clu1” namespace: “mongodb” from “s3-ms-mdb-backup-prod.yaml”: no matches for kind “PerconaServerMongoDB” in version “psmdb.percona.com/v1beta1”
ensure CRDs are installed first

I’m sure I’ve missed something simple, would be most grateful if someone could point me in the right direction.

Also, is it possible/correct practise to run different versions of an operator in separate namespaces ?

Many thanks,

Mike

2

Hi

I have to admit to being fairly cross by now. The answer to this issue should be simple, and if any action had been taken on previous suggestions (a decent helm based complete worked example) then I’m fairly sure I could have fixed this myself and never had to bother anyone.

As it is, I appear to have gone backwards. To simplify, existing test env DB was wiped and reinstalled using v1.13 of the operator to match the working conditions of production. As stated, the only real change was to add a value for prefix in the bucket section, where it was previously empty.

Try as I might, I can’t get any variation of the previous config to install, using the form “kubectl apply -f file.yaml”

I’ll skip the fun of being on a memory limit boundary and having to double the memory assigned after the latest Debian 12 patch, and ask what I really need to know.

What is the correct skeleton outline for a basic backup ?

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
finalizers:

  • delete-backup
    name: backupms-test-1
    storages:
    s3-eu-west:

tasks:

  • name: daily-s3-eu-west

If I try using this format, with or without a backup section, or using PerconaServerMongoDB in the kind section - I will encounter the following error

Error from server (BadRequest): error when creating “ms-s3-feb29-1.yaml”: PerconaServerMongoDBBackup in version “v1” cannot be handled as a PerconaServerMongoDBBackup: strict decoding error: unknown field “metadata.storages”, unknown field “metadata.tasks”
root@kube-1:~# nano ms-s3-feb29-1.yaml
root@kube-1:~# kubectl -n mongodb apply -f ms-s3-feb29-1.yaml
Error from server (BadRequest): error when creating “ms-s3-feb29-1.yaml”: PerconaServerMongoDB in version “v1” cannot be handled as a PerconaServerMongoDB: strict decoding error: unknown field “metadata.storages”, unknown field “metadata.tasks”

I just need a clue as what needs adjusting - many thanks.

Mike

3

Hi @Michael_Shield !
I think your yaml for the backup is wrong.
Some basic template for backup doesn’t include “storages” key, but “storageName” instead.
Here’s an example:

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  finalizers:
  - delete-backup
  name: backup1
spec:
  clusterName: my-cluster-name
  storageName: s3-us-west
#  type: physical
#  compressionType: gzip
#  compressionLevel: 6

You can find a basic example for backup and restore in the source tree here: percona-server-mongodb-operator/deploy/backup at main · percona/percona-server-mongodb-operator · GitHub

Storages are defined in cr.yaml in backup section and there can be more than one, but it is not part of backup itself: percona-server-mongodb-operator/deploy/cr.yaml at main · percona/percona-server-mongodb-operator · GitHub

4

Hi Tomislav,

I think somewhere in the thread I gave us both a clue as to what I was doing wrong, but it took me a while to spot my mistake, so that you and the team can learn from that and we can move on to more complicated matters, for me at least.

The key word is “Helm”, which means, unless you know otherwise, that I’d forgotten all my previous hard work and reverted to incorrect practises. The docs still need a helm backup worked example, and helm needs a “modify” command, but once the changes were applied using “helm upgrade --reuse-values -f myfile.yaml” all was well, and the changes to version 1.15.0 are appreciated.

When I get some time I should perhaps just send in the required text, and the docs can be updated.

Many thanks, back shortly

Mike

1 Like