Can't get backup working (operator crashes)

MongoDB Percona Operator for MongoDB
1

I’m trying to setup backup for my mongo replicaset. Even in the simplest config possible, I can’t get it working.

This is what I did:

  1. Install and configure minio
  2. Create an S3 bucket in minio with preset accessKey and secretKey
  3. Install the operator in the db namespace
  4. Put the minio-backup-key.yaml in the db namespace
  5. Create the replicaset with following cr.yaml
apiVersion: psmdb.percona.com/v1-9-0
kind: PerconaServerMongoDB
metadata:
  name: mongodb
spec:
  crVersion: 1.9.0
  image: percona/percona-server-mongodb:4.4.6-8
  imagePullPolicy: Always
  secrets:
    users: mongodb-secrets
  replsets:
  - name: rs0
    size: 3
    podDisruptionBudget:
      maxUnavailable: 1
    expose:
      enabled: false
    arbiter:
      enabled: false
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 10Gi

  sharding:
    enabled: false

  mongod:
    operationProfiling:
      mode: slowOp
      slowOpThresholdMs: 100
      rateLimit: 100
    systemLog:
      verbosity: 0
    security:
      redactClientLogData: false
      enableEncryption: true
      encryptionKeySecret: mongodb-encryption-key
      encryptionCipherMode: AES256-CBC

  backup:
    enabled: true
    restartOnFailure: true
    image: percona/percona-server-mongodb-operator:1.9.0-backup
    serviceAccountName: percona-server-mongodb-operator
    storages:
      s3-minio:
        type: s3
        s3:
          bucket: operator-testing
          credentialsSecret: backup-minio-key
          region: eu-central-1
          endpointUrl: http://minio.minio.svc.cluster.local:9000

The mongo replicaset is build and seems to work ok.
When inspecting the backup-agent logs I see a lot of messages:

E [agentCheckup] check storage connecion: unable to get storage: get config: get: mongo: no documents in result

Seems not OK. Also tried with AWS S3 config, but same issue.

To check if the backup is working anyway I applied an on demand backup.yaml
The moment the backup starts, the operator dies.

{"level":"info","ts":1630513398.6012661,"logger":"cmd","msg":"Git commit: 327bb6fd6af19017f05ee88a98a267114aa9abce Git branch: release-1-9-0"}
{"level":"info","ts":1630513398.6013076,"logger":"cmd","msg":"Go Version: go1.16.5"}
{"level":"info","ts":1630513398.6013186,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1630513398.6013303,"logger":"cmd","msg":"operator-sdk Version: v0.17.2"}
{"level":"info","ts":1630513398.6016169,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1630513400.4733415,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."}
{"level":"info","ts":1630513400.4734046,"logger":"leader","msg":"Continuing as the leader."}
{"level":"info","ts":1630513402.3294938,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1630513402.3299258,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1630513402.3383148,"logger":"controller_psmdb","msg":"server version","platform":"kubernetes","version":"v1.20.8-gke.2100"}
{"level":"info","ts":1630513402.338853,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1630513402.3396916,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbbackup-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1630513402.3407724,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbrestore-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1630513402.3445296,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"psmdb-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1630513402.3446467,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1630513402.4415796,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbbackup-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1630513402.4458094,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbrestore-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1630513402.451896,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"psmdb-controller"}
{"level":"info","ts":1630513402.5428116,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"perconaservermongodbbackup-controller"}
{"level":"info","ts":1630513402.5433543,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"perconaservermongodbbackup-controller","worker count":1}
{"level":"info","ts":1630513402.5462518,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"perconaservermongodbrestore-controller"}
{"level":"info","ts":1630513402.5528839,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"psmdb-controller","worker count":1}
E0901 16:23:22.644651       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 534 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x19afa60, 0x2a5a9d0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
panic(0x19afa60, 0x2a5a9d0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/percona/percona-server-mongodb-operator/pkg/psmdb.GetAddr(0xc0004d3c00, 0xc00047b410, 0xd, 0xc000759874, 0x3, 0x113eaef, 0x0)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:230 +0x178
github.com/percona/percona-server-mongodb-operator/pkg/psmdb.MongoHost(0x1f42d28, 0xc000a8e690, 0xc0004d3c00, 0xc000759874, 0x3, 0x1b68c00, 0x1842d6d, 0x3, 0x1bb9a65, 0x2, ...)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:209 +0x13a
github.com/percona/percona-server-mongodb-operator/pkg/psmdb.GetReplsetAddrs(0x1f42d28, 0xc000a8e690, 0xc0004d3c00, 0xc000759874, 0x3, 0x0, 0xc000470000, 0x3, 0x3, 0x0, ...)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:193 +0x158
github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM(0x1f42d28, 0xc000a8e690, 0xc0004d3c00, 0xc000ab319a, 0x2, 0xc000ab31b8)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:70 +0x55f
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).newBackup(0xc0008979f8, 0xc000b1a8c0, 0x0, 0xc000ab319a, 0x2)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/backup.go:27 +0xfa
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile(0xc0008979f8, 0xc000ab319a, 0x2, 0xc000ab3180, 0xb, 0xeeca5600, 0x0, 0x0, 0x0)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:133 +0x35d
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0009fe240, 0x1a15fe0, 0xc000bc6280, 0xc000534600)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0009fe240, 0x0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000bfe050)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000bfe050, 0x3b9aca00, 0x0, 0x100000000000001, 0xc0009ba4e0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc000bfe050, 0x3b9aca00, 0xc0009ba4e0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x32d
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17569f8]

goroutine 534 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x109
panic(0x19afa60, 0x2a5a9d0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/percona/percona-server-mongodb-operator/pkg/psmdb.GetAddr(0xc0004d3c00, 0xc00047b410, 0xd, 0xc000759874, 0x3, 0x113eaef, 0x0)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:230 +0x178
github.com/percona/percona-server-mongodb-operator/pkg/psmdb.MongoHost(0x1f42d28, 0xc000a8e690, 0xc0004d3c00, 0xc000759874, 0x3, 0x1b68c00, 0x1842d6d, 0x3, 0x1bb9a65, 0x2, ...)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:209 +0x13a
github.com/percona/percona-server-mongodb-operator/pkg/psmdb.GetReplsetAddrs(0x1f42d28, 0xc000a8e690, 0xc0004d3c00, 0xc000759874, 0x3, 0x0, 0xc000470000, 0x3, 0x3, 0x0, ...)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:193 +0x158
github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM(0x1f42d28, 0xc000a8e690, 0xc0004d3c00, 0xc000ab319a, 0x2, 0xc000ab31b8)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:70 +0x55f
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).newBackup(0xc0008979f8, 0xc000b1a8c0, 0x0, 0xc000ab319a, 0x2)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/backup.go:27 +0xfa
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile(0xc0008979f8, 0xc000ab319a, 0x2, 0xc000ab3180, 0xb, 0xeeca5600, 0x0, 0x0, 0x0)
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:133 +0x35d
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0009fe240, 0x1a15fe0, 0xc000bc6280, 0xc000534600)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0009fe240, 0x0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000bfe050)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000bfe050, 0x3b9aca00, 0x0, 0x100000000000001, 0xc0009ba4e0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc000bfe050, 0x3b9aca00, 0xc0009ba4e0)
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x32d

What’s wrong in my configuration?
Testing on a newly build cluster on GKE (3 nodes e2-medium)

1 Like