MongoDB Backups failing intermitently - cluster with status initializing

MongoDB Percona Operator for MongoDB
1

I have a number of mongo db replicasets deployed on Openshift which are configured to backup to AWS S3 with backup config like this:

  backup:
    enabled: true
    image: >-
      registry.connect.redhat.com/percona/percona-server-mongodb-operator-containers@sha256:071ac076bdd1ddec2754f4a8e784b09dc1ab457ce1aaf29ba292df993005abff
    pitr:
      enabled: false
    restartOnFailure: true
    serviceAccountName: percona-server-mongodb-operator
    storages:
      s3-eu-west-2:
        s3:
          bucket: xxx-mongodb-backup
          credentialsSecret: openshift-mongodb-backup-credentials
          prefix: UAT
          region: eu-west-2
        type: s3
    tasks:
      - enabled: true
        keep: 3
        name: daily-s3-eu-west-2
        schedule: 0 23 * * *
        storageName: s3-eu-west-2

These mostly work fine but ocassionally they will fail with the error:

“reconcile backup: failed to run backup: allowUnsafeConfigurations must be true to run backup on cluster with status initializing”

Unfortunately when this happens they do not retry and so there are gaps in my backups that would be a problem if a failure coincided with a missed backup. Looking at the PerconaServerMongoDB resource that the backup fails to backup I can see it moving to initializing seemingly in response to the backup which was created at ‘2022-07-25T09:54:24Z’.

status:
  conditions: 
    - lastTransitionTime: '2022-07-25T09:56:55Z'
      status: 'True'
      type: initializing
    - lastTransitionTime: '2022-07-25T09:57:10Z'
      message: 'rs0: ready'
      reason: RSReady
      status: 'True'
      type: ready
    - lastTransitionTime: '2022-07-25T09:57:10Z'
      status: 'True'
      type: initializing
    - lastTransitionTime: '2022-07-25T09:57:15Z'
      status: 'True'
      type: ready

Is there a way to make it retry or is there something I can fix so it doesn’t fail? I need the backups to run reliably so welcome any advice

1 Like
2

Hi @edoatley,

I think you should focus on why cluster becomes unready for these short periods of time. It doesn’t seem normal. If you provide the operator logs we can help you more.

1 Like
3

No problem I would be happy to do that but it seems maybe because I am new to the forum I am not allowed to upload the file. I can share a chunk of it here:

{"level":"error","ts":1658742638.7572837,"logger":"controller_perconaservermongodbbackup","msg":"failed to update backup status","backup":"cron-percona-mongodb--20220723230002-nqbdv","error":"send update: Operation cannot be fulfilled on perconaservermongodbbackups.psmdb.percona.com \"cron-percona-mongodb--20220723230002-nqbdv\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/psmdb.percona.com/perconaservermongodbbackups/percona-mongodb/cron-percona-mongodb--20220723230002-nqbdv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 88ef631a-53ff-47de-b12d-4a7948299da4, UID in object meta: ","errorVerbose":"Operation cannot be fulfilled on perconaservermongodbbackups.psmdb.percona.com \"cron-percona-mongodb--20220723230002-nqbdv\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/psmdb.percona.com/perconaservermongodbbackups/percona-mongodb/cron-percona-mongodb--20220723230002-nqbdv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 88ef631a-53ff-47de-b12d-4a7948299da4, UID in object meta: \nsend update\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).updateStatus\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:245\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:121\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:164\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:123\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:164\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1658742648.8425832,"logger":"controller_perconaservermongodbbackup","msg":"failed to update backup status","backup":"cron-percona-mongodb--20220724230002-7q7k6","error":"send update: Operation cannot be fulfilled on perconaservermongodbbackups.psmdb.percona.com \"cron-percona-mongodb--20220724230002-7q7k6\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/psmdb.percona.com/perconaservermongodbbackups/percona-mongodb/cron-percona-mongodb--20220724230002-7q7k6, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: fc8e391e-b71d-4706-8ffb-c450723e1415, UID in object meta: ","errorVerbose":"Operation cannot be fulfilled on perconaservermongodbbackups.psmdb.percona.com \"cron-percona-mongodb--20220724230002-7q7k6\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/psmdb.percona.com/perconaservermongodbbackups/percona-mongodb/cron-percona-mongodb--20220724230002-7q7k6, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: fc8e391e-b71d-4706-8ffb-c450723e1415, UID in object meta: \nsend update\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).updateStatus\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:245\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:121\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:164\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:123\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:164\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1658742653.8836193,"logger":"controller_perconaservermongodbbackup","msg":"failed to update backup status","backup":"percona-mongodb-uat-202207250905","error":"send update: Operation cannot be fulfilled on perconaservermongodbbackups.psmdb.percona.com \"percona-mongodb-uat-202207250905\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/psmdb.percona.com/perconaservermongodbbackups/percona-mongodb/percona-mongodb-uat-202207250905, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 138eb8c0-af0f-44bc-8e28-4b97fc4c87eb, UID in object meta: ","errorVerbose":"Operation cannot be fulfilled on perconaservermongodbbackups.psmdb.percona.com \"percona-mongodb-uat-202207250905\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/psmdb.percona.com/perconaservermongodbbackups/percona-mongodb/percona-mongodb-uat-202207250905, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 138eb8c0-af0f-44bc-8e28-4b97fc4c87eb, UID in object meta: \nsend update\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).updateStatus\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:245\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:121\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:164\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:123\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:164\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1658743017.758037,"logger":"controller_perconaservermongodbbackup","msg":"failed to make backup","backup":"percona-mongodb-uat-202207251054","error":"failed to run backup: allowUnsafeConfigurations must be true to run backup on cluster with status initializing","errorVerbose":"allowUnsafeConfigurations must be true to run backup on cluster with status initializing\ngithub.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1.(*PerconaServerMongoDB).CanBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1/psmdb_types.go:709\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:175\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:159\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nfailed to run backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:176\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:159\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:117\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:161\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1658743017.7635167,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"perconaservermongodbbackup-controller","request":"percona-mongodb/percona-mongodb-uat-202207251054","error":"reconcile backup: failed to run backup: allowUnsafeConfigurations must be true to run backup on cluster with status initializing","errorVerbose":"allowUnsafeConfigurations must be true to run backup on cluster with status initializing\ngithub.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1.(*PerconaServerMongoDB).CanBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1/psmdb_types.go:709\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:175\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:159\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nfailed to run backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:176\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:159\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nreconcile backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:161\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

This was as much as I can share in one post

1 Like
4

Could you paste the full logs to a Github gist or something similar? This logs doesn’t give any clue about why your cluster becomes unready.

1 Like
5

Good idea @Ege_Gunes - hopefully you can see this percona-mongo-operator.log · GitHub

1 Like
6

Since raising this post all of the subsequent backups have worked with no issue - it seems likely that there is some sort of network issue causing this so will look into that

1 Like
7

I’m facing the same issue. My cluster is on initializing status for several days but the replicaset is working correctly.
the last transition

Name:         mongodb-rs
Namespace:    default
Labels:       app.kubernetes.io/instance=experimentation
Annotations:  argocd.argoproj.io/sync-wave: 1
API Version:  psmdb.percona.com/v1
Kind:         PerconaServerMongoDB
Metadata:
  Creation Timestamp:  2022-07-25T19:13:37Z
  Finalizers:
    delete-psmdb-pods-in-order
  Generation:  4
  Managed Fields:
    API Version:  psmdb.percona.com/v1-12-0
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:argocd.argoproj.io/sync-wave:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:finalizers:
          .:
          v:"delete-psmdb-pods-in-order":
        f:labels:
          .:
          f:app.kubernetes.io/instance:
      f:spec:
        .:
        f:allowUnsafeConfigurations:
        f:backup:
          .:
          f:enabled:
          f:image:
          f:pitr:
            .:
            f:compressionLevel:
            f:compressionType:
            f:enabled:
          f:serviceAccountName:
          f:storages:
            .:
            f:ovh:
              .:
              f:s3:
                .:
                f:bucket:
                f:credentialsSecret:
                f:endpointUrl:
                f:insecureSkipTLSVerify:
                f:prefix:
                f:region:
              f:type:
          f:tasks:
        f:crVersion:
        f:image:
        f:imagePullPolicy:
        f:pmm:
          .:
          f:enabled:
          f:image:
          f:serverHost:
        f:replsets:
        f:secrets:
          .:
          f:encryptionKey:
          f:users:
        f:sharding:
          .:
          f:configsvrReplSet:
            .:
            f:affinity:
              .:
              f:antiAffinityTopologyKey:
            f:expose:
              .:
              f:enabled:
              f:exposeType:
            f:podDisruptionBudget:
              .:
              f:maxUnavailable:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:size:
            f:volumeSpec:
              .:
              f:persistentVolumeClaim:
                .:
                f:resources:
                  .:
                  f:requests:
                    .:
                    f:storage:
          f:enabled:
          f:mongos:
            .:
            f:affinity:
              .:
              f:antiAffinityTopologyKey:
            f:expose:
              .:
              f:exposeType:
            f:podDisruptionBudget:
              .:
              f:maxUnavailable:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:size:
        f:updateStrategy:
        f:upgradeOptions:
          .:
          f:apply:
          f:schedule:
          f:setFCV:
          f:versionServiceEndpoint:
    Manager:      argocd-application-controller
    Operation:    Update
    Time:         2022-07-25T19:13:37Z
    API Version:  psmdb.percona.com/v1-12-0
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:host:
        f:mongoImage:
        f:mongoVersion:
        f:observedGeneration:
        f:ready:
        f:replsets:
          .:
          f:rs0:
            .:
            f:initialized:
            f:ready:
            f:size:
            f:status:
        f:size:
        f:state:
    Manager:         percona-server-mongodb-operator
    Operation:       Update
    Subresource:     status
    Time:            2022-07-25T21:03:03Z
  Resource Version:  29966884201
  UID:               8c911a9d-81bc-475c-b361-45c1acb32903
Spec:
  Allow Unsafe Configurations:  false
  Backup:
    Enabled:  true
    Image:    perconalab/percona-server-mongodb-operator:main-backup
    Pitr:
      Compression Level:   6
      Compression Type:    gzip
      Enabled:             true
    Service Account Name:  percona-server-mongodb-operator
    Storages:
      Ovh:
        s3:
          Bucket:                    mongodb-backup
          Credentials Secret:        mongodb-backup-s3
          Endpoint URL:              https://s3.gra.cloud.ovh.net/
          Insecure Skip TLS Verify:  false
          Prefix:                    
          Region:                    gra
        Type:                        s3
    Tasks:
      Compression Level:  6
      Compression Type:   gzip
      Enabled:            true
      Keep:               10
      Name:               daily
      Schedule:           0 2,11,20 * * *
      Storage Name:       ovh
      Compression Level:  6
      Compression Type:   gzip
      Enabled:            true
      Keep:               3
      Name:               weekly
      Schedule:           0 21 * * 0
      Storage Name:       ovh
      Compression Level:  6
      Compression Type:   gzip
      Enabled:            true
      Keep:               6
      Name:               monthly
      Schedule:           0 0 1 * *
      Storage Name:       ovh
  Cr Version:             1.12.0
  Image:                  percona/percona-server-mongodb:5.0
  Image Pull Policy:      Always
  Pmm:
    Enabled:      false
    Image:        percona/pmm-client:2.27.0
    Server Host:  monitoring-service
  Replsets:
    Affinity:
      Anti Affinity Topology Key:  kubernetes.io/hostname
    Arbiter:
      Affinity:
        Anti Affinity Topology Key:  kubernetes.io/hostname
      Enabled:                       true
      Size:                          1
    Expose:
      Enabled:      false
      Expose Type:  ClusterIP
    Name:           rs0
    Nonvoting:
      Affinity:
        Anti Affinity Topology Key:  kubernetes.io/hostname
      Enabled:                       false
      Pod Disruption Budget:
        Max Unavailable:  1
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:     300m
          Memory:  0.5G
      Size:        3
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage:  3Gi
    Pod Disruption Budget:
      Max Unavailable:  1
    Resources:
      Limits:
        Cpu:     0.25
        Memory:  0.5G
      Requests:
        Cpu:     0.25
        Memory:  0.5G
    Size:        3
    Volume Spec:
      Persistent Volume Claim:
        Resources:
          Requests:
            Storage:         30Gi
        Storage Class Name:  csi-cinder-high-speed
  Secrets:
    Encryption Key:  mongodb-rs-encryption-key
    Users:           mongodb-user-credentials
  Sharding:
    Configsvr Repl Set:
      Affinity:
        Anti Affinity Topology Key:  kubernetes.io/hostname
      Expose:
        Enabled:      false
        Expose Type:  ClusterIP
      Pod Disruption Budget:
        Max Unavailable:  1
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:     300m
          Memory:  0.5G
      Size:        3
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage:  3Gi
    Enabled:            false
    Mongos:
      Affinity:
        Anti Affinity Topology Key:  kubernetes.io/hostname
      Expose:
        Expose Type:  ClusterIP
      Pod Disruption Budget:
        Max Unavailable:  1
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:      300m
          Memory:   0.5G
      Size:         3
  Update Strategy:  SmartUpdate
  Upgrade Options:
    Apply:                     disabled
    Schedule:                  0 2 * * *
    Set FCV:                   false
    Version Service Endpoint:  https://check.percona.com
Status:
  Conditions:
    Last Transition Time:  2022-07-25T19:13:39Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2022-07-25T19:15:48Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2022-07-25T19:15:48Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2022-07-25T19:16:10Z
    Status:                True
    Type:                  ready
    Last Transition Time:  2022-07-25T20:16:08Z
    Message:               create pbm object: create PBM connection to mongodb-rs-rs0-2.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-3.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-0.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-arbiter-0.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-1.mongodb-rs-rs0.default.svc.cluster.local:27017: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mongodb-rs-rs0-2.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup mongodb-rs-rs0-2.mongodb-rs-rs0.default.svc.cluster.local on 10.3.0.10:53: no such host }, { Addr: mongodb-rs-rs0-3.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup mongodb-rs-rs0-3.mongodb-rs-rs0.default.svc.cluster.local on 10.3.0.10:53: no such host }, { Addr: mongodb-rs-rs0-0.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: RSSecondary, Tag sets: podName=mongodb-rs-rs0-0,serviceName=mongodb-rs, Average RTT: 1130838 }, { Addr: mongodb-rs-rs0-arbiter-0.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: RSArbiter, Average RTT: 2451879 }, { Addr: mongodb-rs-rs0-1.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup mongodb-rs-rs0-1.mongodb-rs-rs0.default.svc.cluster.local on 10.3.0.10:53: no such host }, ] }
    Reason:                ErrorReconcile
    Status:                True
    Type:                  error
    Last Transition Time:  2022-07-25T20:16:09Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2022-07-25T20:17:06Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2022-07-25T20:17:06Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2022-07-25T20:17:11Z
    Status:                True
    Type:                  ready
    Last Transition Time:  2022-07-25T20:35:51Z
    Message:               create pbm object: create PBM connection to mongodb-rs-rs0-0.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-arbiter-0.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-1.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-2.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-3.mongodb-rs-rs0.default.svc.cluster.local:27017: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: mongodb-rs-rs0-0.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup mongodb-rs-rs0-0.mongodb-rs-rs0.default.svc.cluster.local on 10.3.0.10:53: no such host }, { Addr: mongodb-rs-rs0-arbiter-0.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1922353 }, { Addr: mongodb-rs-rs0-1.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup mongodb-rs-rs0-1.mongodb-rs-rs0.default.svc.cluster.local on 10.3.0.10:53: no such host }, { Addr: mongodb-rs-rs0-2.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup mongodb-rs-rs0-2.mongodb-rs-rs0.default.svc.cluster.local on 10.3.0.10:53: no such host }, { Addr: mongodb-rs-rs0-3.mongodb-rs-rs0.default.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup mongodb-rs-rs0-3.mongodb-rs-rs0.default.svc.cluster.local on 10.3.0.10:53: no such host }, ] }
    Reason:                ErrorReconcile
    Status:                True
    Type:                  error
    Last Transition Time:  2022-07-25T20:35:51Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2022-07-25T20:36:53Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2022-07-25T21:02:53Z
    Message:               create pbm object: create PBM connection to mongodb-rs-rs0-1.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-2.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-3.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-0.mongodb-rs-rs0.default.svc.cluster.local:27017,mongodb-rs-rs0-arbiter-0.mongodb-rs-rs0.default.svc.cluster.local:27017: setup a new backups db: ensure log collection: (NotWritablePrimary) not master
    Reason:                ErrorReconcile
    Status:                True
    Type:                  error
    Last Transition Time:  2022-07-25T21:03:13Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2022-07-25T21:04:47Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2022-07-25T21:04:47Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2022-07-25T21:05:03Z
    Status:                True
    Type:                  ready
    Last Transition Time:  2022-07-30T04:34:30Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2022-07-30T04:35:22Z
    Status:                True
    Type:                  ready
    Last Transition Time:  2022-07-30T04:35:23Z
    Status:                True
    Type:                  initializing
  Host:                    mongodb-rs-rs0.default.svc.cluster.local
  Mongo Image:             percona/percona-server-mongodb:5.0
  Mongo Version:           5.0.9-8
  Observed Generation:     4
  Ready:                   5
  Replsets:
    rs0:
      Initialized:  true
      Ready:        5
      Size:         5
      Status:       ready
  Size:             5
  State:            initializing
Events:             <none>

I have the same error message. The RS is in status ready but not the PerconaServerMongoDB.

K8s: v1.22.9
Percona Operator: v1.12.0

Let me know if you need more data.

1 Like
8

Here is the log

{"level":"info","ts":1659428008.1981142,"logger":"controller_psmdb","msg":"Replset size will be changed from 3 to 4 due to safe config"}
{"level":"info","ts":1659428008.1981714,"logger":"controller_psmdb","msg":"Set allowUnsafeConfigurations=true to disable safe configuration"}
{"level":"error","ts":1659428008.3457024,"logger":"controller_psmdb","msg":"failed to reconcile cluster","Request.Namespace":"default","Request.Name":"mongodb-rs","replset":"rs0","error":"fix tags: write mongo config: replSetReconfig: (Location51024) BSON field 'version' value must be < 2147483648, actual value '2147510334'","errorVerbose":"(Location51024) BSON field 'version' value must be < 2147483648, actual value '2147510334'\nreplSetReconfig\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo.WriteConfig\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo/mongo.go:232\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:263\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:493\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nfix tags: write mongo config\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:264\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:493\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

with

"msg": "failed to reconcile cluster",
"Request.Namespace":"default",
"Request.Name":"mongodb-rs",
"replset":"rs0",
"error":"fix tags: write mongo config: replSetReconfig: (Location51024) BSON field 'version' value must be < 2147483648, actual value '2147510334'",
"errorVerbose":"(Location51024) BSON field 'version' value must be < 2147483648, actual value '2147510334'
replSetReconfig
github.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo.WriteConfig
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo/mongo.go:232..."
1 Like
9

Here is an update.

I try to update the config directly on the server using db.adminCommand({ replSetReconfig ..}) and I got the same previous error

"errmsg" : "BSON field 'version' value must be < 2147483648, actual value '2147579250'",
"code" : 51024,
"codeName" : "Location51024",

But when using rs.reconfig({}} it works with no error.
Actually mongodb 4.4 add a field term in the document to reach the consensus but it looks like db.adminCommand is different from rs.reconfig

I hope this will help you to sort out this issue as my cluster is working but my replicaSet will never update.

Update 2:

It seems that not using the force:true attribute allow to specify the version number in an incremental way, varying with term property as well and it work well. Using the force attribute trigger the error listed upper.

1 Like
10

Ok found related issue. It should be addressed in v1.13.0 ([K8SPSMDB-666] Don't force reconfig by default - Percona JIRA)

1 Like
11

@Jean-Baptiste_PIN do you think you can try if K8SPSMDB-666 solves your problem?
The code is merged and running it from the main branch would do the trick.

1 Like