I have several mongo db replica sets deployed using an operator in kubernetes.
crVersion: 1.14.0
image: percona/percona-server-mongodb:6.0.4-3
also configured backup to backblaze S3 with the following configuration:
backup:
enabled: true
image: perconalab/percona-server-mongodb-operator:main-backup
resources:
limits:
memory: "1G"
requests:
cpu: "300m"
memory: "0.5G"
storages:
minio:
type: s3
s3:
bucket: xxx-backup
region: us-east-1
credentialsSecret: mongodb-backup-s3
endpointUrl: https://s3.us-west-004.backblazeb2.com
insecureSkipTLSVerify: false
prefix: "dev"
pitr:
enabled: false
oplogSpanMin: 10
compressionType: none
compressionLevel: 6
tasks:
- name: "daily-night-backup"
enabled: true
schedule: "25 6 * * *"
keep: 14
type: physical
storageName: minio
compressionType: gzip
compressionLevel: 6
Almost every routine ends with an error:
“failed to run backup: allowUnsafeConfigurations must be true to run backup on cluster with status initializing”
Unfortunately when this happens, they don’t retry. Looking at the PerconaServerMongoDB resource that failed to back up, I see that it is trying to update one of the replicas that is being backed up at the time of the backup and is going into initialization. accordingly, the backup job on this replica completes and an error occurs.
2023-09-14T06:25:25.472Z INFO balancer enabled {"controller": "psmdb-controller", "object": {"name":"mongodb","namespace":"infra"}, "namespace": "infra", "name": "mongodb", "reconcileID": "dce7f268-5bf9-4a9c-adc1-6784a2542637"}
2023-09-14T07:58:29.288Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"mongodb","namespace":"infra"}, "namespace": "infra", "name": "mongodb", "reconcileID": "9b3bdd8a-c5b5-4f13-aa88-362b4790555f", "sts": "mongodb-rs1"}
2023-09-14T07:58:29.400Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"mongodb","namespace":"infra"}, "namespace": "infra", "name": "mongodb", "reconcileID": "9b3bdd8a-c5b5-4f13-aa88-362b4790555f", "previous": "ready", "current": "initializing"}
2023-09-14T07:58:31.371Z ERROR failed to make backup {"controller": "perconaservermongodbbackup-controller", "object": {"name":"cron-mongodb-20230914062500-8tzfx","namespace":"infra"}, "namespace": "infra", "name": "cron-mongodb-20230914062500-8tzfx", "reconcileID": "fb534d13-4121-4f72-815b-6eec6477c712", "backup": "cron-mongodb-20230914062500-8tzfx", "error": "failed to run backup: allowUnsafeConfigurations must be true to run backup on cluster with status initializing", "errorVerbose": "allowUnsafeConfigurations must be true to run backup on cluster with status initializing\ngithub.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1.(*PerconaServerMongoDB).CanBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1/psmdb_types.go:932\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:207\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:186\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nfailed to run backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:208\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:186\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:124
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:188
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-09-14T07:58:31.385Z ERROR Reconciler error {"controller": "perconaservermongodbbackup-controller", "object": {"name":"cron-mongodb-20230914062500-8tzfx","namespace":"infra"}, "namespace": "infra", "name": "cron-mongodb-20230914062500-8tzfx", "reconcileID": "fb534d13-4121-4f72-815b-6eec6477c712", "error": "reconcile backup: failed to run backup: allowUnsafeConfigurations must be true to run backup on cluster with status initializing", "errorVerbose": "allowUnsafeConfigurations must be true to run backup on cluster with status initializing\ngithub.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1.(*PerconaServerMongoDB).CanBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1/psmdb_types.go:932\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:207\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:186\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nfailed to run backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:208\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:186\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nreconcile backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:188\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-09-14T07:58:34.994Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"mongodb","namespace":"infra"}, "namespace": "infra", "name": "mongodb", "reconcileID": "c562c158-ab2e-46e9-9027-66ae7598d0c4", "sts": "mongodb-cfg"}
2023-09-14T07:58:46.225Z INFO StatefulSet is not up to date {"controller": "psmdb-controller", "object": {"name":"mongodb","namespace":"infra"}, "namespace": "infra", "name": "mongodb", "reconcileID": "f883e54c-b194-4f06-8a07-8d8f68eec785", "sts": "mongodb-rs1"}
2023-09-14T07:59:53.097Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"mongodb","namespace":"infra"}, "namespace": "infra", "name": "mongodb", "reconcileID": "c1246e01-c557-4186-bdf4-18fbd9ecc801", "previous": "initializing", "current": "ready"}
At this point, I can see that the pod with the backup job is in terminating status. and operator create new pod which is no longer backed up.
Last logs in termination pod:
2023-09-14T06:38:17.000+0000 D [backup/2023-09-14T06:25:21Z] uploading: /data/db/collection-67--7343008675608480213.wt 42.29GB
2023/09/14 07:58:24 [entrypoint] got terminated, shutting down
2023/09/14 07:58:24 [entrypoint] kill `pbm-agent` (14): <nil>
Is it possible to somehow prohibit the operator from performing such operations while the backup is running, or that the backup continues to be executed from a new pod? why such strange behavior at all, because I don’t make any updates in the configurations.