# Old backup jobs in operator even after uninstallation and namespace deletion

**URL:** <https://forums.percona.com/t/old-backup-jobs-in-operator-even-after-uninstallation-and-namespace-deletion/12826>\
**Category:** Percona XtraDB Cluster 8.x\
**Created:** [November 1, 2021, 9:39pm UTC](https://forums.percona.com/t/old-backup-jobs-in-operator-even-after-uninstallation-and-namespace-deletion/12826 "2021-11-01T21:39:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michael\_Chudinov](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_chudinov/32/4732_2.png) [@Michael\_Chudinov](https://forums.percona.com/u/Michael_Chudinov)\
**Post date:** [November 1, 2021, 9:39pm UTC](https://forums.percona.com/t/old-backup-jobs-in-operator-even-after-uninstallation-and-namespace-deletion/12826/1 "2021-11-01T21:39:10Z")

</div>

I have a fresh newly installed XtraDB using last Helm chart with 3 nodes in Kubernetes.  
There is a backup configured in an external s3 storage.

It looks like the `xtradb-operator` runs old backup jobs after Helm uninstallation and new installation.  
This is very confusing!

The scenario is following:

1. Install xtradb-operator using Helm
2. Create a XtraDB cluster with **backup1** using Helm
3. Uninstall XtraDB cluster using Helm
4. Uninstall xtradb-operator using Helm
5. Delete the whole namespace for XTraDB and operator
6. Install NEW xtradb-operator using Helm
7. Create a NEW XtraDB cluster with **backup2** using Helm  
Now xtradb-oparator runs jobs from **backup1**!  
Operator continues to try to use old secrets for backup and complains they are not exists.  
Why so? How to avoid it?

---

<div class="post-metadata">

**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\
**Post date:** [November 2, 2021, 6:37am UTC](https://forums.percona.com/t/old-backup-jobs-in-operator-even-after-uninstallation-and-namespace-deletion/12826/2 "2021-11-02T06:37:39Z")

</div>

Hi @Michael_Chudinov,

What you’re saying is really strange. Could you please share your cr.yaml (`kubectl get pxc <cluster-name> -o yaml` if you don’t have) and logs from the operator pod?

---

<div class="post-metadata">

**Author:** ![Michael\_Chudinov](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_chudinov/32/4732_2.png) [@Michael\_Chudinov](https://forums.percona.com/u/Michael_Chudinov)\
**Post date:** [November 10, 2021, 5:24pm UTC](https://forums.percona.com/t/old-backup-jobs-in-operator-even-after-uninstallation-and-namespace-deletion/12826/3 "2021-11-10T17:24:06Z")

</div>

How to find that cluster name ?  
cluster is installed using helm like this

```auto
  helm install xtradb percona/pxc-db --namespace xtradb \
    --set cluster.enabled=true

```

```auto
kubectl get pxc xtradb -o yaml
Error from server (NotFound): perconaxtradbclusters.pxc.percona.com "xtradb" not found

```

---

<div class="post-metadata">

**Author:** ![Michael\_Chudinov](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_chudinov/32/4732_2.png) [@Michael\_Chudinov](https://forums.percona.com/u/Michael_Chudinov)\
**Post date:** [November 10, 2021, 5:31pm UTC](https://forums.percona.com/t/old-backup-jobs-in-operator-even-after-uninstallation-and-namespace-deletion/12826/4 "2021-11-10T17:31:32Z")

</div>

all the logs from operator like this are from **previous** installation

```auto
Failed to delete backup from s3","backup path":"s3://phpipam-platform-backup/cfcr2/xtradb-pxc-db-2021-11-10-15:20:00-full","error":"failed to list objects: Bucket name contains invalid characters"

```

The current installation is fresh new, and did not create yet any backups.  
How this NEW operator can know about backups created by previous operator ?  
That old operator was removed with helm and the namespace was removed as well!

Here are logs from the operator pod:

```auto
{"level":"info","ts":1636564266.9609706,"logger":"cmd","msg":"Runs on","platform":"kubernetes","version":"v1.17.9"}
{"level":"info","ts":1636564266.9610329,"logger":"cmd","msg":"Git commit: 76ae5055f48f7ca7d6af83b0c73c72c1bca42ca5 Git branch: release-1-9-0 Build time: 2021-08-07T18:53:29Z"}
{"level":"info","ts":1636564266.9610436,"logger":"cmd","msg":"Go Version: go1.16.7"}
{"level":"info","ts":1636564266.961053,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1636564266.9610608,"logger":"cmd","msg":"operator-sdk Version: v0.17.1"}
{"level":"info","ts":1636564266.9613683,"logger":"leader","msg":"Trying to become the leader."}
I1110 17:11:08.011883 1 request.go:621] Throttling request took 1.046439215s, request: GET:https://10.100.200.1:443/apis/longhorn.io/v1beta1?timeout=32s
{"level":"info","ts":1636564268.1230705,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1636564268.1310494,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1636564269.284617,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1636564269.2866464,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1636564269.5971894,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-percona-xtradbcluster"}
{"level":"info","ts":1636564269.597272,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1636564269.5975375,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1636564269.5975943,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1636564269.5978293,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1636564269.59795,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbcluster-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.5980296,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1636564269.5981016,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbclusterrestore-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.597812,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbclusterbackup-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.5979562,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1636564269.6984932,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbcluster-controller"}
{"level":"info","ts":1636564269.6987152,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbcluster-controller","worker count":1}
{"level":"info","ts":1636564269.6999826,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbclusterbackup-controller"}
{"level":"info","ts":1636564269.7000227,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbclusterbackup-controller","worker count":1}
{"level":"info","ts":1636564269.7001133,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbclusterrestore-controller"}
{"level":"info","ts":1636564269.7001255,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbclusterrestore-controller","worker count":1}
{"level":"info","ts":1636564269.8014824,"caller":"zapr/zapr.go:69","msg":"all workers are busy - skip backup deletion for now","backup":"cron-xtradb-pxc-db-s3-storage-2021111015100-3o0c4","in progress":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"error","ts":1636564269.9026127,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9028661,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9029667,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9030077,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9030464,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903081,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903121,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9031568,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9031918,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903257,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"info","ts":1636564274.801953,"caller":"zapr/zapr.go:69","msg":"all workers are busy - skip backup deletion for now","backup":"cron-xtradb-pxc-db-s3-storage-2021111015100-3o0c4","in progress":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"error","ts":1636564274.8020964,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8021839,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8021386,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.802246,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8022947,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8023336,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}

......
{"level":"info","ts":1636564284.8038828,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"info","ts":1636564284.8040156,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4"}
{"level":"info","ts":1636564284.804135,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4"}
{"level":"info","ts":1636564284.804224,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4"}

```

---

<div class="post-metadata">

**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\
**Post date:** [November 11, 2021, 4:20pm UTC](https://forums.percona.com/t/old-backup-jobs-in-operator-even-after-uninstallation-and-namespace-deletion/12826/5 "2021-11-11T16:20:46Z")

</div>

You can list your clusters using `kubectl get pxc`.

I couldn’t see a section called `cluster` in [helm chart.](https://github.com/percona/percona-helm-charts/blob/main/charts/pxc-db/values.yaml) Where is it coming from?

---

<div class="post-metadata">

**Author:** ![Michael\_Chudinov](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_chudinov/32/4732_2.png) [@Michael\_Chudinov](https://forums.percona.com/u/Michael_Chudinov)\
**Post date:** [November 12, 2021, 9:11am UTC](https://forums.percona.com/t/old-backup-jobs-in-operator-even-after-uninstallation-and-namespace-deletion/12826/6 "2021-11-12T09:11:37Z")

</div>

I am pretty sure that this line was in the installation instruction using helm [Install with Helm - Percona Operator for MySQL based on Percona XtraDB Cluster](https://www.percona.com/doc/kubernetes-operator-for-pxc/helm.html)

> [@Michael\_Chudinov](#):
>
> `--set cluster.enabled=true`

bu it is not there any more. Perhaps it is not needed longer.