Old backup jobs in operator even after uninstallation and namespace deletion

I have a fresh newly installed XtraDB using last Helm chart with 3 nodes in Kubernetes.
There is a backup configured in an external s3 storage.

It looks like the xtradb-operator runs old backup jobs after Helm uninstallation and new installation.
This is very confusing!

The scenario is following:

  1. Install xtradb-operator using Helm
  2. Create a XtraDB cluster with backup1 using Helm
  3. Uninstall XtraDB cluster using Helm
  4. Uninstall xtradb-operator using Helm
  5. Delete the whole namespace for XTraDB and operator
  6. Install NEW xtradb-operator using Helm
  7. Create a NEW XtraDB cluster with backup2 using Helm
    Now xtradb-oparator runs jobs from backup1 !
    Operator continues to try to use old secrets for backup and complains they are not exists.
    Why so? How to avoid it?
1 Like

Hi @Michael_Chudinov,

What you’re saying is really strange. Could you please share your cr.yaml (kubectl get pxc <cluster-name> -o yaml if you don’t have) and logs from the operator pod?

1 Like

How to find that cluster name ?
cluster is installed using helm like this

  helm install xtradb percona/pxc-db --namespace xtradb \
    --set cluster.enabled=true
kubectl get pxc xtradb -o yaml
Error from server (NotFound): perconaxtradbclusters.pxc.percona.com "xtradb" not found
1 Like

all the logs from operator like this are from previous installation

Failed to delete backup from s3","backup path":"s3://phpipam-platform-backup/cfcr2/xtradb-pxc-db-2021-11-10-15:20:00-full","error":"failed to list objects: Bucket name contains invalid characters"

The current installation is fresh new, and did not create yet any backups.
How this NEW operator can know about backups created by previous operator ?
That old operator was removed with helm and the namespace was removed as well!

Here are logs from the operator pod:

{"level":"info","ts":1636564266.9609706,"logger":"cmd","msg":"Runs on","platform":"kubernetes","version":"v1.17.9"}
{"level":"info","ts":1636564266.9610329,"logger":"cmd","msg":"Git commit: 76ae5055f48f7ca7d6af83b0c73c72c1bca42ca5 Git branch: release-1-9-0 Build time: 2021-08-07T18:53:29Z"}
{"level":"info","ts":1636564266.9610436,"logger":"cmd","msg":"Go Version: go1.16.7"}
{"level":"info","ts":1636564266.961053,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1636564266.9610608,"logger":"cmd","msg":"operator-sdk Version: v0.17.1"}
{"level":"info","ts":1636564266.9613683,"logger":"leader","msg":"Trying to become the leader."}
I1110 17:11:08.011883       1 request.go:621] Throttling request took 1.046439215s, request: GET:https://10.100.200.1:443/apis/longhorn.io/v1beta1?timeout=32s
{"level":"info","ts":1636564268.1230705,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1636564268.1310494,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1636564269.284617,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1636564269.2866464,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1636564269.5971894,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-percona-xtradbcluster"}
{"level":"info","ts":1636564269.597272,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1636564269.5975375,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1636564269.5975943,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1636564269.5978293,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1636564269.59795,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbcluster-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.5980296,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1636564269.5981016,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbclusterrestore-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.597812,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbclusterbackup-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.5979562,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1636564269.6984932,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbcluster-controller"}
{"level":"info","ts":1636564269.6987152,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbcluster-controller","worker count":1}
{"level":"info","ts":1636564269.6999826,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbclusterbackup-controller"}
{"level":"info","ts":1636564269.7000227,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbclusterbackup-controller","worker count":1}
{"level":"info","ts":1636564269.7001133,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbclusterrestore-controller"}
{"level":"info","ts":1636564269.7001255,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbclusterrestore-controller","worker count":1}
{"level":"info","ts":1636564269.8014824,"caller":"zapr/zapr.go:69","msg":"all workers are busy - skip backup deletion for now","backup":"cron-xtradb-pxc-db-s3-storage-2021111015100-3o0c4","in progress":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"error","ts":1636564269.9026127,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9028661,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9029667,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9030077,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9030464,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903081,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903121,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9031568,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9031918,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903257,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"info","ts":1636564274.801953,"caller":"zapr/zapr.go:69","msg":"all workers are busy - skip backup deletion for now","backup":"cron-xtradb-pxc-db-s3-storage-2021111015100-3o0c4","in progress":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"error","ts":1636564274.8020964,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8021839,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8021386,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.802246,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8022947,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8023336,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}

......
{"level":"info","ts":1636564284.8038828,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"info","ts":1636564284.8040156,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4"}
{"level":"info","ts":1636564284.804135,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4"}
{"level":"info","ts":1636564284.804224,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4"}

1 Like

You can list your clusters using kubectl get pxc.

I couldn’t see a section called cluster in helm chart. Where is it coming from?

1 Like

I am pretty sure that this line was in the installation instruction using helm Install Percona XtraDB Cluster using Helm

bu it is not there any more. Perhaps it is not needed longer.