Old backup jobs in operator even after uninstallation and namespace deletion

MySQL & MariaDB Percona XtraDB Cluster 8.x
1

I have a fresh newly installed XtraDB using last Helm chart with 3 nodes in Kubernetes.
There is a backup configured in an external s3 storage.

It looks like the xtradb-operator runs old backup jobs after Helm uninstallation and new installation.
This is very confusing!

The scenario is following:

  1. Install xtradb-operator using Helm
  2. Create a XtraDB cluster with backup1 using Helm
  3. Uninstall XtraDB cluster using Helm
  4. Uninstall xtradb-operator using Helm
  5. Delete the whole namespace for XTraDB and operator
  6. Install NEW xtradb-operator using Helm
  7. Create a NEW XtraDB cluster with backup2 using Helm
    Now xtradb-oparator runs jobs from backup1 !
    Operator continues to try to use old secrets for backup and complains they are not exists.
    Why so? How to avoid it?
1 Like
2

Hi @Michael_Chudinov,

What you’re saying is really strange. Could you please share your cr.yaml (kubectl get pxc <cluster-name> -o yaml if you don’t have) and logs from the operator pod?

1 Like
3

How to find that cluster name ?
cluster is installed using helm like this

  helm install xtradb percona/pxc-db --namespace xtradb \
    --set cluster.enabled=true

kubectl get pxc xtradb -o yaml
Error from server (NotFound): perconaxtradbclusters.pxc.percona.com "xtradb" not found
1 Like
4

all the logs from operator like this are from previous installation

Failed to delete backup from s3","backup path":"s3://phpipam-platform-backup/cfcr2/xtradb-pxc-db-2021-11-10-15:20:00-full","error":"failed to list objects: Bucket name contains invalid characters"

The current installation is fresh new, and did not create yet any backups.
How this NEW operator can know about backups created by previous operator ?
That old operator was removed with helm and the namespace was removed as well!

Here are logs from the operator pod:

{"level":"info","ts":1636564266.9609706,"logger":"cmd","msg":"Runs on","platform":"kubernetes","version":"v1.17.9"}
{"level":"info","ts":1636564266.9610329,"logger":"cmd","msg":"Git commit: 76ae5055f48f7ca7d6af83b0c73c72c1bca42ca5 Git branch: release-1-9-0 Build time: 2021-08-07T18:53:29Z"}
{"level":"info","ts":1636564266.9610436,"logger":"cmd","msg":"Go Version: go1.16.7"}
{"level":"info","ts":1636564266.961053,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1636564266.9610608,"logger":"cmd","msg":"operator-sdk Version: v0.17.1"}
{"level":"info","ts":1636564266.9613683,"logger":"leader","msg":"Trying to become the leader."}
I1110 17:11:08.011883       1 request.go:621] Throttling request took 1.046439215s, request: GET:https://10.100.200.1:443/apis/longhorn.io/v1beta1?timeout=32s
{"level":"info","ts":1636564268.1230705,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1636564268.1310494,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1636564269.284617,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1636564269.2866464,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1636564269.5971894,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-percona-xtradbcluster"}
{"level":"info","ts":1636564269.597272,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1636564269.5975375,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1636564269.5975943,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1636564269.5978293,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1636564269.59795,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbcluster-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.5980296,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1636564269.5981016,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbclusterrestore-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.597812,"logger":"controller","msg":"Starting EventSource","controller":"perconaxtradbclusterbackup-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1636564269.5979562,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1636564269.6984932,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbcluster-controller"}
{"level":"info","ts":1636564269.6987152,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbcluster-controller","worker count":1}
{"level":"info","ts":1636564269.6999826,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbclusterbackup-controller"}
{"level":"info","ts":1636564269.7000227,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbclusterbackup-controller","worker count":1}
{"level":"info","ts":1636564269.7001133,"logger":"controller","msg":"Starting Controller","controller":"perconaxtradbclusterrestore-controller"}
{"level":"info","ts":1636564269.7001255,"logger":"controller","msg":"Starting workers","controller":"perconaxtradbclusterrestore-controller","worker count":1}
{"level":"info","ts":1636564269.8014824,"caller":"zapr/zapr.go:69","msg":"all workers are busy - skip backup deletion for now","backup":"cron-xtradb-pxc-db-s3-storage-2021111015100-3o0c4","in progress":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"error","ts":1636564269.9026127,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9028661,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9029667,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9030077,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9030464,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903081,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903121,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9031568,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.9031918,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564269.903257,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"info","ts":1636564274.801953,"caller":"zapr/zapr.go:69","msg":"all workers are busy - skip backup deletion for now","backup":"cron-xtradb-pxc-db-s3-storage-2021111015100-3o0c4","in progress":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016200-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4, cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015400-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111015500-3o0c4, cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"error","ts":1636564274.8020964,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016100-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8021839,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8021386,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.802246,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8022947,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}
{"level":"error","ts":1636564274.8023336,"caller":"zapr/zapr.go:128","msg":"failed to create s3 client for backup","backup":"cron-xtradb-pxc-db-s3-storage-2021111015300-3o0c4","error":"failed to get secret: Secret \"xtradb-pxc-db-s3-s3-storage\" not found","errorVerbose":"Secret \"xtradb-pxc-db-s3-s3-storage\" not found\nfailed to get secret\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).s3cli\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:417\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:327\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).runS3BackupFinalizer\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:329"}

......
{"level":"info","ts":1636564284.8038828,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111016400-3o0c4"}
{"level":"info","ts":1636564284.8040156,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111016300-3o0c4"}
{"level":"info","ts":1636564284.804135,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-2021111015200-3o0c4"}
{"level":"info","ts":1636564284.804224,"caller":"zapr/zapr.go:69","msg":"deleting backup from s3","name":"cron-xtradb-pxc-db-s3-storage-202111101600-3o0c4"}
1 Like
5

You can list your clusters using kubectl get pxc.

I couldn’t see a section called cluster in helm chart. Where is it coming from?

1 Like
6

I am pretty sure that this line was in the installation instruction using helm Install with Helm - Percona Operator for MySQL based on Percona XtraDB Cluster

bu it is not there any more. Perhaps it is not needed longer.