How to automatically remove backup cron pods?

b602fa9c · February 21, 2023, 9:57am

Currently when a backup job completes the pod is left in “Completed” status. In an environment where backups have to be kept for a long time the number of these pods build up and become a pain when using kubectl for other tasks.

Is there any way to have these pods be automatically removed after some time but keep the pxc-backup resource? Potentially by setting “.spec.ttlSecondsAfterFinished” on the job (Automatic Cleanup for Finished Jobs | Kubernetes)

I couldn’t see any way to set that in the operator.

github.com

percona/percona-xtradb-cluster-operator/blob/main/pkg/pxc/backup/job.go#L41


      
          		},
          		ObjectMeta: metav1.ObjectMeta{
          			Name:        labels["job-name"],
          			Namespace:   cr.Namespace,
          			Labels:      labels,
          			Annotations: cluster.Spec.Backup.Storages[cr.Spec.StorageName].Annotations,
          		},
          	}
          }
          
          
func (bcp *Backup) JobSpec(spec api.PXCBackupSpec, cluster *api.PerconaXtraDBCluster, job *batchv1.Job) (batchv1.JobSpec, error) {
          	manualSelector := true
          	backoffLimit := int32(10)
          	if cluster.CompareVersionWith("1.11.0") >= 0 && cluster.Spec.Backup.BackoffLimit != nil {
          		backoffLimit = *cluster.Spec.Backup.BackoffLimit
          	}
          	verifyTLS := true
          	if cluster.Spec.Backup.Storages[spec.StorageName].VerifyTLS != nil {
          		verifyTLS = *cluster.Spec.Backup.Storages[spec.StorageName].VerifyTLS
          	}
          	return batchv1.JobSpec{

Slava_Sarzhan · February 21, 2023, 4:56pm

@b602fa9c Do you use keep option percona-xtradb-cluster-operator/cr.yaml at main · percona/percona-xtradb-cluster-operator · GitHub in CR?

b602fa9c · February 28, 2023, 4:48pm

Yes we set keep to 0

Slava_Sarzhan · February 28, 2023, 5:01pm

In this case you disabled it. If you want to keep e.g. last 5 backups you need to set it in 5 and all jobs will be cleaned up as well .

b602fa9c · March 13, 2023, 11:40am

I want to keep the backups but not the backup jobs

jackieh · July 5, 2023, 9:58pm

Hi b602fa9c, I am running into the same problem, where failed or succeeded jobs linger. I also have the ‘keep’ option set, but I believe you are right that this is the retention of backups on the storage destination.

Did you ever get an answer on how to solve this problem?

b602fa9c · July 20, 2023, 1:48pm

No, I’m still looking for a solution to this.

elie_mouawad · September 2, 2023, 9:48pm

+1 on this - ttlSecondsAfterFinished is a good idea

Sergey_Pronin · September 4, 2023, 7:23am

@elie_mouawad what would be the best default for this?