Cron job issue for postgresql backup to s3

[Surya_Tyagi]

Descriptions

• We are using operator v2.4.0 , when i configured the backup to s3 or local (pvc) full and incremental . Using the following config in cr.yaml of the helm chart -
  spec:
  backups:
  pgbackrest:
  configuration:
  - secret:
  name: cluster1-pgbackrest-secrets
  global:
  repo1-path: /pgbackrest/repo1
  repo1-retention-full: “3”
  repo1-retention-full-type: count
  repo2-path: /pgbackrest/postgres-operator/cluster1-multi-repo/repo2
  repo2-retention-full: “3”
  repo2-retention-full-type: count
  image: perconalab/percona-postgresql-operator:main-ppg16-pgbackrest
  manual:
  options:
  - --type=full
  repoName: repo1
  repoHost:
  affinity:
  nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  nodeSelectorTerms:
  - matchExpressions:
  - key: db
  operator: In
  values:
  - enabled
  podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - podAffinityTerm:
  labelSelector:
  matchLabels:
  postgres-operator.crunchydata.com/data: pgbackrest
  topologyKey: kubernetes.io/hostname
  weight: 1
  tolerations:
  - effect: NoSchedule
  key: db
  operator: Equal
  value: allowed
  repos:
  - name: repo1
  schedules:
  full: 0 0 * * 6
  incremental: 0 1 * * 1-6
  volume:
  volumeClaimSpec:
  accessModes:
  - ReadWriteOnce
  resources:
  requests:
  storage: 500Gi
  - name: repo2
  s3:
  bucket: postgres-backup
  endpoint: s3.us-east-2…
  region: us-east-2
  schedules:
  full: 0 0 * * 0
  incremental: 0 3 * * 1-6

• When this get schedules according to cron it works fine for first time but as per cron when second time backup job tries to schedule pod again to run backup , it does not get scheduled.

• If we check kubectl get events we get error message -

not starting job because prior execution is running and concurrency policy is forbid

• And the job which ran first time for backup have been successfully completed , you can see in screen shot .

Steps to Reproduce:

[Step-by-step instructions on how to reproduce the issue, including any specific settings or configurations]

• Install percona operator postgresql
• Add config for s3 or local back using cron
• It will run successfully for the first time and will say the above as mentioned at second time onwards.

Version:

v2.4.0

Additional Information:

• if i edit and add ( ttlSecondsAfterFinished: 6000 ) in cronjob.batch it works as it deletes the job entry after it finished . so , when corn setup new job it doesn’t find the old entry and run the backup successfully .
• But i am unable to set this ttlSecondsAfterFinished in cr.yaml for persistence , it does not recognise this parameter . otherwise whenever i do new deployment this parameter will be gone.

[bljurg_geiwn]

@Surya_Tyagi
were you able to resolve this?

[Surya_Tyagi]

not yet @bljurg_geiwn

[Slava_Sarzhan]

Hi @Surya_Tyagi , as I can see you have cronjob objects but pg operator does not use k8s cron jobs in v2.4.0. The operator schedules backups by himself using “robfig/cron” go module.
P.S. I have tried to reproduce the issue using the latest PGO v2.5.0 but without any results.

❯ kubectl get pg-backup
NAME                                 CLUSTER            REPO    DESTINATION                                                         STATUS      TYPE   COMPLETED   AGE
scheduled-backup-backup-hvl6-lxtgl   scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   full   75m         77m
scheduled-backup-repo1-full-242tw    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   full   64m         67m
scheduled-backup-repo1-full-5w5kj    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   full   54m         56m
scheduled-backup-repo1-full-8wwx7    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   full   4m8s        6m57s
scheduled-backup-repo1-full-k8f88    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   full   68m         72m
scheduled-backup-repo1-incr-26w6s    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   incr   3m31s       6m28s
scheduled-backup-repo1-incr-2h4d9    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   incr   53m         56m
scheduled-backup-repo1-incr-ggrfj    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   incr   16m         16m
scheduled-backup-repo1-incr-rsl97    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   incr   36m         36m
scheduled-backup-repo1-incr-wd22d    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   incr   46m         46m
scheduled-backup-repo1-incr-zh6rp    scheduled-backup   repo1   s3://pg-operator-testing/backrestrepo/postgres-operator/890/repo1   Succeeded   incr   26m         26m