# Percona XtraDB Operator version 8.0 fail to backup both s3 and local storage

**URL:** <https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084>\
**Category:** Percona Operator for MySQL\
**Tags:** percona\
**Created:** [September 12, 2024, 3:55am UTC](https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084 "2024-09-12T03:55:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pratak\_Eak](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pratak_eak/32/18056_2.png) [@Pratak\_Eak](https://forums.percona.com/u/Pratak_Eak)\
**Post date:** [September 12, 2024, 3:55am UTC](https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084/1 "2024-09-12T03:55:39Z")

</div>

## Description:

I am fail to run backup percona extradb operator manual and cron on private kubenetes microk8s. How do I fix this error?

pxc-backup-minio.yaml

```auto
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterBackup
metadata:
  finalizers:
    - percona.com/delete-backup
  name: backup-minio
spec:
  pxcCluster: cluster
  storageName: minio

```

## Steps to Reproduce:

run this command to start backup manual  
k -n pxc apply -f pxc-backup-minio.yaml

check backup result  
`sh> k -n pxc get pxc-backup`

> NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE  
> cron-cluster-minio-proen-20249120048-372f8 cluster minio-proen s3://backup/cluster-2024-09-12-00:00:48-full Failed 3h47m  
> backup-minio cluster minio-proen s3://backup/cluster-2024-09-12-03:34:00-full Failed 14m

## Version:

pxc image  
perconalab/percona-xtradb-cluster-operator:main-pxc8.0  
backup image  
perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup

## Logs:

FATAL: /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs\_group.cpp:group\_check\_proto\_ver():341: Group requested gcs\_proto\_ver: 4, max supported by this node: 2.Upgrade the node before joining this group.Need to abort.

> Defaulted container “xtrabackup” out of: xtrabackup, backup-init (init)
> 
> - LIB\_PATH=/usr/lib/pxc
> - . /usr/lib/pxc/backup.sh  
> ++ set -o errexit  
> ++ SST\_INFO\_NAME=sst\_info  
> ++ XBCLOUD\_ARGS='–curl-retriable-errors=7 ’  
> ++ INSECURE\_ARG=  
> ++ ‘[’ -n true ‘]’  
> ++ [[true == \f\a\l\s\e]]  
> ++ S3\_BUCKET\_PATH=cluster-2024-09-12-03:34:00-full  
> +++ date +%F-%H-%M  
> ++ BACKUP\_PATH=cluster-pxc-2024-09-12-03-36-xtrabackup.stream
> - GARBD\_OPTS=
> - check\_ssl
> - CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
> - ‘[’ -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ‘]’
> - SSL\_DIR=/etc/mysql/ssl
> - ‘[’ -f /etc/mysql/ssl/ca.crt ‘]’
> - CA=/etc/mysql/ssl/ca.crt
> - SSL\_INTERNAL\_DIR=/etc/mysql/ssl-internal
> - ‘[’ -f /etc/mysql/ssl-internal/ca.crt ‘]’
> - CA=/etc/mysql/ssl-internal/ca.crt
> - KEY=/etc/mysql/ssl/tls.key
> - CERT=/etc/mysql/ssl/tls.crt
> - ‘[’ -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ‘]’
> - KEY=/etc/mysql/ssl-internal/tls.key
> - CERT=/etc/mysql/ssl-internal/tls.crt
> - ‘[’ -f /etc/mysql/ssl-internal/ca.crt -a -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ‘]’
> - GARBD\_OPTS=‘socket.ssl\_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl\_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl\_key=/etc/mysql/ssl-internal/tls.key;socket.ssl\_cipher=;pc.weight=0;’
> - ‘[’ -n backup ‘]’
> - clean\_backup\_s3
> - mc\_add\_bucket\_dest
> - echo '+ mc -C /tmp/mc config host add dest [https://s3.aspiredigitalgroup.com.au](https://s3.aspiredigitalgroup.com.au) ACCESS\_KEY\_ID SECRET\_ACCESS\_KEY ’
> - mc -C /tmp/mc config host add dest [https://s3.aspiredigitalgroup.com.au](https://s3.aspiredigitalgroup.com.au) ACCESS\_KEY\_ID SECRET\_ACCESS\_KEY  
> Added `dest` successfully.
> - is\_object\_exist backup cluster-2024-09-12-03:34:00-full.sst\_info
> - local bucket=backup
> - local object=cluster-2024-09-12-03:34:00-full.sst\_info  
> ++ mc -C /tmp/mc --json ls dest/backup/cluster-2024-09-12-03:34:00-full.sst\_info  
> ++ jq .status
> - [[-n ‘’]]
> - is\_object\_exist backup cluster-2024-09-12-03:34:00-full/
> - local bucket=backup
> - local object=cluster-2024-09-12-03:34:00-full/  
> ++ mc -C /tmp/mc --json ls dest/backup/cluster-2024-09-12-03:34:00-full/  
> ++ jq .status
> - [[-n ‘’]]
> - request\_streaming  
> ++ hostname -i  
> ++ sed -E ‘s/._\b([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})\b._/\1/’
> - local LOCAL\_IP=10.1.167.63  
> ++ get\_backup\_source  
> +++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=cluster-pxc  
> +++ grep wsrep\_cluster\_size  
> +++ sort  
> +++ tail -1  
> +++ cut -d : -f 12  
> ++ CLUSTER\_SIZE=3  
> ++ ‘[’ -z 3 ‘]’  
> +++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=cluster-pxc  
> +++ grep wsrep\_ready:ON:wsrep\_connected:ON:wsrep\_local\_state\_comment:Synced:wsrep\_cluster\_status:Primary  
> +++ sort -r  
> +++ tail -1  
> +++ cut -d : -f 2  
> +++ cut -d . -f 1  
> ++ FIRST\_NODE=cluster-pxc-0  
> ++ SKIP\_FIRST\_POD=‘|’  
> ++ (( 3 \> 1 ))  
> ++ SKIP\_FIRST\_POD=cluster-pxc-0  
> ++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=cluster-pxc  
> ++ grep wsrep\_ready:ON:wsrep\_connected:ON:wsrep\_local\_state\_comment:Synced:wsrep\_cluster\_status:Primary  
> ++ grep -v cluster-pxc-0  
> ++ sort  
> ++ tail -1  
> ++ cut -d : -f 2  
> ++ cut -d . -f 1
> - local NODE\_NAME=cluster-pxc-2
> - ‘[’ -z cluster-pxc-2 ‘]’
> - set +o errexit
> - log INFO ‘Garbd was started’  
> 2024-09-12 03:36:11 [INFO] Garbd was started
> - garbd --address ‘gcomm://cluster-pxc-2.cluster-pxc?gmcast.listen\_addr=tcp://0.0.0.0:4567’ --donor cluster-pxc-2 --group cluster-pxc --options ‘socket.ssl\_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl\_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl\_key=/etc/mysql/ssl-internal/tls.key;socket.ssl\_cipher=;pc.weight=0;’ --sst xtrabackup-v2:10.1.167.63:4444/xtrabackup\_sst//1 --recv-script=/usr/bin/run\_backup.sh  
> 2024-09-12 03:36:11.590 INFO: CRC-32C: using 64-bit x86 acceleration.  
> 2024-09-12 03:36:11.590 INFO: Read config:  
> daemon: 0  
> name: garb  
> address: gcomm://cluster-pxc-2.cluster-pxc?gmcast.listen\_addr=tcp://0.0.0.0:4567  
> group: cluster-pxc  
> sst: xtrabackup-v2:10.1.167.63:4444/xtrabackup\_sst//1  
> donor: cluster-pxc-2  
> options: socket.ssl\_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl\_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl\_key=/etc/mysql/ssl-internal/tls.key;socket.ssl\_cipher=;pc.weight=0;; gcs.fc\_limit=9999999; gcs.fc\_factor=1.0; gcs.fc\_single\_primary=yes; socket.ssl=YES  
> cfg:  
> log:  
> recv\_script: /usr/bin/run\_backup.sh  
> workdir:
> 
> 2024-09-12 03:36:11.591 WARN: SSL compression is not effective. The option socket.ssl\_compression is deprecated and will be removed in future releases.  
> 2024-09-12 03:36:11.591 WARN: Parameter ‘socket.ssl\_compression’ is deprecated and will be removed in future versions  
> 2024-09-12 03:36:11.598 INFO: protonet asio version 0  
> 2024-09-12 03:36:11.600 INFO: Using CRC-32C for message checksums.  
> 2024-09-12 03:36:11.600 INFO: backend: asio  
> 2024-09-12 03:36:11.600 INFO: gcomm thread scheduling priority set to other:0  
> 2024-09-12 03:36:11.600 INFO: Fail to access the file (./gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown  
> 2024-09-12 03:36:11.600 INFO: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown  
> 2024-09-12 03:36:11.600 INFO: GMCast version 0  
> 2024-09-12 03:36:11.601 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567  
> 2024-09-12 03:36:11.601 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1  
> 2024-09-12 03:36:11.602 INFO: EVS version 1  
> 2024-09-12 03:36:11.602 INFO: gcomm: connecting to group ‘cluster-pxc’, peer ‘cluster-pxc-2.cluster-pxc:’  
> 2024-09-12 03:36:11.611 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) connection established to af8cc657-b8d9 ssl://10.1.109.118:4567  
> 2024-09-12 03:36:11.611 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: ssl://10.1.106.165:4567 ssl://10.1.9.71:4567  
> 2024-09-12 03:36:11.718 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) connection established to cee986a2-a54c ssl://10.1.9.71:4567  
> 2024-09-12 03:36:11.730 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) connection established to ea92447e-b4de ssl://10.1.106.165:4567  
> 2024-09-12 03:36:11.730 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) connection established to ea92447e-b4de ssl://10.1.106.165:4567  
> 2024-09-12 03:36:12.104 INFO: EVS version upgrade 0 → 1  
> 2024-09-12 03:36:12.104 INFO: declaring af8cc657-b8d9 at ssl://10.1.109.118:4567 stable  
> 2024-09-12 03:36:12.104 INFO: declaring cee986a2-a54c at ssl://10.1.9.71:4567 stable  
> 2024-09-12 03:36:12.104 INFO: declaring ea92447e-b4de at ssl://10.1.106.165:4567 stable  
> 2024-09-12 03:36:12.105 INFO: PC protocol upgrade 0 → 1  
> 2024-09-12 03:36:12.105 INFO: Node af8cc657-b8d9 state primary  
> 2024-09-12 03:36:12.106 INFO: Current view of cluster as seen by this node  
> view (view\_id(PRIM,27cfeea8-badb,88)  
> memb {  
> 27cfeea8-badb,0  
> af8cc657-b8d9,0  
> cee986a2-a54c,0  
> ea92447e-b4de,0  
> }  
> joined {  
> }  
> left {  
> }  
> partitioned {  
> }  
> )  
> 2024-09-12 03:36:12.106 INFO: Save the discovered primary-component to disk  
> 2024-09-12 03:36:12.106 WARN: open file(./gvwstate.dat.tmp) failed(Permission denied)  
> 2024-09-12 03:36:12.602 INFO: gcomm: connected  
> 2024-09-12 03:36:12.602 INFO: Changing maximum packet size to 64500, resulting msg size: 32636  
> 2024-09-12 03:36:12.602 INFO: Shifting CLOSED → OPEN (TO: 0)  
> 2024-09-12 03:36:12.602 INFO: Opened channel ‘cluster-pxc’  
> 2024-09-12 03:36:12.603 INFO: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 0, memb\_num = 4  
> 2024-09-12 03:36:12.603 INFO: STATE\_EXCHANGE: sent state UUID: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8  
> 2024-09-12 03:36:12.604 INFO: STATE EXCHANGE: sent state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8  
> 2024-09-12 03:36:12.605 INFO: STATE EXCHANGE: got state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8 from 0 (garb)  
> 2024-09-12 03:36:12.605 INFO: STATE EXCHANGE: got state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8 from 1 (cluster-pxc-2)  
> 2024-09-12 03:36:12.605 INFO: STATE EXCHANGE: got state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8 from 2 (cluster-pxc-1)  
> 2024-09-12 03:36:12.605 INFO: STATE EXCHANGE: got state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8 from 3 (cluster-pxc-0)  
> 2024-09-12 03:36:12.605 INFO: Quorum results:  
> version = 6,  
> component = PRIMARY,  
> conf\_id = 87,  
> members = 3/4 (primary/total),  
> act\_id = 2345,  
> last\_appl. = 2325,  
> protocols = 4/11/4 (gcs/repl/appl),  
> vote policy= 0,  
> group UUID = f92fb1be-6ffe-11ef-9a87-0ac4a615d14f  
> 2024-09-12 03:36:12.605 FATAL: /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs\_group.cpp:group\_check\_proto\_ver():341: Group requested gcs\_proto\_ver: 4, max supported by this node: 2.Upgrade the node before joining this group.Need to abort.  
> 2024-09-12 03:36:12.605 INFO: garbd: Terminated.  
> /usr/bin/backup.sh: line 68: 368 Aborted garbd --address “gcomm://$NODE\_NAME.$PXC\_SERVICE?gmcast.listen\_addr=tcp://0.0.0.0:4567” --donor “$NODE\_NAME” --group “$PXC\_SERVICE” --options “$GARBD\_OPTS” --sst “xtrabackup-v2:$LOCAL\_IP:4444/xtrabackup\_sst//1” --recv-script=“/usr/bin/run\_backup.sh”
> 
> - EXID\_CODE=134
> - ‘[’ -f /tmp/backup-is-completed ‘]’
> - log ERROR ‘Backup was finished unsuccessfull’  
> 2024-09-12 03:36:12 [ERROR] Backup was finished unsuccessfull
> - exit 134

## Expected Result:

NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE  
cron-cluster-minio-proen-20249120048-372f8 cluster minio-proen s3://backup/cluster-2024-09-12-00:00:48-full Completed 3h47m  
backup-minio cluster minio-proen s3://backup/cluster-2024-09-12-03:34:00-full Completed 14m

## Actual Result:

NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE  
cron-cluster-minio-proen-20249120048-372f8 cluster minio-proen s3://backup/cluster-2024-09-12-00:00:48-full Failed 3h47m  
backup-minio cluster minio-proen s3://backup/cluster-2024-09-12-03:34:00-full Failed 14m

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [September 12, 2024, 3:46pm UTC](https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084/2 "2024-09-12T15:46:25Z")

</div>

> [@Pratak\_Eak](#):
>
> pxc image  
> perconalab/percona-xtradb-cluster-operator:main-pxc8.0  
> backup image  
> perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup

Hi @Pratak_Eak , please do not use images with main tag. The latest PXCO release is 1.15.0 [percona-xtradb-cluster-operator/deploy/cr.yaml at v1.15.0 · percona/percona-xtradb-cluster-operator · GitHub](https://github.com/percona/percona-xtradb-cluster-operator/blob/v1.15.0/deploy/cr.yaml)  
You need to use this ver. Images with main tag based on main branch and can be broken during the development cycle.

---

<div class="post-metadata">

**Author:** ![Pratak\_Eak](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pratak_eak/32/18056_2.png) [@Pratak\_Eak](https://forums.percona.com/u/Pratak_Eak)\
**Post date:** [September 15, 2024, 4:03am UTC](https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084/3 "2024-09-15T04:03:10Z")

</div>

Hi @Slava_Sarzhan  
all pod start successfully but pcx status still error. I custom persistence storage to specific storageclass

```auto
kubectl -n pxc describe pxc

```

Status:  
Backup:  
Conditions:  
Last Transition Time: 2024-09-15T05:27:21Z  
Status: True  
Type: initializing  
Last Transition Time: 2024-09-15T05:29:27Z  
Message: requested storage (187Gi) is less than actual storage (200Gi)  
Reason: ErrorReconcile  
Status: True  
Type: error  
Last Transition Time: 2024-09-15T05:41:10Z  
Status: True  
Type: initializing  
Last Transition Time: 2024-09-15T05:41:11Z  
Message: requested storage (187Gi) is less than actual storage (200Gi)  
Reason: ErrorReconcile  
Status: True  
Type: error  
Haproxy:  
Label Selector Path: [app.kubernetes.io/component=haproxy,app.kubernetes.io/instance=cluster,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster](http://app.kubernetes.io/component=haproxy,app.kubernetes.io/instance=cluster,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster)  
Ready: 3  
Size: 3  
Status: ready  
Host: cluster-haproxy.pxc  
Logcollector:  
Message:  
Error: requested storage (187Gi) is less than actual storage (200Gi)  
Observed Generation: 2

here is cr.yaml

```auto
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: cluster
  finalizers:
    - percona.com/delete-pxc-pods-in-order
# - percona.com/delete-ssl
# - percona.com/delete-proxysql-pvc
# - percona.com/delete-pxc-pvc
# annotations:
# percona.com/issue-vault-token: "true"
spec:
  crVersion: 1.15.0
# ignoreAnnotations:
# - iam.amazonaws.com/role
# ignoreLabels:
# - rack
# secretsName: cluster1-secrets
# vaultSecretName: keyring-secret-vault
# sslSecretName: cluster1-ssl
# sslInternalSecretName: cluster1-ssl-internal
# logCollectorSecretName: cluster1-log-collector-secrets
# initContainer:
# image: percona/percona-xtradb-cluster-operator:1.15.0
# resources:
# requests:
# memory: 100M
# cpu: 100m
# limits:
# memory: 200M
# cpu: 200m
# enableCRValidationWebhook: true
  tls:
    enabled: true
# SANs:
# - pxc-1.example.com
# - pxc-2.example.com
# - pxc-3.example.com
# issuerConf:
# name: special-selfsigned-issuer
# kind: ClusterIssuer
# group: cert-manager.io
  unsafeFlags:
# tls: false
# pxcSize: false
# proxySize: false
    backupIfUnhealthy: true
  # pause: true
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: "0 4 * * *"
  pxc:
    size: 3
    image: percona/percona-xtradb-cluster:8.0.36-28.1
    autoRecovery: true
# expose:
# enabled: true
# type: LoadBalancer
# externalTrafficPolicy: Local
# internalTrafficPolicy: Local
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# loadBalancerIP: 127.0.0.1
# annotations:
# networking.gke.io/load-balancer-type: "Internal"
# labels:
# rack: rack-22
# replicationChannels:
# - name: pxc1_to_pxc2
# isSource: true
# - name: pxc2_to_pxc1
# isSource: false
# configuration:
# sourceRetryCount: 3
# sourceConnectRetry: 60
# ssl: false
# sslSkipVerify: true
# ca: '/etc/mysql/ssl/ca.crt'
# sourcesList:
# - host: 10.95.251.101
# port: 3306
# weight: 100
# schedulerName: mycustom-scheduler
# readinessDelaySec: 15
# livenessDelaySec: 600
    configuration: |
      [mysqld]
      wsrep_debug=CLIENT
      wsrep_provider_options="gcache.size=1G; gcache.recover=yes"
      max_connections = 500
      interactive_timeout = 300
      wait_timeout = 300
      skip_name_resolve
      [sst]
      xbstream-opts=--decompress
      [xtrabackup]
      compress=lz4
      # for PXC 5.7
      # [xtrabackup]
      # compress
# imagePullSecrets:
# - name: private-registry-credentials
# priorityClassName: high-priority
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# readinessProbes:
# initialDelaySeconds: 15
# timeoutSeconds: 15
# periodSeconds: 30
# successThreshold: 1
# failureThreshold: 5
# livenessProbes:
# initialDelaySeconds: 300
# timeoutSeconds: 5
# periodSeconds: 10
# successThreshold: 1
# failureThreshold: 3
# containerSecurityContext:
# privileged: false
# podSecurityContext:
# runAsUser: 1001
# runAsGroup: 1001
# supplementalGroups: [1001]
# serviceAccountName: percona-xtradb-cluster-operator-workload
# imagePullPolicy: Always
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
# resources:
# requests:
# memory: 100M
# cpu: 100m
# limits:
# memory: 200M
# cpu: 200m
# envVarsSecret: my-env-var-secrets
    resources:
      requests:
        memory: 8G
        cpu: 1
# ephemeral-storage: 1G
# limits:
# memory: 1G
# cpu: "1"
# ephemeral-storage: 1G
# nodeSelector:
# disktype: ssd
# topologySpreadConstraints:
# - labelSelector:
# matchLabels:
# app.kubernetes.io/name: percona-xtradb-cluster
# maxSkew: 1
# topologyKey: kubernetes.io/hostname
# whenUnsatisfiable: DoNotSchedule
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
    podDisruptionBudget:
      maxUnavailable: 1
# minAvailable: 0
    volumeSpec:
# emptyDir: {}
# hostPath:
# path: /data
# type: Directory
      persistentVolumeClaim:
        storageClassName: pxc
        accessModes: ["ReadWriteOnce"]
# dataSource:
# name: new-snapshot-test
# kind: VolumeSnapshot
# apiGroup: snapshot.storage.k8s.io
        resources:
          requests:
            storage: 200G
    gracePeriod: 600
# lifecycle:
# preStop:
# exec:
# command: ["/bin/true"]
# postStart:
# exec:
# command: ["/bin/true"]
  haproxy:
    enabled: true
    size: 3
    image: percona/haproxy:2.8.5
# imagePullPolicy: Always
# schedulerName: mycustom-scheduler
# readinessDelaySec: 15
# livenessDelaySec: 600
# configuration: |
#
# the actual default configuration file can be found here https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/main/build/haproxy-global.cfg
#
# global
# maxconn 2048
# external-check
# insecure-fork-wanted
# stats socket /etc/haproxy/pxc/haproxy.sock mode 600 expose-fd listeners level admin
#
# defaults
# default-server init-addr last,libc,none
# log global
# mode tcp
# retries 10
# timeout client 28800s
# timeout connect 100500
# timeout server 28800s
#
# resolvers kubernetes
# parse-resolv-conf
#
# frontend galera-in
# bind *:3309 accept-proxy
# bind *:3306
# mode tcp
# option clitcpka
# default_backend galera-nodes
#
# frontend galera-admin-in
# bind *:33062
# mode tcp
# option clitcpka
# default_backend galera-admin-nodes
#
# frontend galera-replica-in
# bind *:3307
# mode tcp
# option clitcpka
# default_backend galera-replica-nodes
#
# frontend galera-mysqlx-in
# bind *:33060
# mode tcp
# option clitcpka
# default_backend galera-mysqlx-nodes
#
# frontend stats
# bind *:8404
# mode http
# http-request use-service prometheus-exporter if { path /metrics }
# imagePullSecrets:
# - name: private-registry-credentials
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# readinessProbes:
# initialDelaySeconds: 15
# timeoutSeconds: 1
# periodSeconds: 5
# successThreshold: 1
# failureThreshold: 3
# livenessProbes:
# initialDelaySeconds: 60
# timeoutSeconds: 5
# periodSeconds: 30
# successThreshold: 1
# failureThreshold: 4
# exposePrimary:
# enabled: false
# type: ClusterIP
# annotations:
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
# externalTrafficPolicy: Cluster
# internalTrafficPolicy: Cluster
# labels:
# rack: rack-22
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# loadBalancerIP: 127.0.0.1
# exposeReplicas:
# enabled: true
# onlyReaders: false
# type: ClusterIP
# annotations:
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
# externalTrafficPolicy: Cluster
# internalTrafficPolicy: Cluster
# labels:
# rack: rack-22
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# loadBalancerIP: 127.0.0.1
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
# resources:
# requests:
# memory: 100M
# cpu: 100m
# limits:
# memory: 200M
# cpu: 200m
# envVarsSecret: my-env-var-secrets
    resources:
      requests:
        memory: 1G
        cpu: 600m
# limits:
# memory: 1G
# cpu: 700m
# priorityClassName: high-priority
# nodeSelector:
# disktype: ssd
# sidecarResources:
# requests:
# memory: 1G
# cpu: 500m
# limits:
# memory: 2G
# cpu: 600m
# containerSecurityContext:
# privileged: false
# podSecurityContext:
# runAsUser: 1001
# runAsGroup: 1001
# supplementalGroups: [1001]
# serviceAccountName: percona-xtradb-cluster-operator-workload
# topologySpreadConstraints:
# - labelSelector:
# matchLabels:
# app.kubernetes.io/name: percona-xtradb-cluster
# maxSkew: 1
# topologyKey: kubernetes.io/hostname
# whenUnsatisfiable: DoNotSchedule
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
    podDisruptionBudget:
      maxUnavailable: 1
# minAvailable: 0
    gracePeriod: 30
# lifecycle:
# preStop:
# exec:
# command: ["/bin/true"]
# postStart:
# exec:
# command: ["/bin/true"]
  proxysql:
    enabled: false
    size: 3
    image: percona/proxysql2:2.5.5
# imagePullPolicy: Always
# configuration: |
# datadir="/var/lib/proxysql"
#
# admin_variables =
# {
# admin_credentials="proxyadmin:admin_password"
# mysql_ifaces="0.0.0.0:6032"
# refresh_interval=2000
#
# cluster_username="proxyadmin"
# cluster_password="admin_password"
# checksum_admin_variables=false
# checksum_ldap_variables=false
# checksum_mysql_variables=false
# cluster_check_interval_ms=200
# cluster_check_status_frequency=100
# cluster_mysql_query_rules_save_to_disk=true
# cluster_mysql_servers_save_to_disk=true
# cluster_mysql_users_save_to_disk=true
# cluster_proxysql_servers_save_to_disk=true
# cluster_mysql_query_rules_diffs_before_sync=1
# cluster_mysql_servers_diffs_before_sync=1
# cluster_mysql_users_diffs_before_sync=1
# cluster_proxysql_servers_diffs_before_sync=1
# }
#
# mysql_variables=
# {
# monitor_password="monitor"
# monitor_galera_healthcheck_interval=1000
# threads=2
# max_connections=2048
# default_query_delay=0
# default_query_timeout=10000
# poll_timeout=2000
# interfaces="0.0.0.0:3306"
# default_schema="information_schema"
# stacksize=1048576
# connect_timeout_server=10000
# monitor_history=60000
# monitor_connect_interval=20000
# monitor_ping_interval=10000
# ping_timeout_server=200
# commands_stats=true
# sessions_sort=true
# have_ssl=true
# ssl_p2s_ca="/etc/proxysql/ssl-internal/ca.crt"
# ssl_p2s_cert="/etc/proxysql/ssl-internal/tls.crt"
# ssl_p2s_key="/etc/proxysql/ssl-internal/tls.key"
# ssl_p2s_cipher="ECDHE-RSA-AES128-GCM-SHA256"
# }
# readinessDelaySec: 15
# livenessDelaySec: 600
# schedulerName: mycustom-scheduler
# imagePullSecrets:
# - name: private-registry-credentials
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# expose:
# enabled: false
# type: ClusterIP
# annotations:
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
# externalTrafficPolicy: Cluster
# internalTrafficPolicy: Cluster
# labels:
# rack: rack-22
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# loadBalancerIP: 127.0.0.1
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
# resources:
# requests:
# memory: 100M
# cpu: 100m
# limits:
# memory: 200M
# cpu: 200m
# envVarsSecret: my-env-var-secrets
    resources:
      requests:
        memory: 1G
        cpu: 600m
# limits:
# memory: 1G
# cpu: 700m
# priorityClassName: high-priority
# nodeSelector:
# disktype: ssd
# sidecarResources:
# requests:
# memory: 1G
# cpu: 500m
# limits:
# memory: 2G
# cpu: 600m
# containerSecurityContext:
# privileged: false
# podSecurityContext:
# runAsUser: 1001
# runAsGroup: 1001
# supplementalGroups: [1001]
# serviceAccountName: percona-xtradb-cluster-operator-workload
# topologySpreadConstraints:
# - labelSelector:
# matchLabels:
# app.kubernetes.io/name: percona-xtradb-cluster-operator
# maxSkew: 1
# topologyKey: kubernetes.io/hostname
# whenUnsatisfiable: DoNotSchedule
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
    volumeSpec:
# emptyDir: {}
# hostPath:
# path: /data
# type: Directory
      persistentVolumeClaim:
# storageClassName: standard
# accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 2G
    podDisruptionBudget:
      maxUnavailable: 1
# minAvailable: 0
    gracePeriod: 30
# lifecycle:
# preStop:
# exec:
# command: ["/bin/true"]
# postStart:
# exec:
# command: ["/bin/true"]
# loadBalancerSourceRanges:
# - 10.0.0.0/8
  logcollector:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.15.0-logcollector-fluentbit3.1.4
# configuration: |
# [OUTPUT]
# Name es
# Match *
# Host 192.168.2.3
# Port 9200
# Index my_index
# Type my_type
    resources:
      requests:
        memory: 100M
        cpu: 200m
  pmm:
    enabled: false
    image: percona/pmm-client:2.42.0
    serverHost: monitoring-service
# serverUser: admin
# pxcParams: "--disable-tablestats-limit=2000"
# proxysqlParams: "--custom-labels=CUSTOM-LABELS"
# containerSecurityContext:
# privileged: false
    resources:
      requests:
        memory: 150M
        cpu: 300m
  backup:
    allowParallel: true 
    image: percona/percona-xtradb-cluster-operator:1.15.0-pxc8.0-backup-pxb8.0.35
    backoffLimit: 3
# serviceAccountName: percona-xtradb-cluster-operator
# imagePullSecrets:
# - name: private-registry-credentials
    pitr:
      enabled: false
      storageName: fs-pvc
      timeBetweenUploads: 60
      timeoutSeconds: 60
# resources:
# requests:
# memory: 0.1G
# cpu: 100m
# limits:
# memory: 1G
# cpu: 700m
    storages:
      s3-us-west:
        type: s3
        verifyTLS: true
# nodeSelector:
# storage: tape
# backupWorker: 'True'
# resources:
# requests:
# memory: 1G
# cpu: 600m
# topologySpreadConstraints:
# - labelSelector:
# matchLabels:
# app.kubernetes.io/name: percona-xtradb-cluster
# maxSkew: 1
# topologyKey: kubernetes.io/hostname
# whenUnsatisfiable: DoNotSchedule
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: backupWorker
# operator: In
# values:
# - 'True'
# tolerations:
# - key: "backupWorker"
# operator: "Equal"
# value: "True"
# effect: "NoSchedule"
# annotations:
# testName: scheduled-backup
# labels:
# backupWorker: 'True'
# schedulerName: 'default-scheduler'
# priorityClassName: 'high-priority'
# containerSecurityContext:
# privileged: true
# podSecurityContext:
# fsGroup: 1001
# supplementalGroups: [1001, 1002, 1003]
# containerOptions:
# env:
# - name: VERIFY_TLS
# value: "false"
# args:
# xtrabackup:
# - "--someflag=abc"
# xbcloud:
# - "--someflag=abc"
# xbstream:
# - "--someflag=abc"
        s3:
          bucket: S3-BACKUP-BUCKET-NAME-HERE
          credentialsSecret: my-cluster-name-backup-s3
          region: us-west-2
      azure-blob:
        type: azure
        azure:
          credentialsSecret: azure-secret
          container: test
# endpointUrl: https://accountName.blob.core.windows.net
# storageClass: Hot
      fs-pvc:
        type: filesystem
# nodeSelector:
# storage: tape
# backupWorker: 'True'
# resources:
# requests:
# memory: 1G
# cpu: 600m
# topologySpreadConstraints:
# - labelSelector:
# matchLabels:
# app.kubernetes.io/name: percona-xtradb-cluster
# maxSkew: 1
# topologyKey: kubernetes.io/hostname
# whenUnsatisfiable: DoNotSchedule
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: backupWorker
# operator: In
# values:
# - 'True'
# tolerations:
# - key: "backupWorker"
# operator: "Equal"
# value: "True"
# effect: "NoSchedule"
# annotations:
# testName: scheduled-backup
# labels:
# backupWorker: 'True'
# schedulerName: 'default-scheduler'
# priorityClassName: 'high-priority'
# containerSecurityContext:
# privileged: true
# podSecurityContext:
# fsGroup: 1001
# supplementalGroups: [1001, 1002, 1003]
        volume:
          persistentVolumeClaim:
            storageClassName: pxc-backup
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 200G
    schedule:
# - name: "sat-night-backup"
# schedule: "0 0 * * 6"
# keep: 3
# storageName: s3-us-west
      - name: "daily-backup"
        schedule: "0 0 * * *"
        keep: 5
        storageName: fs-pvc

```

I fix in cr.yaml add “Gi” in storage size.

```auto
      persistentVolumeClaim:
# storageClassName: standard
# accessModes: ["ReadWriteOnce"]
# dataSource:
# name: new-snapshot-test
# kind: VolumeSnapshot
# apiGroup: snapshot.storage.k8s.io
        resources:
          requests:
-- storage: 200G
++ storage: 200Gi

```

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [September 15, 2024, 5:42pm UTC](https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084/4 "2024-09-15T17:42:26Z")

</div>

@Pratak_Eak, As I understand, the cluster is okay, but you still have problems with backups. Can you send the log from the backup pod? I need to know your k8s version and the platform that you are using.

---

<div class="post-metadata">

**Author:** ![Frederic\_Coelho1](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/frederic_coelho1/32/12460_2.png) [@Frederic\_Coelho1](https://forums.percona.com/u/Frederic_Coelho1)\
**Post date:** [September 27, 2024, 8:07am UTC](https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084/5 "2024-09-27T08:07:21Z")

</div>

Hello, same issue for me  
`FATAL: /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:group_check_proto_ver():341: Group requested gcs_proto_ver: 4, max supported by this node: 2.Upgrade the node before joining this group.Need to abort.`  
operator :` percona/percona-xtradb-cluster-operator:1.15.0`  
pxc : `percona/percona-xtradb-cluster:8.0`  
backup : `percona/percona-xtradb-cluster-operator:1.15.0-pxc8.0-backup-pxb8.0.35`

of course no issue with 1.14

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [September 30, 2024, 6:48pm UTC](https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084/6 "2024-09-30T18:48:40Z")

</div>

@Frederic_Coelho1 the issue is different. More detailed information can be found [backup mysql 8.0 fail on 1.15 · Issue #1832 · percona/percona-xtradb-cluster-operator · GitHub](https://github.com/percona/percona-xtradb-cluster-operator/issues/1832#issuecomment-2383682732)