Description:
I am fail to run backup percona extradb operator manual and cron on private kubenetes microk8s. How do I fix this error?
pxc-backup-minio.yaml
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterBackup
metadata:
finalizers:
- percona.com/delete-backup
name: backup-minio
spec:
pxcCluster: cluster
storageName: minio
Steps to Reproduce:
run this command to start backup manual
k -n pxc apply -f pxc-backup-minio.yaml
check backup result
sh> k -n pxc get pxc-backup
NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE
cron-cluster-minio-proen-20249120048-372f8 cluster minio-proen s3://backup/cluster-2024-09-12-00:00:48-full Failed 3h47m
backup-minio cluster minio-proen s3://backup/cluster-2024-09-12-03:34:00-full Failed 14m
Version:
pxc image
perconalab/percona-xtradb-cluster-operator:main-pxc8.0
backup image
perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
Logs:
FATAL: /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:group_check_proto_ver():341: Group requested gcs_proto_ver: 4, max supported by this node: 2.Upgrade the node before joining this group.Need to abort.
Defaulted container “xtrabackup” out of: xtrabackup, backup-init (init)
- LIB_PATH=/usr/lib/pxc
- . /usr/lib/pxc/backup.sh
++ set -o errexit
++ SST_INFO_NAME=sst_info
++ XBCLOUD_ARGS='–curl-retriable-errors=7 ’
++ INSECURE_ARG=
++ ‘[’ -n true ‘]’
++ [[ true == \f\a\l\s\e ]]
++ S3_BUCKET_PATH=cluster-2024-09-12-03:34:00-full
+++ date +%F-%H-%M
++ BACKUP_PATH=cluster-pxc-2024-09-12-03-36-xtrabackup.stream- GARBD_OPTS=
- check_ssl
- CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- ‘[’ -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ‘]’
- SSL_DIR=/etc/mysql/ssl
- ‘[’ -f /etc/mysql/ssl/ca.crt ‘]’
- CA=/etc/mysql/ssl/ca.crt
- SSL_INTERNAL_DIR=/etc/mysql/ssl-internal
- ‘[’ -f /etc/mysql/ssl-internal/ca.crt ‘]’
- CA=/etc/mysql/ssl-internal/ca.crt
- KEY=/etc/mysql/ssl/tls.key
- CERT=/etc/mysql/ssl/tls.crt
- ‘[’ -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ‘]’
- KEY=/etc/mysql/ssl-internal/tls.key
- CERT=/etc/mysql/ssl-internal/tls.crt
- ‘[’ -f /etc/mysql/ssl-internal/ca.crt -a -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ‘]’
- GARBD_OPTS=‘socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;’
- ‘[’ -n backup ‘]’
- clean_backup_s3
- mc_add_bucket_dest
- echo '+ mc -C /tmp/mc config host add dest https://s3.aspiredigitalgroup.com.au ACCESS_KEY_ID SECRET_ACCESS_KEY ’
- mc -C /tmp/mc config host add dest https://s3.aspiredigitalgroup.com.au ACCESS_KEY_ID SECRET_ACCESS_KEY
Addeddest
successfully.- is_object_exist backup cluster-2024-09-12-03:34:00-full.sst_info
- local bucket=backup
- local object=cluster-2024-09-12-03:34:00-full.sst_info
++ mc -C /tmp/mc --json ls dest/backup/cluster-2024-09-12-03:34:00-full.sst_info
++ jq .status- [[ -n ‘’ ]]
- is_object_exist backup cluster-2024-09-12-03:34:00-full/
- local bucket=backup
- local object=cluster-2024-09-12-03:34:00-full/
++ mc -C /tmp/mc --json ls dest/backup/cluster-2024-09-12-03:34:00-full/
++ jq .status- [[ -n ‘’ ]]
- request_streaming
++ hostname -i
++ sed -E ‘s/.\b([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})\b./\1/’- local LOCAL_IP=10.1.167.63
++ get_backup_source
+++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=cluster-pxc
+++ grep wsrep_cluster_size
+++ sort
+++ tail -1
+++ cut -d : -f 12
++ CLUSTER_SIZE=3
++ ‘[’ -z 3 ‘]’
+++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=cluster-pxc
+++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
+++ sort -r
+++ tail -1
+++ cut -d : -f 2
+++ cut -d . -f 1
++ FIRST_NODE=cluster-pxc-0
++ SKIP_FIRST_POD=‘|’
++ (( 3 > 1 ))
++ SKIP_FIRST_POD=cluster-pxc-0
++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=cluster-pxc
++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
++ grep -v cluster-pxc-0
++ sort
++ tail -1
++ cut -d : -f 2
++ cut -d . -f 1- local NODE_NAME=cluster-pxc-2
- ‘[’ -z cluster-pxc-2 ‘]’
- set +o errexit
- log INFO ‘Garbd was started’
2024-09-12 03:36:11 [INFO] Garbd was started- garbd --address ‘gcomm://cluster-pxc-2.cluster-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567’ --donor cluster-pxc-2 --group cluster-pxc --options ‘socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;’ --sst xtrabackup-v2:10.1.167.63:4444/xtrabackup_sst//1 --recv-script=/usr/bin/run_backup.sh
2024-09-12 03:36:11.590 INFO: CRC-32C: using 64-bit x86 acceleration.
2024-09-12 03:36:11.590 INFO: Read config:
daemon: 0
name: garb
address: gcomm://cluster-pxc-2.cluster-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567
group: cluster-pxc
sst: xtrabackup-v2:10.1.167.63:4444/xtrabackup_sst//1
donor: cluster-pxc-2
options: socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;; gcs.fc_limit=9999999; gcs.fc_factor=1.0; gcs.fc_single_primary=yes; socket.ssl=YES
cfg:
log:
recv_script: /usr/bin/run_backup.sh
workdir:2024-09-12 03:36:11.591 WARN: SSL compression is not effective. The option socket.ssl_compression is deprecated and will be removed in future releases.
2024-09-12 03:36:11.591 WARN: Parameter ‘socket.ssl_compression’ is deprecated and will be removed in future versions
2024-09-12 03:36:11.598 INFO: protonet asio version 0
2024-09-12 03:36:11.600 INFO: Using CRC-32C for message checksums.
2024-09-12 03:36:11.600 INFO: backend: asio
2024-09-12 03:36:11.600 INFO: gcomm thread scheduling priority set to other:0
2024-09-12 03:36:11.600 INFO: Fail to access the file (./gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2024-09-12 03:36:11.600 INFO: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2024-09-12 03:36:11.600 INFO: GMCast version 0
2024-09-12 03:36:11.601 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567
2024-09-12 03:36:11.601 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1
2024-09-12 03:36:11.602 INFO: EVS version 1
2024-09-12 03:36:11.602 INFO: gcomm: connecting to group ‘cluster-pxc’, peer ‘cluster-pxc-2.cluster-pxc:’
2024-09-12 03:36:11.611 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) connection established to af8cc657-b8d9 ssl://10.1.109.118:4567
2024-09-12 03:36:11.611 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: ssl://10.1.106.165:4567 ssl://10.1.9.71:4567
2024-09-12 03:36:11.718 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) connection established to cee986a2-a54c ssl://10.1.9.71:4567
2024-09-12 03:36:11.730 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) connection established to ea92447e-b4de ssl://10.1.106.165:4567
2024-09-12 03:36:11.730 INFO: (27cfeea8-badb, ‘ssl://0.0.0.0:4567’) connection established to ea92447e-b4de ssl://10.1.106.165:4567
2024-09-12 03:36:12.104 INFO: EVS version upgrade 0 → 1
2024-09-12 03:36:12.104 INFO: declaring af8cc657-b8d9 at ssl://10.1.109.118:4567 stable
2024-09-12 03:36:12.104 INFO: declaring cee986a2-a54c at ssl://10.1.9.71:4567 stable
2024-09-12 03:36:12.104 INFO: declaring ea92447e-b4de at ssl://10.1.106.165:4567 stable
2024-09-12 03:36:12.105 INFO: PC protocol upgrade 0 → 1
2024-09-12 03:36:12.105 INFO: Node af8cc657-b8d9 state primary
2024-09-12 03:36:12.106 INFO: Current view of cluster as seen by this node
view (view_id(PRIM,27cfeea8-badb,88)
memb {
27cfeea8-badb,0
af8cc657-b8d9,0
cee986a2-a54c,0
ea92447e-b4de,0
}
joined {
}
left {
}
partitioned {
}
)
2024-09-12 03:36:12.106 INFO: Save the discovered primary-component to disk
2024-09-12 03:36:12.106 WARN: open file(./gvwstate.dat.tmp) failed(Permission denied)
2024-09-12 03:36:12.602 INFO: gcomm: connected
2024-09-12 03:36:12.602 INFO: Changing maximum packet size to 64500, resulting msg size: 32636
2024-09-12 03:36:12.602 INFO: Shifting CLOSED → OPEN (TO: 0)
2024-09-12 03:36:12.602 INFO: Opened channel ‘cluster-pxc’
2024-09-12 03:36:12.603 INFO: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 4
2024-09-12 03:36:12.603 INFO: STATE_EXCHANGE: sent state UUID: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8
2024-09-12 03:36:12.604 INFO: STATE EXCHANGE: sent state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8
2024-09-12 03:36:12.605 INFO: STATE EXCHANGE: got state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8 from 0 (garb)
2024-09-12 03:36:12.605 INFO: STATE EXCHANGE: got state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8 from 1 (cluster-pxc-2)
2024-09-12 03:36:12.605 INFO: STATE EXCHANGE: got state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8 from 2 (cluster-pxc-1)
2024-09-12 03:36:12.605 INFO: STATE EXCHANGE: got state msg: 2868e45b-70b8-11ef-b05f-8bbdc0af14f8 from 3 (cluster-pxc-0)
2024-09-12 03:36:12.605 INFO: Quorum results:
version = 6,
component = PRIMARY,
conf_id = 87,
members = 3/4 (primary/total),
act_id = 2345,
last_appl. = 2325,
protocols = 4/11/4 (gcs/repl/appl),
vote policy= 0,
group UUID = f92fb1be-6ffe-11ef-9a87-0ac4a615d14f
2024-09-12 03:36:12.605 FATAL: /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:group_check_proto_ver():341: Group requested gcs_proto_ver: 4, max supported by this node: 2.Upgrade the node before joining this group.Need to abort.
2024-09-12 03:36:12.605 INFO: garbd: Terminated.
/usr/bin/backup.sh: line 68: 368 Aborted garbd --address “gcomm://$NODE_NAME.$PXC_SERVICE?gmcast.listen_addr=tcp://0.0.0.0:4567” --donor “$NODE_NAME” --group “$PXC_SERVICE” --options “$GARBD_OPTS” --sst “xtrabackup-v2:$LOCAL_IP:4444/xtrabackup_sst//1” --recv-script=“/usr/bin/run_backup.sh”
- EXID_CODE=134
- ‘[’ -f /tmp/backup-is-completed ‘]’
- log ERROR ‘Backup was finished unsuccessfull’
2024-09-12 03:36:12 [ERROR] Backup was finished unsuccessfull- exit 134
Expected Result:
NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE
cron-cluster-minio-proen-20249120048-372f8 cluster minio-proen s3://backup/cluster-2024-09-12-00:00:48-full Completed 3h47m
backup-minio cluster minio-proen s3://backup/cluster-2024-09-12-03:34:00-full Completed 14m
Actual Result:
NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE
cron-cluster-minio-proen-20249120048-372f8 cluster minio-proen s3://backup/cluster-2024-09-12-00:00:48-full Failed 3h47m
backup-minio cluster minio-proen s3://backup/cluster-2024-09-12-03:34:00-full Failed 14m