Hello
- I get error in time backup cron and manual backup
- I use pxc-db and percona-operator in kubernetes
My helm release config 1.16.1 pxc-db 1.16.1 · percona/percona (1.14 work)
unsafeFlags:
proxySize: true
tls: true # eсли false будет ошибка
tls:
enabled: false
pause: false
pxc:
affinity:
antiAffinityTopologyKey: "none" # все на 1 машине для kind кластера
#disableTLS: true deprecated
persistence:
enabled: true
# storageClass: gp3-fast
size: 3Gi
configuration: |
[mysqld]
expire_logs_days = 15
pxc-encrypt-cluster-traffic = OFF
# https://forums.percona.com/t/pmm-slow-query-log-location/24852/8
slow_query_log=ON
slow_query_log_file=/var/lib/mysql/slow
long_query_time=10
log_slow_rate_limit=100
log_slow_rate_type=query
log_slow_verbosity=full
log_slow_admin_statements=ON
max_slowlog_size=500M
max_slowlog_files=10
# critical slow sql query
slow_query_log_always_write_time=120
resources:
requests:
memory: 1Gi
cpu: 500m
limits:
memory: 2Gi
cpu: 999m
haproxy:
size: 2
affinity:
antiAffinityTopologyKey: "none" # все на 1 машине для kind кластера
# antiAffinityTopologyKey: kubernetes.io/hostname
# resources:
# requests:
# memory: 800Mi
# cpu: 150m
# limits:
# memory: 1Gi
# cpu: 500m
pmm:
enabled: true
serverHost: monitoring-service.monitoring.svc.cluster.local
serverUser: admin
pxcParams: "--query-source=slowlog --size-slow-logs=1GiB --disable-tablestats-limit=2000 "
# resources:
# requests:
# memory: 150Mi
# cpu: 150m
# limits:
# memory: 800Mi
# cpu: 500m
secrets:
passwords:
root: password123
replication: password123
xtrabackup: password123
monitor: password123
clustercheck: password123
proxyadmin: password123
operator: password123
# need in pmm server monitoring
pmmserver: password123
pmm:
enabled: true
serverHost: monitoring-service.percona.svc.cluster.local
serverUser: admin
backup:
storages:
fs-pvc:
type: filesystem
verifyTLS: false
volume:
persistentVolumeClaim:
storageClassName: standard
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 3Gi
containerOptions:
env:
- name: VERIFY_TLS
value: "false"
args:
xtrabackup:
- "--no-server-version-check" # fix If the source system version is greater than the XtraBackup version, XtraBackup stops the backup and returns an error message.
error
+ LIB_PATH=/usr/lib/pxc
+ . /usr/lib/pxc/backup.sh
++ set -o errexit
++ SST_INFO_NAME=sst_info
++ XBCLOUD_ARGS='--curl-retriable-errors=7 '
++ INSECURE_ARG=
++ '[' -n false ']'
++ [[ false == \f\a\l\s\e ]]
++ INSECURE_ARG=--insecure
++ XBCLOUD_ARGS='--insecure --curl-retriable-errors=7 '
+++ date +%F-%H-%M
++ S3_BUCKET_PATH=percona-cluster-pxc-d-pxc-2025-02-03-15-22-xtrabackup.stream
+++ date +%F-%H-%M
++ BACKUP_PATH=percona-cluster-pxc-d-pxc-2025-02-03-15-22-xtrabackup.stream
+ GARBD_OPTS=
+ check_ssl
+ CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+ '[' -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ']'
+ SSL_DIR=/etc/mysql/ssl
+ '[' -f /etc/mysql/ssl/ca.crt ']'
+ CA=/etc/mysql/ssl/ca.crt
+ SSL_INTERNAL_DIR=/etc/mysql/ssl-internal
+ '[' -f /etc/mysql/ssl-internal/ca.crt ']'
+ CA=/etc/mysql/ssl-internal/ca.crt
+ KEY=/etc/mysql/ssl/tls.key
+ CERT=/etc/mysql/ssl/tls.crt
+ '[' -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ']'
+ KEY=/etc/mysql/ssl-internal/tls.key
+ CERT=/etc/mysql/ssl-internal/tls.crt
+ '[' -f /etc/mysql/ssl-internal/ca.crt -a -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ']'
+ GARBD_OPTS='socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;'
+ '[' -n '' ']'
+ request_streaming
++ hostname -i
++ sed -E 's/.*\b([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\b.*/\1/'
+ local LOCAL_IP=10.244.0.83
++ get_backup_source
+++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=percona-cluster-pxc-d-pxc
+++ grep wsrep_cluster_size
+++ sort
+++ tail -1
+++ cut -d : -f 12
++ CLUSTER_SIZE=3
++ '[' -z 3 ']'
+++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=percona-cluster-pxc-d-pxc
+++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
+++ sort -r
+++ tail -1
+++ cut -d . -f 1
+++ cut -d : -f 2
++ FIRST_NODE=percona-cluster-pxc-d-pxc-0
++ SKIP_FIRST_POD='|'
++ (( 3 > 1 ))
++ SKIP_FIRST_POD=percona-cluster-pxc-d-pxc-0
++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=percona-cluster-pxc-d-pxc
++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
++ grep -v percona-cluster-pxc-d-pxc-0
++ sort
++ tail -1
++ cut -d : -f 2
++ cut -d . -f 1
+ local NODE_NAME=percona-cluster-pxc-d-pxc-2
+ '[' -z percona-cluster-pxc-d-pxc-2 ']'
+ set +o errexit
+ log INFO 'Garbd was started'
2025-02-03 15:22:19 [INFO] Garbd was started
+ garbd --address 'gcomm://percona-cluster-pxc-d-pxc-2.percona-cluster-pxc-d-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567' --donor percona-cluster-pxc-d-pxc-2 --group percona-cluster-pxc-d-pxc --options 'socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;' --sst xtrabackup-v2:10.244.0.83:4444/xtrabackup_sst//1 --recv-script=/usr/bin/run_backup.sh
2025-02-03 15:22:19.534 INFO: CRC-32C: using 64-bit x86 acceleration.
2025-02-03 15:22:19.534 INFO: Read config:
daemon: 0
name: garb
address: gcomm://percona-cluster-pxc-d-pxc-2.percona-cluster-pxc-d-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567
group: percona-cluster-pxc-d-pxc
sst: xtrabackup-v2:10.244.0.83:4444/xtrabackup_sst//1
donor: percona-cluster-pxc-d-pxc-2
options: socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;; gcs.fc_limit=9999999; gcs.fc_factor=1.0; gcs.fc_single_primary=yes; socket.ssl=YES
cfg:
log:
recv_script: /usr/bin/run_backup.sh
workdir:
2025-02-03 15:22:19.534 INFO: not using SSL compression
2025-02-03 15:22:19.538 INFO: protonet asio version 0
2025-02-03 15:22:19.539 INFO: Using CRC-32C for message checksums.
2025-02-03 15:22:19.539 INFO: backend: asio
2025-02-03 15:22:19.539 INFO: gcomm thread scheduling priority set to other:0
2025-02-03 15:22:19.541 INFO: Fail to access the file (./gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2025-02-03 15:22:19.541 INFO: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2025-02-03 15:22:19.541 INFO: GMCast version 0
2025-02-03 15:22:19.542 INFO: (a88eaf30-8d00, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2025-02-03 15:22:19.542 INFO: (a88eaf30-8d00, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2025-02-03 15:22:19.543 INFO: EVS version 1
2025-02-03 15:22:19.543 INFO: gcomm: connecting to group 'percona-cluster-pxc-d-pxc', peer 'percona-cluster-pxc-d-pxc-2.percona-cluster-pxc-d-pxc:'
2025-02-03 15:22:19.545 INFO: Failed to establish connection: wrong version number
2025-02-03 15:22:20.543 INFO: EVS version upgrade 0 -> 1
2025-02-03 15:22:20.544 INFO: PC protocol upgrade 0 -> 1
2025-02-03 15:22:20.544 INFO: No nodes coming from primary view, primary view is not possible
2025-02-03 15:22:20.544 INFO: Current view of cluster as seen by this node
view (view_id(NON_PRIM,a88eaf30-8d00,1)
memb {
a88eaf30-8d00,0
}
joined {
}
left {
}
partitioned {
}
)
2025-02-03 15:22:21.044 WARN: last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT1.50083S), skipping check
2025-02-03 15:22:21.045 INFO: Failed to establish connection: wrong version number
2025-02-03 15:22:22.544 INFO: Failed to establish connection: wrong version number
2025-02-03 15:22:22.544 INFO: announce period timed out (pc.announce_timeout)
2025-02-03 15:22:24.045 INFO: Failed to establish connection: wrong version number
...
2025-02-03 15:22:51.048 INFO: Failed to establish connection: wrong version number
2025-02-03 15:22:52.548 INFO: Failed to establish connection: wrong version number
2025-02-03 15:22:52.559 INFO: Current view of cluster as seen by this node
view (view_id(NON_PRIM,a88eaf30-8d00,1)
memb {
a88eaf30-8d00,0
}
joined {
}
left {
}
partitioned {
}
)
2025-02-03 15:22:52.560 INFO: PC protocol downgrade 1 -> 0
2025-02-03 15:22:52.560 INFO: Current view of cluster as seen by this node
view ((empty))
2025-02-03 15:22:52.564 ERROR: failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout)
at ../../../../percona-xtradb-cluster-galera/gcomm/src/pc.cpp:connect():176
2025-02-03 15:22:52.565 ERROR: ../../../../percona-xtradb-cluster-galera/gcs/src/gcs_core.cpp:gcs_core_open():222: Failed to open backend connection: -110 (Connection timed out)
2025-02-03 15:22:53.566 INFO: gcomm: terminating thread
2025-02-03 15:22:53.566 INFO: gcomm: joining thread
2025-02-03 15:22:53.567 ERROR: ../../../../percona-xtradb-cluster-galera/gcs/src/gcs.cpp:gcs_open():1934: Failed to open channel 'percona-cluster-pxc-d-pxc' at 'gcomm://percona-cluster-pxc-d-pxc-2.percona-cluster-pxc-d-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567': -110 (Connection timed out)
2025-02-03 15:22:53.568 INFO: Shifting CLOSED -> DESTROYED (TO: 0)
2025-02-03 15:22:53.568 FATAL: Garbd exiting with error: Failed to open connection to group
at ../../../percona-xtradb-cluster-galera/garb/garb_gcs.cpp:Gcs():35
+ EXID_CODE=1
+ '[' -f /tmp/backup-is-completed ']'
+ log ERROR 'Backup was finished unsuccessful'
2025-02-03 15:22:53 [ERROR] Backup was finished unsuccessful
+ exit 1
What is try:
- Use flag --no-server-version-check
- Enable and disable TLS