Percona XtraDB MySQL Backup is failing due to SSL certificate expired | Please let me know the process to renew the SSL certs

MySQL & MariaDB Percona Operator for MySQL
1

We have installed the MySQL cluster using the Percona-xtradb-cluster-operator in redhat Openshift cluster. I have configured to backup the database to S3 storage and backup is working fine but the backup is failing due to certificate validation error.
I have verified the secret (my-cluster-ssl-internal) which is having the certificates and the ssl certs are expired on 29th March 2024. The backup is failing post this.
Can you please let me know how to renew the SSL certs which are generated by the percona operator initially?

Environment Details:

RedHat Openshift Version: 4.8.25
percona-xtradb-cluster-operator version: 1.11.0
MySQL DB version: mysql Ver 14.14 Distrib 5.7.35-38

Backup logs:

  • LIB_PATH=/usr/lib/pxc
  • . /usr/lib/pxc/vault.sh
    ++ set -o errexit
    ++ keyring_vault=/etc/mysql/vault-keyring-secret/keyring_vault.conf
  • GARBD_OPTS=
  • SOCAT_OPTS=TCP-LISTEN:4444,reuseaddr,retry=30
  • SST_INFO_NAME=sst_info
  • INSECURE_ARG=
  • ‘[’ -n false ‘]’
  • [[ false == \f\a\l\s\e ]]
  • INSECURE_ARG=–insecure
  • check_ssl
  • CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  • ‘[’ -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ‘]’
  • CA=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
  • SSL_DIR=/etc/mysql/ssl
  • ‘[’ -f /etc/mysql/ssl/ca.crt ‘]’
  • CA=/etc/mysql/ssl/ca.crt
  • SSL_INTERNAL_DIR=/etc/mysql/ssl-internal
  • ‘[’ -f /etc/mysql/ssl-internal/ca.crt ‘]’
  • CA=/etc/mysql/ssl-internal/ca.crt
  • KEY=/etc/mysql/ssl/tls.key
  • CERT=/etc/mysql/ssl/tls.crt
  • ‘[’ -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ‘]’
  • KEY=/etc/mysql/ssl-internal/tls.key
  • CERT=/etc/mysql/ssl-internal/tls.crt
  • ‘[’ -f /etc/mysql/ssl-internal/ca.crt -a -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ‘]’
  • GARBD_OPTS=‘socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;’
  • SOCAT_OPTS=openssl-listen:4444,reuseaddr,cert=/etc/mysql/ssl-internal/tls.crt,key=/etc/mysql/ssl-internal/tls.key,cafile=/etc/mysql/ssl-internal/ca.crt,verify=1,retry=30
  • ‘[’ -n mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072 ‘]’
  • backup_s3
  • S3_BUCKET_PATH=mysql-staging-cluster
  • echo ‘[INFO] Backup to s3://mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072/mysql-staging-cluster-2024-03-31-13:30:00-full started’
    [INFO] Backup to s3://mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072/mysql-staging-cluster-2024-03-31-13:30:00-full started
  • mc -C /tmp/mc --insecure config host add dest https://s3.openshift-storage.svc:443 ACCESS_KEY_ID SECRET_ACCESS_KEY
    Added dest successfully.
  • is_object_exist mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072 mysql-staging-cluster-2024-03-31-13:30:00-full.sst_info
  • local bucket=mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072
  • local object=mysql-staging-cluster-2024-03-31-13:30:00-full.sst_info
    ++ jq .status
    ++ mc -C /tmp/mc --insecure --json ls dest/mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072/mysql-staging-cluster-2024-03-31-13:30:00-full.sst_info
  • [[ -n ‘’ ]]
  • is_object_exist mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072 mysql-staging-cluster-2024-03-31-13:30:00-full
  • local bucket=mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072
  • local object=mysql-staging-cluster-2024-03-31-13:30:00-full
    ++ mc -C /tmp/mc --insecure --json ls dest/mysql-staging-backup-5e1df473-db91-42a6-93c5-1a195f11c072/mysql-staging-cluster-2024-03-31-13:30:00-full
    ++ jq .status
  • [[ -n ‘’ ]]
  • request_streaming
    ++ hostname -i
  • local LOCAL_IP=100.64.28.230
    ++ get_backup_source
    +++ peer-list -on-start=/usr/bin/get-pxc-state -service=mysql-staging-cluster-pxc
    +++ grep wsrep_cluster_size
    +++ sort
    +++ tail -1
    +++ cut -d : -f 12
    ++ CLUSTER_SIZE=3
    +++ peer-list -on-start=/usr/bin/get-pxc-state -service=mysql-staging-cluster-pxc
    +++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
    +++ sort -r
    +++ tail -1
    +++ cut -d : -f 2
    +++ cut -d . -f 1
    ++ FIRST_NODE=mysql-staging-cluster-pxc-0
    ++ SKIP_FIRST_POD=‘|’
    ++ (( 3 > 1 ))
    ++ SKIP_FIRST_POD=mysql-staging-cluster-pxc-0
    ++ peer-list -on-start=/usr/bin/get-pxc-state -service=mysql-staging-cluster-pxc
    ++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
    ++ grep -v mysql-staging-cluster-pxc-0
    ++ sort
    ++ tail -1
    ++ cut -d : -f 2
    ++ cut -d . -f 1
  • local NODE_NAME=mysql-staging-cluster-pxc-2
  • ‘[’ -z mysql-staging-cluster-pxc-2 ‘]’
  • timeout -k 25 20 garbd --address ‘gcomm://mysql-staging-cluster-pxc-2.mysql-staging-cluster-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567’ --donor mysql-staging-cluster-pxc-2 --group mysql-staging-cluster-pxc --options ‘socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;’ --sst xtrabackup-v2:100.64.28.230:4444/xtrabackup_sst//1
  • tee /tmp/garbd.log
    2024-03-31 13:54:12.917 INFO: CRC-32C: using 64-bit x86 acceleration.
    2024-03-31 13:54:12.917 INFO: Read config:
    daemon: 0
    name: garb
    address: gcomm://mysql-staging-cluster-pxc-2.mysql-staging-cluster-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567
    group: mysql-staging-cluster-pxc
    sst: xtrabackup-v2:100.64.28.230:4444/xtrabackup_sst//1
    donor: mysql-staging-cluster-pxc-2
    options: socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;; gcs.fc_limit=9999999; gcs.fc_factor=1.0; gcs.fc_master_slave=yes
    cfg:
    log:

2024-03-31 13:54:12.919 INFO: Using CRC-32C for message checksums.
2024-03-31 13:54:12.919 INFO: initializing ssl context
2024-03-31 13:54:12.919 INFO: gcomm thread scheduling priority set to other:0
2024-03-31 13:54:12.919 WARN: Fail to access the file (./gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2024-03-31 13:54:12.919 INFO: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2024-03-31 13:54:12.919 INFO: GMCast version 0
2024-03-31 13:54:12.921 INFO: (27d7731e, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567
2024-03-31 13:54:12.921 INFO: (27d7731e, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1
2024-03-31 13:54:12.921 INFO: EVS version 0
2024-03-31 13:54:12.921 INFO: gcomm: connecting to group ‘mysql-staging-cluster-pxc’, peer ‘mysql-staging-cluster-pxc-2.mysql-staging-cluster-pxc:’
2024-03-31 13:54:12.926 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:14.426 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:15.922 INFO: announce period timed out (pc.announce_timeout)
2024-03-31 13:54:15.923 WARN: no nodes coming from prim view, prim not possible
2024-03-31 13:54:15.923 INFO: Current view of cluster as seen by this node
view (view_id(NON_PRIM,27d7731e,1)
memb {
27d7731e,0
}
joined {
}
left {
}
partitioned {
}
)
2024-03-31 13:54:15.927 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:16.423 WARN: last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50179S), skipping check
2024-03-31 13:54:17.426 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:18.926 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:20.427 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:21.926 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:23.428 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:24.929 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:26.429 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:27.927 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:29.428 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:30.928 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)
2024-03-31 13:54:32.429 ERROR: handshake with remote endpoint ssl://100.64.26.86:4567 failed: asio.ssl:337047686: ‘certificate verify failed’ ( 337047686: ‘error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed’)

  • grep ‘State transfer request failed’ /tmp/garbd.log
  • grep ‘WARN: Protocol violation. JOIN message sender … (garb) is not in state transfer’ /tmp/garbd.log
  • grep ‘WARN: Rejecting JOIN message from … (garb): new State Transfer required.’ /tmp/garbd.log
  • grep ‘INFO: Shifting CLOSED → DESTROYED (TO: -1)’ /tmp/garbd.log
  • grep ‘INFO: Sending state transfer request’ /tmp/garbd.log
  • exit 1
2

@santosh240 please have a look here: Transport Encryption (TLS/SSL) - Percona Operator for MySQL based on Percona XtraDB Cluster

3

Thanks @Sergey_Pronin I have renewed the SSL certificates as per above and the cluster is running fine now with the renewed certificate.