Helm Chart versions:
pxc-operator:1.13.0
pxc-db:1.13.0
Distro: RKE2, Ubuntu 22 LTS
Automatic upgrades for my PXC cluster isn’t working.
I’ve also tested creating a new, separate PXC cluster install (via Helm) and it shows similar errors:
PXC pod logs: (truncated because it’s long)
trap exit SIGTERM
+ '[' m = - ']'
+ CFG=/etc/mysql/node.cnf
+ wantHelp=
+ for arg in "$@"
+ case "$arg" in
++ mysqld -V
++ awk '{print $3}'
++ awk -F. '{print $1"."$2}'
+ MYSQL_VERSION=8.0
++ mysqld -V
++ awk '{print $3}'
++ awk -F. '{print $3}'
++ awk -F- '{print $1}'
+ MYSQL_PATCH_VERSION=32
+ vault_secret=/etc/mysql/vault-keyring-secret/keyring_vault.conf
+ '[' -f /etc/mysql/vault-keyring-secret/keyring_vault.conf ']'
+ '[' -f /usr/lib64/mysql/plugin/binlog_utils_udf.so ']'
+ sed -i '/\[mysqld\]/a plugin_load="binlog_utils_udf=binlog_utils_udf.so"' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a gtid-mode=ON' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a enforce-gtid-consistency' /etc/mysql/node.cnf
+ grep -q '^progress=' /etc/mysql/node.cnf
+ sed -i 's|^progress=.*|progress=1|' /etc/mysql/node.cnf
+ grep -q '^\[sst\]' /etc/mysql/node.cnf
+ grep -q '^cpat=' /etc/mysql/node.cnf
+ sed '/^\[sst\]/a cpat=.*\\.pem$\\|.*init\\.ok$\\|.*galera\\.cache$\\|.*wsrep_recovery_verbose\\.log$\\|.*readiness-check\\.sh$\\|.*liveness-check\\.sh$\\|.*get-pxc-state$\\|.*sst_in_progress$\\|.*pmm-prerun\\.sh$\\|.*sst-xb-tmpdir$\\|.*\\.sst$\\|.*gvwstate\\.dat$\\|.*grastate\\.dat$\\|.*\\.err$\\|.*\\.log$\\|.*RPM_UPGRADE_MARKER$\\|.*RPM_UPGRADE_HISTORY$\\|.*pxc-entrypoint\\.sh$\\|.*unsafe-bootstrap\\.sh$\\|.*pxc-configure-pxc\\.sh\\|.*peer-list$\\|.*auth_plugin$' /etc/mysql/node.cnf
+ [[ 8.0 == \8\.\0 ]]
+ [[ 32 -ge 26 ]]
+ grep -q '^skip_replica_start=ON' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a skip_replica_start=ON' /etc/mysql/node.cnf
+ auth_plugin=mysql_native_password
+ [[ -f /var/lib/mysql/auth_plugin ]]
++ cat /var/lib/mysql/auth_plugin
+ prev_auth_plugin=mysql_native_password
+ [[ mysql_native_password != \m\y\s\q\l\_\n\a\t\i\v\e\_\p\a\s\s\w\o\r\d ]]
+ [[ -z mysql_native_password ]]
+ [[ 8.0 == \5\.\7 ]]
+ echo mysql_native_password
+ sed -i /default_authentication_plugin/d /etc/mysql/node.cnf
+ [[ 8.0 == \8\.\0 ]]
+ [[ 32 -ge 27 ]]
+ sed -i '/\[mysqld\]/a authentication_policy=mysql_native_password,,' /etc/mysql/node.cnf
+ file_env XTRABACKUP_PASSWORD xtrabackup xtrabackup
Percona XtraDB Cluster: Finding peers
2023/08/04 04:22:56 Peer finder enter
2023/08/04 04:22:56 Determined Domain to be percona.svc.cluster.local
2023/08/04 04:22:56 Peer list updated
was []
now [10-42-189-67.pxc-db-pxc-unready.percona.svc.cluster.local 10-42-235-156.pxc-db-pxc-unready.percona.svc.cluster.local 10-42-42-179.pxc-db-pxc-unready.percona.svc.cluster.local]
2023/08/04 04:22:56 execing: /var/lib/mysql/pxc-configure-pxc.sh with stdin: 10-42-189-67.pxc-db-pxc-unready.percona.svc.cluster.local
10-42-235-156.pxc-db-pxc-unready.percona.svc.cluster.local
10-42-42-179.pxc-db-pxc-unready.percona.svc.cluster.local
2023/08/04 04:22:56 ++ hostname -I
++ awk ' { print $1 } '
+ NODE_IP=10.42.42.179
++ hostname -f
++ cut -d. -f2
+ CLUSTER_NAME=pxc-db-pxc
+ SERVER_NUM=2
+ SERVER_ID=18884702
++ hostname -f
+ NODE_NAME=pxc-db-pxc-2.pxc-db-pxc.percona.svc.cluster.local
+ NODE_PORT=3306
+ read -ra LINE
+ echo 'read line 10-42-189-67.pxc-db-pxc-unready.percona.svc.cluster.local'
read line 10-42-189-67.pxc-db-pxc-unready.percona.svc.cluster.local
++ getent hosts 10-42-189-67.pxc-db-pxc-unready.percona.svc.cluster.local
++ awk '{ print $1 }'
+ LINE_IP=10.42.189.67
+ '[' 10.42.189.67 '!=' 10.42.42.179 ']'
++ mysql_root_exec 10.42.189.67 'select @@hostname'
++ local server=10.42.189.67
++ local 'query=select @@hostname'
+ LINE_HOST=pxc-db-pxc-0
+ '[' -n pxc-db-pxc-0 ']'
+ PEERS=("${PEERS[@]}" $LINE_HOST)
+ PEERS_FULL=("${PEERS_FULL[@]}" "$LINE_HOST.$CLUSTER_NAME")
+ read -ra LINE
+ echo 'read line 10-42-235-156.pxc-db-pxc-unready.percona.svc.cluster.local'
read line 10-42-235-156.pxc-db-pxc-unready.percona.svc.cluster.local
++ getent hosts 10-42-235-156.pxc-db-pxc-unready.percona.svc.cluster.local
++ awk '{ print $1 }'
+ LINE_IP=10.42.235.156
+ '[' 10.42.235.156 '!=' 10.42.42.179 ']'
++ mysql_root_exec 10.42.235.156 'select @@hostname'
++ local server=10.42.235.156
++ local 'query=select @@hostname'
+ LINE_HOST=pxc-db-pxc-1
+ '[' -n pxc-db-pxc-1 ']'
+ PEERS=("${PEERS[@]}" $LINE_HOST)
+ PEERS_FULL=("${PEERS_FULL[@]}" "$LINE_HOST.$CLUSTER_NAME")
+ read -ra LINE
+ echo 'read line 10-42-42-179.pxc-db-pxc-unready.percona.svc.cluster.local'
read line 10-42-42-179.pxc-db-pxc-unready.percona.svc.cluster.local
++ getent hosts 10-42-42-179.pxc-db-pxc-unready.percona.svc.cluster.local
++ awk '{ print $1 }'
+ LINE_IP=10.42.42.179
+ '[' 10.42.42.179 '!=' 10.42.42.179 ']'
+ read -ra LINE
+ '[' 2 '!=' 0 ']'
++ printf '%s\n' pxc-db-pxc-0 pxc-db-pxc-1 pxc-db-pxc-2
++ sort --version-sort
++ uniq
++ grep -v -- '-0$'
++ sed '$d'
++ tr '\n' ,
++ sed 's/^,$//'
+ DONOR_ADDRESS=pxc-db-pxc-1,
+ '[' 2 '!=' 0 ']'
++ printf '%s\n' pxc-db-pxc-0.pxc-db-pxc pxc-db-pxc-1.pxc-db-pxc
++ sort --version-sort
++ tr '\n' ,
++ sed 's/,$//'
+ WSREP_CLUSTER_ADDRESS=pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc
+ CFG=/etc/mysql/node.cnf
++ mysqld -V
++ awk '{print $3}'
++ awk -F. '{print $1"."$2}'
+ MYSQL_VERSION=8.0
+ '[' 8.0 == 8.0 ']'
+ grep -E -q '^[#]?admin-address' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a admin-address=\n' /etc/mysql/node.cnf
+ grep -E -q '^[#]?log_error_suppression_list' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a log_error_suppression_list="MY-010055"\n' /etc/mysql/node.cnf
+ '[' yes == yes ']'
+ grep -E -q '^[#]?log-error' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a log-error=/var/lib/mysql/mysqld-error.log\n' /etc/mysql/node.cnf
+ grep -E -q '^[#]?wsrep_sst_donor' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a wsrep_sst_donor=\n' /etc/mysql/node.cnf
+ grep -E -q '^[#]?wsrep_node_incoming_address' /etc/mysql/node.cnf
+ grep -E -q '^[#]?wsrep_provider_options' /etc/mysql/node.cnf
+ sed '/^\[mysqld\]/a wsrep_provider_options="pc.weight=10"\n' /etc/mysql/node.cnf
+ sed -r 's|^[#]?server_id=.*$|server_id=18884702|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?coredumper$|coredumper|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_node_address=.*$|wsrep_node_address=10.42.42.179|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_cluster_name=.*$|wsrep_cluster_name=pxc-db-pxc|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_sst_donor=.*$|wsrep_sst_donor=pxc-db-pxc-1,|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_cluster_address=.*$|wsrep_cluster_address=gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?wsrep_node_incoming_address=.*$|wsrep_node_incoming_address=pxc-db-pxc-2.pxc-db-pxc.percona.svc.cluster.local:3306|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?admin-address=.*$|admin-address=10.42.42.179|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?extra_max_connections=.*$|extra_max_connections=100|' /etc/mysql/node.cnf
+ sed -r 's|^[#]?extra_port=.*$|extra_port=33062|' /etc/mysql/node.cnf
+ CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+ '[' -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ']'
+ SSL_DIR=/etc/mysql/ssl
+ '[' -f /etc/mysql/ssl/ca.crt ']'
+ SSL_INTERNAL_DIR=/etc/mysql/ssl-internal
+ '[' -f /etc/mysql/ssl-internal/ca.crt ']'
+ KEY=/etc/mysql/ssl/tls.key
+ CERT=/etc/mysql/ssl/tls.crt
+ '[' -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ']'
+ '[' -f /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -a -f /etc/mysql/ssl/tls.key -a -f /etc/mysql/ssl/tls.crt ']'
+ sed '/^\[mysqld\]/a pxc-encrypt-cluster-traffic=OFF' /etc/mysql/node.cnf
2023/08/04 04:22:57 Peer finder exiting
Cluster address set to: pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc
/usr/sbin/mysqld Ver 8.0.32-24.2 for Linux on x86_64 (Percona XtraDB Cluster (GPL), Release rel24, Revision 2119e75, WSREP version 26.1.4.3)
1c1
< /usr/sbin/mysqld Ver 8.0.32-24.2 for Linux on x86_64 (Percona XtraDB Cluster (GPL), Release rel24, Revision 2119e75, WSREP version 26.1.4.3)
---
> /usr/sbin/mysqld Ver 8.0.31-23.2 for Linux on x86_64 (Percona XtraDB Cluster (GPL), Release rel23, Revision e6e483f, WSREP version 26.1.4.3)
+ for i in {120..0}
+ echo 'SELECT 1'
+ mysql --protocol=socket -uoperator -hlocalhost --socket=/tmp/mysql.sock --password= -px
+ echo 'MySQL init process in progress...'
+ sleep 1
.....
MySQL init process in progress...
+ for i in {120..0}
+ echo 'SELECT 1'
+ mysql --protocol=socket -uoperator -hlocalhost --socket=/tmp/mysql.sock --password= -px
+ echo 'MySQL init process in progress...'
MySQL init process in progress...
+ sleep 1
+ '[' 0 = 0 ']'
+ echo 'MySQL init process failed.'
MySQL init process failed.
+ exit 1
I wasn’t expecting init (initialise from a version upgrade?).
Log pod:
pxc-db-pxc-2_logs.log (3.0 MB)
PXC update config:
updateStrategy: SmartUpdate
upgradeOptions:
apply: recommended
schedule: 0 4 * * *
versionServiceEndpoint: https://check.percona.com
Previous image:
image:
repository: percona/percona-xtradb-cluster
tag: 8.0.31-23.2
