We have successfully deployed a 3 node Percona XtraDB Cluster v8.4.4 via the el8 (glibc2.28) “miminal” tarballs on 3 RHEL8.10 servers. Everything is running as root and deployed under /root directory (tarball extracted into /root/install/ and datadir set as /root/datastore/var/lib/mysql)
- Node 1 = ‘ha4’ - example IP address of 1.1.1.1
- Node 2 = ‘ha5’ - example IP address of 2.2.2.2
- Node 3 = ‘ha6’ - example IP address of 3.3.3.3
We are then trying to do a rolling minor version upgrade of the cluster to v8.4.8, guided generally by the docs at Upgrade Percona XtraDB Cluster - Percona XtraDB Cluster and so far we cannot get an upgraded node to rejoin the cluster and IST/SST times out on that node after approx 100s and the node aborts.
Our general steps to attempt to upgrade a node are:
- Gracefully stop the mysql process
- Download and extract the PXC8.4.8 glibc2.28 minimal tarball to the /root/install dir
- Update the PATH variable in the /root/.bashrc file to reference the newer Percona packages, then source the file.
- Update the plugin_dir and wsrep_provider properties in /root/.my.cnf to point to the new PXC8.4.8 directory
- Start the mysql process
The node will startup and give indication it is becoming a JOINER and commence an IST/SST (we generally would expect IST only as our connected applications are shutdown - it should only be a few transactions behind the cluster.). We see one of the other nodes become a DONOR and seemingly sends across the required data successfully. The upgraded node will then do nothing for approx 100s then abort. In our environment this is happening every time for any of the 3 nodes that we try to upgrade first.
The most suspicious log entry we can see is:
2026-09-10T02:51:00.954208-04:00 0 [ERROR] [MY-000000] [WSREP-SST] Possible timeout in receving first data from donor in gtid/keyring stage.
2026-09-10T02:51:00.954215-04:00 0 [ERROR] [MY-000000] [WSREP-SST] After waiting for the 100 seconds
Curiously, if we revert the change i.e put the PATH and config back to PXC8.4.4 and start the node, it happily rejoins the cluster.
We have also tried upgrading a node to all released versions from PXC8.4.5 up to PXC8.4.10 with the same result.
Questions
- Is anyone aware of issues performing rolling upgrades from PXC 8.4.4 to 8.4.8 in a tarball deployment?
- Has anyone seen the
gtid/keyring stagetimeout during an IST-based rejoin? - Are there any known changes in the SST/xtrabackup/keyring handling between 8.4.4 and 8.4.8 that could explain donor-side success but joiner-side timeouts?
- Are there additional debug options for
wsrep_sst_xtrabackup-v2that Percona recommends for this scenario?
The most puzzling aspect is that the donor appears satisfied that IST can proceed and complete, while the 8.4.8 joiner consistently times out and never reaches Synced, yet the same procedure works immediately when reverting the node back to 8.4.4.
MySQL config (from node2) provided below (other nodes are essentially identical) and log snippets provided from an attempt to upgrade node2 (ha5) with node1 (ha4) being the DONOR in that case.
many thanks
Node 3(ha6)_mysqld.log.txt (11.5 KB)
Node 2(ha5)_mysqld.log.txt (24.4 KB)
Node 1(ha4)_mysqld.log.txt (17 KB)
.my.cnf_contents.txt (3.3 KB)