Unable to add 3rd node to percona mysql cluster 8.0.26

1

Hi,
I am unable to add 3rd node to percona galera cluster.
The version is 8.0.26.
We are getting errors as below:

Logs on joiner node:
2025-07-13T10:33:45.229167Z 0 [ERROR] [MY-000000] [WSREP-SST] Killing SST (1376068) with SIGKILL after stalling for 120 seconds
2025-07-13T10:33:45.233181Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/wsrep_sst_xtrabackup-v2: line 185: 1376070 Killed
socat -u TCP-LISTEN:4444,reuseaddr,retry=30 stdio
2025-07-13T10:33:45.233199Z 0 [Note] [MY-000000] [WSREP-SST] 1376071 | /usr/bin/pxc_extra/pxb-8.0/bin/xbstre
am -x
2025-07-13T10:33:45.233822Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2025-07-13T10:33:45.233840Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while getting data from donor node: exit codes: 137 137
2025-07-13T10:33:45.233848Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296
2025-07-13T10:33:45.233854Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************************************************

Logs on donor node:
xtrabackup: Error writing file ‘’ (OS errno 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error writing file ‘’ (OS errno 32 - Broken pipe)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
xtrabackup: Error writing file ‘’ (OS errno 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error writing file ‘’ (OS errno 32 - Broken pipe)
[04] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[04] xtrabackup: Error: failed to copy datafile.
250713 12:33:46 >> log scanned up to (6598375090747)

2025-07-13T10:33:46.178449Z 0 [ERROR] [MY-000000] [WSREP-SST] ------------ innobackup.backup.log (END) ------------
2025-07-13T10:33:46.178454Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************************************************
2025-07-13T10:33:46.178457Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:22
2025-07-13T10:33:46.197075Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘abc-123.test.com:4444/xtrabackup_sst//1’ --socket ‘/var/run/mysqld/mysqld.sock’ --datadir ‘/var/lib/mysql/8.0/data/’ --basedir ‘/usr/’ --plugindir ‘/usr/lib64/mysql/plugin/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘8.0.26-16.1’ --binlog ‘mysql-bin’ --gtid ‘07b8b863-ab5f-11ec-b14b-5b158d5d7671:1068059262’ : 22 (Invalid argument)
2025-07-13T10:33:46.203003Z 0 [Note] [MY-000000] [Galera] SST sending failed: -22

This is a critical database.
Any help would be appreciated.

2

@ulka01

It might be related to SST timeout issue:- https://perconadev.atlassian.net/browse/PXC-4392 OR https://perconadev.atlassian.net/browse/PXC-4585

Did you tried to check the below PXB logs inside the data directory ? This will provide some more verbosity about the failure.

innobackup.backup.log
innobackup.prepare.log

Can you confirm the value of [ TimeoutStartSec] from /etc/systemd/system/mysqld.service or /etc/systemd/system/mysql.service ?

Please also check the OS/Kernel logs if there is no such networks/ other problems or resource issue blocking this ? I am hoping the resources [CPU, IO, Memory] is not a blocker here but please double confirm.

If SST having some problems you can also try to perform a manual streaming backup - https://docs.percona.com/percona-xtrabackup/8.0/take-streaming-backup.html over the newly added node if that works.

Try to perform a clean SST on the new node if you already having some existing data or left over there from previous attempts.

rm -f /var/lib/mysql/grastate.dat
OR
rm -rf /var/lib/mysql/*

Then restart the service again.

systemctl start mysql

Note - Please ensure the SSL Key/certificates are saved or backup before wiping out the complete data directory .

3

Hi,
We have the value as:
[Service]
TimeoutStartSec=9000

I have cleaned the /var/lib/mysql/ directory multiple times.
I will try an option that you suggested.
[sst]
sst-idle-timeout=3600

Also could you please explain in detail how a streaming backup will help.
I mean what actions shall I do after taking the streaming backup.
Never tried this way.

4

Hi,
Finally I was able to add the 3rd node.
I added this to my.cnf as suggested
[sst]
sst-idle-timeout=3600

Many Thanks @anil.joshi :saluting_face: