# PXC8.0.32 - nodes not joining the cluster

**URL:** <https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814>\
**Category:** MySQL & MariaDB\
**Created:** [February 20, 2025, 10:49am UTC](https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814 "2025-02-20T10:49:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abhinandan\_Aithal\_c](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/abhinandan_aithal_c/32/6297_2.png) [@Abhinandan\_Aithal\_c](https://forums.percona.com/u/Abhinandan_Aithal_c)\
**Post date:** [February 20, 2025, 10:49am UTC](https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814/1 "2025-02-20T10:49:38Z")

</div>

Hello,  
We have a 3 node percona cluster, with first node bootstrapped, we are trying to join node 2 and 3 to the cluster, but mysql service is failing after it tries to attempt SST.

> 2025-02-19T08:44:39.527965-05:00 0 [ERROR] [MY-000000] [Galera] async IST sender failed to serve tcp://10.32.74.231:4568: ist send failed: ', asio error ‘Failed to read: Connection reset by peer: 104 (Connection reset by peer)  
> at galerautils/src/gu\_asio\_stream\_react.cpp:throw\_sync\_op\_error():141’: 104 (Connection reset by peer)  
> at galera/src/ist.cpp:send():862  
> 2025-02-19T08:44:39.528075-05:00 0 [Note] [MY-000000] [Galera] async IST sender served  
> 2025-02-19T08:44:39.671549-05:00 0 [Warning] [MY-000000] [WSREP-SST] wsrep\_node\_address or wsrep\_sst\_receive\_address not set. Consider setting them if SST fails.  
> 2025-02-19T08:44:39.887101-05:00 26515 [Warning] [MY-013712] [Server] No suitable ‘keyring\_component\_metadata\_query’ service implementation found to fulfill the request.  
> 2025-02-19T08:44:39.904310-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:39 socat[1610805] E write(7, 0x5614d46b5000, 168): Connection refused  
> 2025-02-19T08:44:40.911086-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:40 socat[1610822] E write(7, 0x562461384000, 168): Connection refused  
> 2025-02-19T08:44:41.917966-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:41 socat[1610825] E write(7, 0x56184b596000, 146): Connection refused  
> 2025-02-19T08:44:42.924766-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:42 socat[1610828] E write(7, 0x55eb20845000, 168): Connection refused  
> 2025-02-19T08:44:43.931327-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:43 socat[1610845] E write(7, 0x564595457000, 168): Connection refused  
> 2025-02-19T08:44:44.531016-05:00 0 [Note] [MY-000000] [Galera] cleaning up 680836fa-8def (tcp://10.32.74.231:4567)  
> 2025-02-19T08:44:44.937792-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:44 socat[1610848] E write(7, 0x5652f4b21000, 168): Connection refused  
> 2025-02-19T08:44:45.944729-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:45 socat[1610851] E write(7, 0x564e9ff42000, 168): Connection refused  
> 2025-02-19T08:44:46.951024-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:46 socat[1610868] E write(7, 0x55b211263000, 168): Connection refused  
> 2025-02-19T08:44:47.958345-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:47 socat[1610871] E write(7, 0x5558906e3000, 168): Connection refused  
> 2025-02-19T08:44:48.965889-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:48 socat[1610874] E write(7, 0x56348728d000, 168): Connection refused  
> 2025-02-19T08:44:49.973197-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:49 socat[1610891] E write(7, 0x5613386a8000, 168): Connection refused

donor is showing below errors:-

> 2025-02-19T08:44:39.527965-05:00 0 [ERROR] [MY-000000] [Galera] async IST sender failed to serve tcp://10.32.74.231:4568: ist send failed: ', asio error ‘Failed to read: Connection reset by peer: 104 (Connection reset by peer)  
> at galerautils/src/gu\_asio\_stream\_react.cpp:throw\_sync\_op\_error():141’: 104 (Connection reset by peer)  
> at galera/src/ist.cpp:send():862  
> 2025-02-19T08:44:39.528075-05:00 0 [Note] [MY-000000] [Galera] async IST sender served  
> 2025-02-19T08:44:39.671549-05:00 0 [Warning] [MY-000000] [WSREP-SST] wsrep\_node\_address or wsrep\_sst\_receive\_address not set. Consider setting them if SST fails.  
> 2025-02-19T08:44:39.887101-05:00 26515 [Warning] [MY-013712] [Server] No suitable ‘keyring\_component\_metadata\_query’ service implementation found to fulfill the request.  
> 2025-02-19T08:44:39.904310-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:39 socat[1610805] E write(7, 0x5614d46b5000, 168): Connection refused  
> 2025-02-19T08:44:40.911086-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:40 socat[1610822] E write(7, 0x562461384000, 168): Connection refused  
> 2025-02-19T08:44:41.917966-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:41 socat[1610825] E write(7, 0x56184b596000, 146): Connection refused  
> 2025-02-19T08:44:42.924766-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:42 socat[1610828] E write(7, 0x55eb20845000, 168): Connection refused  
> 2025-02-19T08:44:43.931327-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:43 socat[1610845] E write(7, 0x564595457000, 168): Connection refused  
> 2025-02-19T08:44:44.531016-05:00 0 [Note] [MY-000000] [Galera] cleaning up 680836fa-8def (tcp://10.32.74.231:4567)  
> 2025-02-19T08:44:44.937792-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:44 socat[1610848] E write(7, 0x5652f4b21000, 168): Connection refused  
> 2025-02-19T08:44:45.944729-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:45 socat[1610851] E write(7, 0x564e9ff42000, 168): Connection refused  
> 2025-02-19T08:44:46.951024-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:46 socat[1610868] E write(7, 0x55b211263000, 168): Connection refused  
> 2025-02-19T08:44:47.958345-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:47 socat[1610871] E write(7, 0x5558906e3000, 168): Connection refused  
> 2025-02-19T08:44:48.965889-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:48 socat[1610874] E write(7, 0x56348728d000, 168): Connection refused  
> 2025-02-19T08:44:49.973197-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:49 socat[1610891] E write(7, 0x5613386a8000, 168): Connection refused

There are no permission issues with `/usr` , `/usr/bin` , `/usr/bin/ls` , `/proc` or `/usr/bin/wsrep_sst_xtrabackup-v2` . There is no connectivity issue between the servers on port `4444` , `4567` and port `4568` selinux and local firewall disabled, no iptables entries.

Can someone please help us fix this issue.

-Abhi

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [February 20, 2025, 10:30pm UTC](https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814/2 "2025-02-20T22:30:25Z")

</div>

> 2025-02-19T08:44:39.904310-05:00 0 [Note] [MY-000000] [WSREP-SST] 2025/02/19 08:44:39 socat[1610805] E write(7, 0x5614d46b5000, 168): Connection refused

This looks like network issues. It states so quite directly. You can test this yourself using `socat` on node1 in listen mode, then use socat on node2 to send some text to node1. If that fails, then you know 100% there are network issues.

---

<div class="post-metadata">

**Author:** ![Abhinandan\_Aithal\_c](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/abhinandan_aithal_c/32/6297_2.png) [@Abhinandan\_Aithal\_c](https://forums.percona.com/u/Abhinandan_Aithal_c)\
**Post date:** [February 21, 2025, 6:28am UTC](https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814/3 "2025-02-21T06:28:50Z")

</div>

Hi @matthewb, We did test with below command on ports 4444, 4567 and 4568 and didn’t see any issues.

> node1: socat - TCP-LISTEN:4444  
> node2: echo “hello” | socat - TCP:ip.adr.of.node1:4444

If we change user to mysql and execute a mysld & command, it works and node got synced with its donor. Strangely the service startup/SST fails via systemd service start (service mysqld restart). May be some permission issue since this is run as root user which we can’t seem to figure out.

-Abhi

---

<div class="post-metadata">

**Author:** ![Yunus](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/yunus/32/6430_2.png) [@Yunus](https://forums.percona.com/u/Yunus)\
**Post date:** [February 21, 2025, 7:54am UTC](https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814/4 "2025-02-21T07:54:09Z")

</div>

systemd should have user and group as mysql  
what command did you exactly execute for starting MySQL?  
And can you share the output of

```auto
systemctl cat mysqld

```

---

<div class="post-metadata">

**Author:** ![Abhinandan\_Aithal\_c](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/abhinandan_aithal_c/32/6297_2.png) [@Abhinandan\_Aithal\_c](https://forums.percona.com/u/Abhinandan_Aithal_c)\
**Post date:** [February 21, 2025, 8:30am UTC](https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814/5 "2025-02-21T08:30:03Z")

</div>

Hi Yunus This is the permission we have if this is what you are asking. I think the permissions are correct on the files/folders. We did validate the same with a working environment.

> ls -la /etc/ |grep systemd  
> drwxr-xr-x. 4 root root 195 May 18 2023 systemd
> 
> ls -la /etc/systemd/system/mysqld.service  
> lrwxrwxrwx 1 root root 37 Jul 25 2023 /etc/systemd/system/mysqld.service → /usr/lib/systemd/system/mysql.service
> 
> ls -la /usr/lib/systemd/system/mysql.service  
> -rw-r–r-- 1 root root 4080 Feb 20 15:38 /usr/lib/systemd/system/mysql.service

I am executing service mysqld start / systemctl start mysql. Here is the error we are currently chasing to solve on the joiner node:-

> 2025-02-20T10:57:13.824732-05:00 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/wsrep\_sst\_xtrabackup-v2: line 1254: /usr/bin/ls: Operation not permitted  
> 2025-02-20T10:57:13.828019-05:00 0 [Note] [MY-000000] [WSREP-SST] (debug) 1258: sockets:  
> 2025-02-20T10:57:14.029559-05:00 0 [Note] [MY-000000] [WSREP-SST] (debug) 1228: Entering loop body : 300  
> 2025-02-20T10:57:14.049399-05:00 0 [Note] [MY-000000] [WSREP-SST] (debug) 1237: Examining pid: 41914  
> 2025-02-20T10:57:14.100138-05:00 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/wsrep\_sst\_xtrabackup-v2: line 1250: /usr/bin/ls: Operation not permitted  
> 2025-02-20T10:57:14.100391-05:00 0 [Note] [MY-000000] [WSREP-SST] (debug) 1250: Testing ls :  
> 2025-02-20T10:57:14.101000-05:00 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/wsrep\_sst\_xtrabackup-v2: line 1251: /usr/bin/ls: Operation not permitted  
> 2025-02-20T10:57:14.101283-05:00 0 [Note] [MY-000000] [WSREP-SST] (debug) 1251: Testing ls :  
> 2025-02-20T10:57:14.101770-05:00 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/wsrep\_sst\_xtrabackup-v2: line 1252: /usr/bin/ls: Operation not permitted  
> 2025-02-20T10:57:14.102034-05:00 0 [Note] [MY-000000] [WSREP-SST] (debug) 1252: Testing ls :  
> 2025-02-20T10:57:14.102819-05:00 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/wsrep\_sst\_xtrabackup-v2: line 1254: /usr/bin/ls: Operation not permitted

Here is systemctl cat mysqld output.

[systemctl.txt](https://forums.percona.com/uploads/short-url/wlkCjkZVLY1K4efnjBBwDNbiGhT.txt) (4.2 KB)

-Abhi

---

<div class="post-metadata">

**Author:** ![CTutte](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ctutte/32/1341_2.png) [@CTutte](https://forums.percona.com/u/CTutte)\
**Post date:** [February 28, 2025, 1:52pm UTC](https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814/6 "2025-02-28T13:52:05Z")

</div>

Hi!

> wsrep\_node\_address or wsrep\_sst\_receive\_address not set.  
> Did you check that configuration is correctly set in all nodes ?

Also did you check that all ports are open on all 3 servers and not just on 1?

All MySQL files owner/group should be MySQL.

Are all nodes the same version and galera protocol ?

You can also try enablin “wsrep\_debug” for extra verbosity.

Regards

---

<div class="post-metadata">

**Author:** ![Abhinandan\_Aithal\_c](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/abhinandan_aithal_c/32/6297_2.png) [@Abhinandan\_Aithal\_c](https://forums.percona.com/u/Abhinandan_Aithal_c)\
**Post date:** [March 1, 2025, 7:09am UTC](https://forums.percona.com/t/pxc8-0-32-nodes-not-joining-the-cluster/36814/7 "2025-03-01T07:09:23Z")

</div>

Hi All,

Commenting out the CapabilityBoundingSet in mysqld.service worked and SST succeeded. There might have been some security hardening on these 3 DB instances which is somehow not allowing to execute ls command. We were able to replicate the same issue via a test service script.

Thanks for all your valuable inputs.

-Abhi