# Canot bring the 3th node up

**URL:** <https://forums.percona.com/t/canot-bring-the-3th-node-up/13897>\
**Category:** Percona XtraDB Cluster\
**Created:** [January 13, 2022, 11:00am UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897 "2022-01-13T11:00:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![liking](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/liking/32/5574_2.png) [@liking](https://forums.percona.com/u/liking)\
**Post date:** [January 13, 2022, 11:00am UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/1 "2022-01-13T11:00:43Z")

</div>

When I attempt to startup the 3th node, I met this error:

```auto
2022-01-13T08:03:32.978322Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST.........
2022-01-13T08:03:33.036563Z 0 [Note] [MY-000000] [WSREP-SST] ............Waiting for SST streaming to complete!
2022-01-13T08:12:38.715388Z 0 [Note] [MY-000000] [Galera] Created page /mysql/pxc/data/gcache.page.000000 of size 592621440 bytes
2022-01-13T08:12:51.193262Z 0 [ERROR] [MY-000000] [WSREP-SST] Killing SST (27632) with SIGKILL after stalling for 120 seconds
2022-01-13T08:12:51.217686Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/wsrep_sst_xtrabackup-v2: line 183: 27634 killed socat -u openssl-listen:4444,reuseaddr,cert=/mysql/pxc/data//server-cert.pem,key=/mysql/pxc/data//server-key.pem,cafile=/mysql/pxc/data//ca.pem,verify=1,retry=30 stdio
2022-01-13T08:12:51.217754Z 0 [Note] [MY-000000] [WSREP-SST] 27635 | /usr/bin/pxc_extra/pxb-8.0/bin/xbstream -x
2022-01-13T08:12:51.218372Z 0 [ERROR] [MY-000000] [WSREP-SST] *******************FATAL ERROR********************** 
2022-01-13T08:12:51.218550Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while getting data from donor node: exit codes: 137 137
2022-01-13T08:12:51.218628Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1268
2022-01-13T08:12:51.218722Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** 
2022-01-13T08:12:51.219631Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32
2022-01-13T08:12:51.431617Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.230.245.214' --datadir '/mysql/pxc/data/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '27097' --mysqld-version '8.0.23-14.1' '' : 32 (Broken pipe)
2022-01-13T08:12:51.431820Z 0 [ERROR] [MY-000000] [WSREP] Failed to read uuid:seqno from joiner script.
2022-01-13T08:12:51.431892Z 0 [ERROR] [MY-000000] [WSREP] SST script aborted with error 32 (Broken pipe)
2022-01-13T08:12:51.432257Z 3 [Note] [MY-000000] [Galera] Processing SST received
2022-01-13T08:12:51.432372Z 3 [Note] [MY-000000] [Galera] SST request was cancelled
2022-01-13T08:12:51.432458Z 3 [ERROR] [MY-000000] [Galera] State transfer request failed unrecoverably: 32 (Broken pipe). Most likely it is due to inability to communicate with the cluster primary component. Restart required.

```

Who can give me help? Thanks a lot.

---

<div class="post-metadata">

**Author:** ![liking](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/liking/32/5574_2.png) [@liking](https://forums.percona.com/u/liking)\
**Post date:** [January 13, 2022, 11:04am UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/2 "2022-01-13T11:04:34Z")

</div>

My PXC cluster is 8.0.23, now there are 2 nodes running in CentOS 7.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [January 13, 2022, 5:56pm UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/3 "2022-01-13T17:56:51Z")

</div>

Your log indicates that port 4444 is not open TCP/UDP to all hosts. Make sure all necessary ports (3306, 4444, 4567, 4568) are open between all nodes.

---

<div class="post-metadata">

**Author:** ![liking](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/liking/32/5574_2.png) [@liking](https://forums.percona.com/u/liking)\
**Post date:** [January 14, 2022, 2:42am UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/4 "2022-01-14T02:42:05Z")

</div>

Thanks for your reply, but I am sure I have closed firewall between all nodes.  
Maybe there is some other issues?

---

<div class="post-metadata">

**Author:** ![Evgeniy\_Patlan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/evgeniy_patlan/32/41_2.png) [@Evgeniy\_Patlan](https://forums.percona.com/u/Evgeniy_Patlan)\
**Post date:** [January 14, 2022, 7:23am UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/5 "2022-01-14T07:23:50Z")

</div>

> [@liking](#):
>
> ` while getting data from donor node: exit codes: 137 137`

Such issue appeared once it is not possible to connect to the needed port. So please recheck your firewall options

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [January 14, 2022, 6:27pm UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/6 "2022-01-14T18:27:01Z")

</div>

> [@liking](#):
>
> I am sure I have closed firewall between all nodes

That’s your problem. You need to OPEN the firewall between nodes, not close it. Use `socat` or `nc` to test connectivity between nodes on the ports I mentioned.

---

<div class="post-metadata">

**Author:** ![liking](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/liking/32/5574_2.png) [@liking](https://forums.percona.com/u/liking)\
**Post date:** [January 15, 2022, 9:21am UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/7 "2022-01-15T09:21:08Z")

</div>

Many thanks to you all, I will do this according to your suggest

---

<div class="post-metadata">

**Author:** ![liking](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/liking/32/5574_2.png) [@liking](https://forums.percona.com/u/liking)\
**Post date:** [January 18, 2022, 2:35am UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/8 "2022-01-18T02:35:25Z")

</div>

I will post the final result when I finished doing this.  
But now I met some trouble, wait some days.

---

<div class="post-metadata">

**Author:** ![liking](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/liking/32/5574_2.png) [@liking](https://forums.percona.com/u/liking)\
**Post date:** [January 24, 2022, 4:12am UTC](https://forums.percona.com/t/canot-bring-the-3th-node-up/13897/9 "2022-01-24T04:12:51Z")

</div>

It is ok now.  
According to your suggest, I modified the netfilter rules on all nodes like this:

1. Accept all input
2. Clear all netfilter rules

```auto
[root@db-1 ~]# iptables -P INPUT ACCEPT
[root@db-1 ~]# iptables -F
[root@db-1 ~]# iptables -X
[root@db-1 ~]# iptables -Z
[root@db-1 ~]# iptables -A INPUT -i lo -j ACCEPT
[root@db-1 ~]# iptables-save
# Generated by iptables-save v1.4.21 on Mon Jan 24 11:33:23 2022
*filter
:INPUT ACCEPT [884:105489]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [685:162312]
-A INPUT -i lo -j ACCEPT
COMMIT
# Completed on Mon Jan 24 11:33:23 2022

```

Now the cluster works fine.  
Thank you again!
