SST can't support IPv6

liking · October 25, 2023, 4:08am

Hi,
I configured IPv6 in PXC nodes, but met errors.
This is my.cnf of node1:

wsrep_cluster_address=gcomm://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4567,[fd15:4ba5:5a2b:1008:f814:e50e:acbd:12]:4567
wsrep_node_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:12]:4567
wsrep_node_incoming_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:12]:3306
wsrep_sst_receive_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4444
wsrep_provider_options="gmcast.listen_addr=tcp://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4567;ist.recv_addr=tcp://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4568;socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"

When node2 request SST, I can see port 4444 supporting IPv4 and IPv6:

[root@node2:2 ~]# ss -ntl
State      Recv-Q Send-Q                          Local Address:Port                                         Peer Address:Port              
LISTEN     0      128                                 127.0.0.1:9000                                                    *:*                  
LISTEN     0      128                                         *:111                                                     *:*                  
LISTEN     0      5                               192.168.122.1:53                                                      *:*                  
LISTEN     0      128                                         *:22                                                      *:*                  
LISTEN     0      128                                 127.0.0.1:631                                                     *:*                  
LISTEN     0      5                                           *:4444                                                    *:*                  
LISTEN     0      128                                        :::111                                                    :::*                  
LISTEN     0      128                                        :::80                                                     :::*                  
LISTEN     0      128                                        :::22                                                     :::*                  
LISTEN     0      128     fd15:4ba5:5a2b:1008:f814:e50e:acbd:12:4567                                                   :::*                  
LISTEN     0      128                                       ::1:631                                                    :::*

But node1 can’t connect to port 4444:

2023-10-24T11:54:28.980520Z 2 [Note] [MY-000000] [WSREP] DONOR thread signaled with 0
2023-10-24T11:54:29.262908Z 20 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2023-10-24T11:54:29.287393Z 0 [Note] [MY-000000] [WSREP-SST] 2023/10/24 19:54:29 socat[34844] E SSL_connect(): Connection refused

I test connecting to port 4444 of node2 by telnet:

[root@node2:2 ~]# telnet node2 4444
Trying fd15:4ba5:5a2b:1008:f814:e50e:acbd:12...
telnet: connect to address fd15:4ba5:5a2b:1008:f814:e50e:acbd:12: Connection refused
Trying 192.168.222.12...
Connected to node2.
Escape character is '^]'.
Connection closed by foreign host.

From mysqld.log I have found port 4567 and 4568 on IPv6 works well, but port 4444 can’t work using IPv6. Who can tell me why?
I am sure I had closed firewall, and there is no access limit.
Many thanks!

liking · October 26, 2023, 1:31am

My PXC version and OS version:

mysql  Ver 8.0.32-24.2 for Linux on x86_64 (Percona XtraDB Cluster (GPL), Release rel24, Revision 2119e75, WSREP version 26.1.4.3)
CentOS Linux release 7.6.1810 (Core) 
Linux node1 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

my.cnf file of node1:

[mysqld]
server-id=1
datadir=/mysql/pxc/data
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
binlog_expire_logs_seconds=604800
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so
wsrep_cluster_address=gcomm://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4567,[fd15:4ba5:5a2b:1008:f814:e50e:acbd:12]:4567
binlog_format=ROW
wsrep_slave_threads=8
wsrep_log_conflicts
innodb_autoinc_lock_mode=2
wsrep_cluster_name=pxc-cluster
wsrep_node_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4567
wsrep_node_incoming_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:3306
wsrep_sst_receive_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4444
wsrep_node_name=node1
pxc_strict_mode=ENFORCING
wsrep_sst_method=xtrabackup-v2
wsrep_provider_options="gmcast.listen_addr=tcp://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4567;ist.recv_addr=tcp://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4568;socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"
lower_case_table_names=1
max_allowed_packet=200M
character-set-server=utf8
secure_file_priv=''
[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem

Kamil_Holubicki · October 30, 2023, 12:27pm

Hi @liking ,
I just check on two AWS instances (ubuntu 22.04, PXC 8.0.33) and everything works as expected
Here is my config:
Node 1 (donor):

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.35.198  netmask 255.255.240.0  broadcast 172.31.47.255
        inet6 fe80::2d:e5ff:fe2b:7377  prefixlen 64  scopeid 0x20<link>
        inet6 2600:1f18:3f4:7c00:ab2b:a674:73aa:d15f  prefixlen 128  scopeid 0x0<global>

mysqld.cnf:

wsrep_cluster_address=gcomm://
wsrep_node_address=[2600:1f18:3f4:7c00:ab2b:a674:73aa:d15f]:4567
wsrep_node_incoming_address=[2600:1f18:3f4:7c00:ab2b:a674:73aa:d15f]:3306
wsrep_provider_options="gmcast.listen_addr=tcp://[2600:1f18:3f4:7c00:ab2b:a674:73aa:d15f]:4567"

# just for simplicity
pxc_encrypt_cluster_traffic=OFF

node2 (joiner):

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.42.254  netmask 255.255.240.0  broadcast 172.31.47.255
        inet6 fe80::f6:abff:fe9a:92e7  prefixlen 64  scopeid 0x20<link>
        inet6 2600:1f18:3f4:7c00:d35d:827f:62c4:44cb  prefixlen 128  scopeid 0x0<global>

mysqld.cnf:

wsrep_cluster_address=gcomm://[2600:1f18:3f4:7c00:ab2b:a674:73aa:d15f]:4567
wsrep_node_address=[2600:1f18:3f4:7c00:d35d:827f:62c4:44cb]:4567
wsrep_node_incoming_address=[2600:1f18:3f4:7c00:d35d:827f:62c4:44cb]:3306
wsrep_provider_options="gmcast.listen_addr=tcp://[2600:1f18:3f4:7c00:d35d:827f:62c4:44cb]:4567"
pxc_encrypt_cluster_traffic=OFF

Hope that helps

liking · November 7, 2023, 9:35am

Thank you very much for your reply.
I will post the solution in the next reply.

liking · November 7, 2023, 9:58am

Finally, I found the solution is turn off pxc_encrypt_cluster_traffic, then it works well.
And this is my major part of conf file:

[mysqld]
wsrep_cluster_address=gcomm://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4567,[fd15:4ba5:5a2b:1008:f814:e50e:acbd:12]:4567
wsrep_node_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4567
wsrep_node_incoming_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:3306
wsrep_sst_receive_address=[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4444
wsrep_cluster_name=pxc-cluster
pxc_strict_mode=ENFORCING
wsrep_sst_method=xtrabackup-v2
pxc_encrypt_cluster_traffic=OFF
wsrep_provider_options="gmcast.listen_addr=tcp://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4567;ist.recv_addr=tcp://[fd15:4ba5:5a2b:1008:f814:e50e:acbd:11]:4568"

So the most important things are these two:

Set every port of IPv6 like above
Set pxc_encrypt_cluster_traffic=OFF

So the question is why SST doesn’t support encrypt cluster traffic via IPv6 ? Is there a bug somewhere?

Kamil_Holubicki · November 7, 2023, 2:27pm

Hi @liking , thank you for your feedback. Could you please create a Jira ticket for this? Please provide all the information that would help to setup the test environment where the issue can be reproduced. It should work with pxc_encrypt_cluster_traffic=ON as well. If it doesn’t, there is apparently a bug there

Edit: Just to be sure: you are using the same certificate files on all nodes, right? (this would help)

liking · November 9, 2023, 9:37am

Yes, I am sure using the same certificate file on all nodes.
And, I had said only 4444 doesn’t work via IPv6, the other ports works well via IPv6.

Kamil_Holubicki · November 9, 2023, 9:46am

In such a case I would say that there is something in your env which is blocking port 4444. Nothing special about this port in PXC

liking · November 9, 2023, 9:56am

But when I turn off encrypt traffic, port 4444 works well.
Wait for some days, I will try test again.
Thanks.

Topic		Replies	Views
SST not working with xtrabackup-v2, only with rsync. IPv6 issue ? Percona XtraDB Cluster 5.x	3	3324	January 25, 2018
Percona xtraDB cluster fails to come up on IPv6 Percona XtraDB Cluster 5.x community , mysql , percona	3	946	November 7, 2022
XtraBackup SST failing when adding 4th node Percona XtraBackup	0	796	June 13, 2013
update pxc node from 5.6.16 to 5.6.19 Percona XtraDB Cluster 5.x	3	1004	March 25, 2016
Percona XtraDB Cluster SST not working using rsync: wsrep_sst_rsync Percona XtraDB Cluster 5.x	4	2976	January 6, 2015

SST can't support IPv6

Related Topics