Node not able to join the cluster - PXC 8.0.27

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

Use case : Node not able to join the cluster ,Percona XtraDB Cluster 8.0.27

Hi there,
I configured 3 node(n1,n2 & n3) cluster ,all the three nodes are online.
Due to huge connections ,the ‘node n3’ is down.In the meantime n1 and n2 nodes are processing the workloads.
When iam trying to add the ‘node-n3’ to cluster , iam facing the below error.
Please let me know how do i bring the ‘node-n3’ online.


2022-05-26T17:22:58.778759+05:30 0 [Note] [MY-000000] [Galera] (b5351f19-a07a, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.2:4567 timed out, no messages seen in PT3S, socket stats: rtt: 132 rttvar: 66 rto: 3216000 lost: 1 last_data_recv: 26619706 cwnd: 1 last_queued_since: 26919705827915 last_delivered_since: 26919705827915 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)

2022-05-26T17:23:02.279023+05:30 0 [Note] [MY-000000] [Galera] (b5351f19-a07a, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.1:4567 timed out, no messages seen in PT3S, socket stats: rtt: 137 rttvar: 68 rto: 3216000 lost: 1 last_data_recv: 26623207 cwnd: 1 last_queued_since: 26923206088048 last_delivered_since: 26923206088048 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
1 Like
2

@AneeshBabu what is your k8s node version?

1 Like
3

@A_Niu
Iam not using k8s.

1 Like
4

There are very few details of the issue for anyone to try and help you. Please provide more details of how your cluster is setup, samples of the my.cnf from the nodes pay special attention to the cluster settings.

1 Like
5

@Wayne_Leutwyler
below are the config files of three nodes.

  # Config file for PXC 8.0.27 Node 1

[client]
socket=/var/lib/mysql/mysql.sock

[mysqld]
wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”
server-id=1


datadir =/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

log_timestamps = system

# Binary log expiration period is 604800 seconds, which equals 7 days

binlog_expire_logs_seconds=604800
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so

# IP address

wsrep_node_address=192.168.1.1
wsrep_cluster_address=gcomm://192.168.1.1,192.168.1.2,192.168.1.3


binlog_format=row
wsrep_applier_threads=64
wsrep_log_conflicts
innodb_autoinc_lock_mode=2

# MySQL parameters
default_storage_engine=InnoDB
lower_case_table_names=1
innodb_buffer_pool_size=8G

# Node IP address

wsrep_cluster_name=dev-cluster
wsrep_node_name=pxc1
pxc_strict_mode=ENFORCING
wsrep_sst_method=xtrabackup-v2
pxc-encrypt-cluster-traffic=on

 [sst]
 encrypt = 4
 ssl-ca=/etc/mysql/certs/ca.pem
 ssl-cert=/etc/mysql/certs/server-cert.pem
 ssl-key=/etc/mysql/certs/server-key.pem

# Config file for PXC 8.0.27 Node 2


[client]
socket=/var/lib/mysql/mysql.sock

[mysqld]
wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”
server-id=2
datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

log_timestamps = system

# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so

# IP address
wsrep_cluster_address=gcomm://192.168.1.1,192.168.1.2,192.168.1.3

binlog_format=row
wsrep_applier_threads=64
wsrep_log_conflicts
innodb_autoinc_lock_mode=2

# Node IP address
wsrep_node_address=192.168.1.2
wsrep_cluster_name=dev-cluster

wsrep_node_name=pxc2
pxc_strict_mode=ENFORCING
wsrep_sst_method=xtrabackup-v2
pxc-encrypt-cluster-traffic=ON

# MySQL parameters
default_storage_engine=InnoDB
lower_case_table_names=1
innodb_buffer_pool_size=8G

[sst]
 encrypt = 4
 ssl-ca=/var/lib/mysql/ca.pem
 ssl-cert=/var/lib/mysql/server-cert.pem
 ssl-key=/var/lib/mysql/server-key.pem

# Config file for PXC 8.0.27 Node 3

[client]
socket=/var/lib/mysql/mysql.sock

[mysqld]
wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”
server-id=3


datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

log_timestamps = system

# Binary log expiration period is 604800 seconds, which equals 7 days

binlog_expire_logs_seconds=604800
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so

# IP address

wsrep_node_address=192.168.1.3
wsrep_cluster_address=gcomm://192.168.1.1,192.168.1.2,192.168.1.3

binlog_format=row
wsrep_applier_threads=64
wsrep_log_conflicts
innodb_autoinc_lock_mode=2

#MySQL parameters
default_storage_engine=InnoDB

lower_case_table_names=1
innodb_buffer_pool_size=8G


wsrep_log_conflicts
innodb_autoinc_lock_mode=2

# Node IP address

wsrep_cluster_name=dev-cluster
wsrep_node_name=pxc3
pxc_strict_mode=ENFORCING

wsrep_sst_method=xtrabackup-v2
pxc-encrypt-cluster-traffic=on


[sst]
 encrypt = 4
 ssl-ca=/etc/mysql/certs/ca.pem
 ssl-cert=/etc/mysql/certs/server-cert.pem
 ssl-key=/etc/mysql/certs/server-key.pem
1 Like
6

From node3 can you ping node2? From node2 can you ping node3?

1 Like
7

@matthewb
Iam able to ping from node3 to node2.But not not able to ping from node2 to node3.

mysql service is not running on node3,Please let me know how to bring the node3 online.

1 Like
8

Fix node2 and node3 so they can ping both ways. Once you have the networking issues sorted out, then you can try to start node3 like you started node2.

1 Like
9

@Wayne_Leutwyler
Thanks …:slight_smile:

1 Like