Can't start node 2 and 3 on ubuntu 20.04

hieu_nguyen · August 25, 2022, 6:59am

i installed percona cluster under :Redirecting
i started systemctl start mysql@bootstrap.service on node 1 it worked fine
until i start node 2 and 3 there is an error

I have configured the port on iptables on all 3 nodes

here is my configuration file in /etc/mysql/my.cnf

[mysqld]

datadir=/var/lib/mysql
user=mysql

# Path to Galera library
wsrep_provider=/usr/lib/libgalera_smm.so

# Cluster connection URL contains IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://192.168.254.1,192.168.254.2,192.168.254.3

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# Using the MyISAM storage engine is not recommended
default_storage_engine=InnoDB

# This InnoDB autoincrement locking mode is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #3 address
wsrep_node_address=192.168.254.1

# Cluster name
wsrep_cluster_name=my_ubuntu_cluster

# SST method
wsrep_sst_method=xtrabackup-v2

matthewb · August 25, 2022, 2:11pm

You need to copy the SSL certificates from node1 over to node2/3. By default, PXC8 uses SSL everywhere. If you don’t copy the certs, then the servers cannot create encrypted connections.

hieu_nguyen · August 25, 2022, 2:23pm

this is node 1 , i start mysql@boostrap

# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock

[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid

# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800

######## wsrep ###############
# Path to Galera library

wsrep_provider=/usr/lib/galera4/libgalera_smm.so

# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.254.1,192.168.254.2,192.168.254.3

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# Slave thread to use
wsrep_slave_threads=8

wsrep_log_conflicts

default_storage_engine=InnoDB
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node IP address
wsrep_node_address=192.168.254.1
# Cluster name
wsrep_cluster_name=pxc-cluster

#If wsrep_node_name is not specified,  then system hostname will be used
wsrep_node_name=pxc1

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

# SST method
wsrep_sst_method=xtrabackup-v2

wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”

[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem

this is node 2:

# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock

[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid

# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800

######## wsrep ###############
# Path to Galera library

wsrep_provider=/usr/lib/galera4/libgalera_smm.so

# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.254.1,192.168.254.2,192.168.254.3

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# Slave thread to use
wsrep_slave_threads=8

wsrep_log_conflicts

default_storage_engine=InnoDB
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node IP address
wsrep_node_address=192.168.254.2
# Cluster name
wsrep_cluster_name=pxc-cluster

#If wsrep_node_name is not specified,  then system hostname will be used
wsrep_node_name=pxc2

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

# SST method
wsrep_sst_method=xtrabackup-v2

wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”

[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem

it still error

2022-08-25T14:11:05.976491Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50446S), skipping check
2022-08-25T14:11:35.488236Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-08-25T14:11:35.488315Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-08-25T14:11:35.488460Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():161
2022-08-25T14:11:35.488512Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)
2022-08-25T14:11:36.488839Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-08-25T14:11:36.488901Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-08-25T14:11:36.489069Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'pxc-cluster' at 'gcomm://192.168.254.132,192.168.254.133,192.168.254.136': -110 (Connection timed out)
2022-08-25T14:11:36.489105Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
2022-08-25T14:11:36.489133Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.254.132,192.168.254.133,192.168.254.136) failed to establish connection with cluster (reason: 7)
2022-08-25T14:11:36.489158Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-08-25T14:11:36.489474Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.28-19.1)  Percona XtraDB Cluster (GPL), Release rel19, Revision f544540, WSREP version 26.4.3.
2022-08-25T14:11:36.490669Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
2022-08-25T14:11:36.490732Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2022-08-25T14:11:36.494683Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-08-25T14:11:36.497005Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-08-25T14:11:36.499293Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-08-25T14:11:36.499331Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
2022-08-25T14:11:36.499348Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
2022-08-25T14:11:36.499363Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
2022-08-25T14:11:36.499385Z 0 [Note] [MY-000000] [Galera] avg deps dist 0
2022-08-25T14:11:36.499402Z 0 [Note] [MY-000000] [Galera] avg cert interval 0
2022-08-25T14:11:36.499418Z 0 [Note] [MY-000000] [Galera] cert index size 0
2022-08-25T14:11:36.499503Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-08-25T14:11:36.499650Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
2022-08-25T14:11:36.499681Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2022-08-25T14:11:36.499909Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)
2022-08-25T14:11:36.500417Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...

matthewb · August 25, 2022, 3:19pm

2022-08-25T14:11:36.489069Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel ‘pxc-cluster’ at ‘gcomm://192.168.254.132,192.168.254.133,192.168.254.136’: -110 (Connection timed out)

What happened here? These are the wrong IP addresses according to your config.

hieu_nguyen · August 25, 2022, 3:28pm

these addresses are exactly my addresses

matthewb · August 25, 2022, 4:30pm

Your error log says this:

Failed to open channel 'pxc-cluster' at gcomm://192.168.254.132,192.168.254.133,192.168.254.136

But your config says this:

wsrep_cluster_address=gcomm://192.168.254.1,192.168.254.2,192.168.254.3
wsrep_node_address=192.168.254.2

You have something strange with your config and your IP addresses.

hieu_nguyen · August 26, 2022, 1:24am

I corrected the ip address when I posted it

matthewb · August 26, 2022, 1:47am

I’m confused. Please post your actual config file and actual error messages from node2.

hieu_nguyen · August 26, 2022, 1:55am

this is the actual configuration on node 2

[client]
socket=/var/run/mysqld/mysqld.sock

[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid

# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800

######## wsrep ###############
# Path to Galera library

wsrep_provider=/usr/lib/galera4/libgalera_smm.so

# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.254.132,192.168.254.133,192.168.254.136

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# Slave thread to use
wsrep_slave_threads=8

wsrep_log_conflicts

default_storage_engine=InnoDB
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node IP address
wsrep_node_address=192.168.254.133
# Cluster name
wsrep_cluster_name=pxc-cluster

#If wsrep_node_name is not specified,  then system hostname will be used
wsrep_node_name=pxc2

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

# SST method
wsrep_sst_method=xtrabackup-v2

wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”

[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem

and here is the error message on it, on all nodes i have apparmor and selinux disabled

2022-08-26T01:54:18.207588Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50301S), skipping check
2022-08-26T01:54:47.717681Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-08-26T01:54:47.717754Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-08-26T01:54:47.717875Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():161
2022-08-26T01:54:47.717911Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)
2022-08-26T01:54:48.718154Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-08-26T01:54:48.718217Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-08-26T01:54:48.718492Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'pxc-cluster' at 'gcomm://192.168.254.132,192.168.254.133,192.168.254.136': -110 (Connection timed out)
2022-08-26T01:54:48.718530Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
2022-08-26T01:54:48.718558Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.254.132,192.168.254.133,192.168.254.136) failed to establish connection with cluster (reason: 7)
2022-08-26T01:54:48.718601Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-08-26T01:54:48.719032Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.28-19.1)  Percona XtraDB Cluster (GPL), Release rel19, Revision f544540, WSREP version 26.4.3.
2022-08-26T01:54:48.721115Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
2022-08-26T01:54:48.721189Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2022-08-26T01:54:48.723535Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-08-26T01:54:48.725741Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-08-26T01:54:48.729707Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-08-26T01:54:48.729756Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
2022-08-26T01:54:48.729776Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
2022-08-26T01:54:48.729791Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
2022-08-26T01:54:48.729813Z 0 [Note] [MY-000000] [Galera] avg deps dist 0
2022-08-26T01:54:48.729830Z 0 [Note] [MY-000000] [Galera] avg cert interval 0
2022-08-26T01:54:48.729845Z 0 [Note] [MY-000000] [Galera] cert index size 0
2022-08-26T01:54:48.729915Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-08-26T01:54:48.729961Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
2022-08-26T01:54:48.729986Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2022-08-26T01:54:48.730165Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)
2022-08-26T01:54:48.827544Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...

matthewb · August 26, 2022, 4:41am

You’re missing wsrep_provider_options=socket.ssl=ON SSL must be explicitly turned on.

hieu_nguyen · August 26, 2022, 6:54am

i added the line wsrep_provider_options=socket.ssl=ON on my node 1 and node 2 it should look like this

[client]
socket=/var/run/mysqld/mysqld.sock

[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid

# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800

######## wsrep ###############
# Path to Galera library

wsrep_provider=/usr/lib/galera4/libgalera_smm.so

# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.254.132,192.168.254.133,192.168.254.136

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# Slave thread to use
wsrep_slave_threads=8

wsrep_log_conflicts

default_storage_engine=InnoDB
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node IP address
wsrep_node_address=192.168.254.133
# Cluster name
wsrep_cluster_name=pxc-cluster

#If wsrep_node_name is not specified,  then system hostname will be used
wsrep_node_name=pxc2

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

# SST method
wsrep_sst_method=xtrabackup-v2

wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”
wsrep_provider_options=socket.ssl=ON

[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem

The result is still the same

matthewb · August 26, 2022, 11:53am

Sorry, I thought you might have understood that all wsrep_provider_options should be on the same variable. Please put them all together. Also, remove the [sst] section completely.

If you want to do PXC without SSL, which might be easier but less secure, just remove all SSL options everywhere and set pxc-encrypt-cluster-traffic=OFF

Topic		Replies	Views
Percona XtraDB 8.0.30-22 - Nodes 2 and 3 refuse to connect Percona XtraDB Cluster 8.x	17	1400	January 25, 2023
can't start percona cluster 8.019 for the first time. Percona XtraDB Cluster 8.x	9	1611	August 10, 2020
Can't start node2 and node3 mySQL server Percona XtraDB Cluster 5.x	4	2434	May 7, 2015
The master node starts normally, and other nodes cannot join Percona XtraDB Cluster 8.x	10	784	October 13, 2023
Clustering 3 node .Why this error 'Percona XtraDB Cluster Node is not synced or non-PRIM' .help me Percona XtraDB Cluster 5.x	5	4148	November 12, 2015

Can't start node 2 and 3 on ubuntu 20.04

Related topics