# Can't start node 2 and 3 on ubuntu 20.04 **URL:** https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150 **Category:** Percona XtraDB Cluster 8.x **Tags:** percona **Created:** [August 25, 2022, 6:59am UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150 "2022-08-25T06:59:21Z") **Posts on this page:** 12 **Page:** 1
### Author: ![hieu\_nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/hieu_nguyen/32/6616_2.png) [@hieu\_nguyen](https://forums.percona.com/u/hieu_nguyen) #### Post date: [August 25, 2022, 6:59am UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/1 "2022-08-25T06:59:21Z")
i installed percona cluster under :[Redirecting](https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/apt.html#apt) i started systemctl start mysql@bootstrap.service on node 1 it worked fine until i start node 2 and 3 there is an error ![Screenshot_213](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/b/bc8e1221dbd2d7a459eea9a8e95a5941ab5a498d.png) I have configured the port on iptables on all 3 nodes ![Screenshot_214](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/d/daee380093e8ce458b610b21deb910e67a725329.png) here is my configuration file in /etc/mysql/my.cnf ```auto [mysqld] datadir=/var/lib/mysql user=mysql # Path to Galera library wsrep_provider=/usr/lib/libgalera_smm.so # Cluster connection URL contains IPs of node#1, node#2 and node#3 wsrep_cluster_address=gcomm://192.168.254.1,192.168.254.2,192.168.254.3 # In order for Galera to work correctly binlog format should be ROW binlog_format=ROW # Using the MyISAM storage engine is not recommended default_storage_engine=InnoDB # This InnoDB autoincrement locking mode is a requirement for Galera innodb_autoinc_lock_mode=2 # Node #3 address wsrep_node_address=192.168.254.1 # Cluster name wsrep_cluster_name=my_ubuntu_cluster # SST method wsrep_sst_method=xtrabackup-v2 ``` ---
### Author: ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb) #### Post date: [August 25, 2022, 2:11pm UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/2 "2022-08-25T14:11:51Z")
You need to copy the SSL certificates from node1 over to node2/3. By default, PXC8 uses SSL everywhere. If you don’t copy the certs, then the servers cannot create encrypted connections. ---
### Author: ![hieu\_nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/hieu_nguyen/32/6616_2.png) [@hieu\_nguyen](https://forums.percona.com/u/hieu_nguyen) #### Post date: [August 25, 2022, 2:23pm UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/3 "2022-08-25T14:23:09Z")
this is node 1 , i start mysql@boostrap ```auto # Template my.cnf for PXC # Edit to your requirements. [client] socket=/var/run/mysqld/mysqld.sock [mysqld] server-id=1 datadir=/var/lib/mysql socket=/var/run/mysqld/mysqld.sock log-error=/var/log/mysql/error.log pid-file=/var/run/mysqld/mysqld.pid # Binary log expiration period is 604800 seconds, which equals 7 days binlog_expire_logs_seconds=604800 ######## wsrep ############### # Path to Galera library wsrep_provider=/usr/lib/galera4/libgalera_smm.so # Cluster connection URL contains IPs of nodes #If no IP is found, this implies that a new cluster needs to be created, #in order to do that you need to bootstrap this node wsrep_cluster_address=gcomm://192.168.254.1,192.168.254.2,192.168.254.3 # In order for Galera to work correctly binlog format should be ROW binlog_format=ROW # Slave thread to use wsrep_slave_threads=8 wsrep_log_conflicts default_storage_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb_autoinc_lock_mode=2 # Node IP address wsrep_node_address=192.168.254.1 # Cluster name wsrep_cluster_name=pxc-cluster #If wsrep_node_name is not specified, then system hostname will be used wsrep_node_name=pxc1 #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER pxc_strict_mode=ENFORCING # SST method wsrep_sst_method=xtrabackup-v2 wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem” [sst] encrypt=4 ssl-key=server-key.pem ssl-ca=ca.pem ssl-cert=server-cert.pem ``` this is node 2: ```auto # Template my.cnf for PXC # Edit to your requirements. [client] socket=/var/run/mysqld/mysqld.sock [mysqld] server-id=1 datadir=/var/lib/mysql socket=/var/run/mysqld/mysqld.sock log-error=/var/log/mysql/error.log pid-file=/var/run/mysqld/mysqld.pid # Binary log expiration period is 604800 seconds, which equals 7 days binlog_expire_logs_seconds=604800 ######## wsrep ############### # Path to Galera library wsrep_provider=/usr/lib/galera4/libgalera_smm.so # Cluster connection URL contains IPs of nodes #If no IP is found, this implies that a new cluster needs to be created, #in order to do that you need to bootstrap this node wsrep_cluster_address=gcomm://192.168.254.1,192.168.254.2,192.168.254.3 # In order for Galera to work correctly binlog format should be ROW binlog_format=ROW # Slave thread to use wsrep_slave_threads=8 wsrep_log_conflicts default_storage_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb_autoinc_lock_mode=2 # Node IP address wsrep_node_address=192.168.254.2 # Cluster name wsrep_cluster_name=pxc-cluster #If wsrep_node_name is not specified, then system hostname will be used wsrep_node_name=pxc2 #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER pxc_strict_mode=ENFORCING # SST method wsrep_sst_method=xtrabackup-v2 wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem” [sst] encrypt=4 ssl-key=server-key.pem ssl-ca=ca.pem ssl-cert=server-cert.pem ``` it still error ```auto 2022-08-25T14:11:05.976491Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50446S), skipping check 2022-08-25T14:11:35.488236Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0 2022-08-25T14:11:35.488315Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view ((empty)) 2022-08-25T14:11:35.488460Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out) at gcomm/src/pc.cpp:connect():161 2022-08-25T14:11:35.488512Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out) 2022-08-25T14:11:36.488839Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread 2022-08-25T14:11:36.488901Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread 2022-08-25T14:11:36.489069Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'pxc-cluster' at 'gcomm://192.168.254.132,192.168.254.133,192.168.254.136': -110 (Connection timed out) 2022-08-25T14:11:36.489105Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out 2022-08-25T14:11:36.489133Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.254.132,192.168.254.133,192.168.254.136) failed to establish connection with cluster (reason: 7) 2022-08-25T14:11:36.489158Z 0 [ERROR] [MY-010119] [Server] Aborting 2022-08-25T14:11:36.489474Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.28-19.1) Percona XtraDB Cluster (GPL), Release rel19, Revision f544540, WSREP version 26.4.3. 2022-08-25T14:11:36.490669Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED 2022-08-25T14:11:36.490732Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2022-08-25T14:11:36.494683Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-08-25T14:11:36.497005Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-08-25T14:11:36.499293Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-08-25T14:11:36.499331Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0 2022-08-25T14:11:36.499348Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0 2022-08-25T14:11:36.499363Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0 2022-08-25T14:11:36.499385Z 0 [Note] [MY-000000] [Galera] avg deps dist 0 2022-08-25T14:11:36.499402Z 0 [Note] [MY-000000] [Galera] avg cert interval 0 2022-08-25T14:11:36.499418Z 0 [Note] [MY-000000] [Galera] cert index size 0 2022-08-25T14:11:36.499503Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed. 2022-08-25T14:11:36.499650Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0 2022-08-25T14:11:36.499681Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2022-08-25T14:11:36.499909Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0) 2022-08-25T14:11:36.500417Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk... ``` ---
### Author: ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb) #### Post date: [August 25, 2022, 3:19pm UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/4 "2022-08-25T15:19:26Z")
> 2022-08-25T14:11:36.489069Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs\_open():1811: Failed to open channel ‘pxc-cluster’ at ‘gcomm://192.168.254.132,192.168.254.133,192.168.254.136’: -110 (Connection timed out) What happened here? These are the wrong IP addresses according to your config. ---
### Author: ![hieu\_nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/hieu_nguyen/32/6616_2.png) [@hieu\_nguyen](https://forums.percona.com/u/hieu_nguyen) #### Post date: [August 25, 2022, 3:28pm UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/5 "2022-08-25T15:28:10Z")
these addresses are exactly my addresses ---
### Author: ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb) #### Post date: [August 25, 2022, 4:30pm UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/6 "2022-08-25T16:30:54Z")
Your error log says this: ``` Failed to open channel 'pxc-cluster' at gcomm://192.168.254.132,192.168.254.133,192.168.254.136 ``` But your config says this: ``` wsrep_cluster_address=gcomm://192.168.254.1,192.168.254.2,192.168.254.3 wsrep_node_address=192.168.254.2 ``` You have something strange with your config and your IP addresses. ---
### Author: ![hieu\_nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/hieu_nguyen/32/6616_2.png) [@hieu\_nguyen](https://forums.percona.com/u/hieu_nguyen) #### Post date: [August 26, 2022, 1:24am UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/7 "2022-08-26T01:24:25Z")
I corrected the ip address when I posted it ---
### Author: ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb) #### Post date: [August 26, 2022, 1:47am UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/8 "2022-08-26T01:47:40Z")
I’m confused. Please post your actual config file and actual error messages from node2. ---
### Author: ![hieu\_nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/hieu_nguyen/32/6616_2.png) [@hieu\_nguyen](https://forums.percona.com/u/hieu_nguyen) #### Post date: [August 26, 2022, 1:55am UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/9 "2022-08-26T01:55:34Z")
this is the actual configuration on node 2 ```auto [client] socket=/var/run/mysqld/mysqld.sock [mysqld] server-id=1 datadir=/var/lib/mysql socket=/var/run/mysqld/mysqld.sock log-error=/var/log/mysql/error.log pid-file=/var/run/mysqld/mysqld.pid # Binary log expiration period is 604800 seconds, which equals 7 days binlog_expire_logs_seconds=604800 ######## wsrep ############### # Path to Galera library wsrep_provider=/usr/lib/galera4/libgalera_smm.so # Cluster connection URL contains IPs of nodes #If no IP is found, this implies that a new cluster needs to be created, #in order to do that you need to bootstrap this node wsrep_cluster_address=gcomm://192.168.254.132,192.168.254.133,192.168.254.136 # In order for Galera to work correctly binlog format should be ROW binlog_format=ROW # Slave thread to use wsrep_slave_threads=8 wsrep_log_conflicts default_storage_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb_autoinc_lock_mode=2 # Node IP address wsrep_node_address=192.168.254.133 # Cluster name wsrep_cluster_name=pxc-cluster #If wsrep_node_name is not specified, then system hostname will be used wsrep_node_name=pxc2 #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER pxc_strict_mode=ENFORCING # SST method wsrep_sst_method=xtrabackup-v2 wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem” [sst] encrypt=4 ssl-key=server-key.pem ssl-ca=ca.pem ssl-cert=server-cert.pem ``` and here is the error message on it, on all nodes i have apparmor and selinux disabled ```auto 2022-08-26T01:54:18.207588Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50301S), skipping check 2022-08-26T01:54:47.717681Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0 2022-08-26T01:54:47.717754Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view ((empty)) 2022-08-26T01:54:47.717875Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out) at gcomm/src/pc.cpp:connect():161 2022-08-26T01:54:47.717911Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out) 2022-08-26T01:54:48.718154Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread 2022-08-26T01:54:48.718217Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread 2022-08-26T01:54:48.718492Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'pxc-cluster' at 'gcomm://192.168.254.132,192.168.254.133,192.168.254.136': -110 (Connection timed out) 2022-08-26T01:54:48.718530Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out 2022-08-26T01:54:48.718558Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.254.132,192.168.254.133,192.168.254.136) failed to establish connection with cluster (reason: 7) 2022-08-26T01:54:48.718601Z 0 [ERROR] [MY-010119] [Server] Aborting 2022-08-26T01:54:48.719032Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.28-19.1) Percona XtraDB Cluster (GPL), Release rel19, Revision f544540, WSREP version 26.4.3. 2022-08-26T01:54:48.721115Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED 2022-08-26T01:54:48.721189Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2022-08-26T01:54:48.723535Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-08-26T01:54:48.725741Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-08-26T01:54:48.729707Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-08-26T01:54:48.729756Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0 2022-08-26T01:54:48.729776Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0 2022-08-26T01:54:48.729791Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0 2022-08-26T01:54:48.729813Z 0 [Note] [MY-000000] [Galera] avg deps dist 0 2022-08-26T01:54:48.729830Z 0 [Note] [MY-000000] [Galera] avg cert interval 0 2022-08-26T01:54:48.729845Z 0 [Note] [MY-000000] [Galera] cert index size 0 2022-08-26T01:54:48.729915Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed. 2022-08-26T01:54:48.729961Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0 2022-08-26T01:54:48.729986Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2022-08-26T01:54:48.730165Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0) 2022-08-26T01:54:48.827544Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk... ``` ---
### Author: ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb) #### Post date: [August 26, 2022, 4:41am UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/10 "2022-08-26T04:41:18Z")
You’re missing `wsrep_provider_options=socket.ssl=ON` SSL must be explicitly turned on. ---
### Author: ![hieu\_nguyen](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/hieu_nguyen/32/6616_2.png) [@hieu\_nguyen](https://forums.percona.com/u/hieu_nguyen) #### Post date: [August 26, 2022, 6:54am UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/11 "2022-08-26T06:54:03Z")
i added the line wsrep\_provider\_options=socket.ssl=ON on my node 1 and node 2 it should look like this ```auto [client] socket=/var/run/mysqld/mysqld.sock [mysqld] server-id=1 datadir=/var/lib/mysql socket=/var/run/mysqld/mysqld.sock log-error=/var/log/mysql/error.log pid-file=/var/run/mysqld/mysqld.pid # Binary log expiration period is 604800 seconds, which equals 7 days binlog_expire_logs_seconds=604800 ######## wsrep ############### # Path to Galera library wsrep_provider=/usr/lib/galera4/libgalera_smm.so # Cluster connection URL contains IPs of nodes #If no IP is found, this implies that a new cluster needs to be created, #in order to do that you need to bootstrap this node wsrep_cluster_address=gcomm://192.168.254.132,192.168.254.133,192.168.254.136 # In order for Galera to work correctly binlog format should be ROW binlog_format=ROW # Slave thread to use wsrep_slave_threads=8 wsrep_log_conflicts default_storage_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb_autoinc_lock_mode=2 # Node IP address wsrep_node_address=192.168.254.133 # Cluster name wsrep_cluster_name=pxc-cluster #If wsrep_node_name is not specified, then system hostname will be used wsrep_node_name=pxc2 #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER pxc_strict_mode=ENFORCING # SST method wsrep_sst_method=xtrabackup-v2 wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem” wsrep_provider_options=socket.ssl=ON [sst] encrypt=4 ssl-key=server-key.pem ssl-ca=ca.pem ssl-cert=server-cert.pem ``` The result is still the same ---
### Author: ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb) #### Post date: [August 26, 2022, 11:53am UTC](https://forums.percona.com/t/cant-start-node-2-and-3-on-ubuntu-20-04/17150/12 "2022-08-26T11:53:47Z")
Sorry, I thought you might have understood that all wsrep\_provider\_options should be on the same variable. Please put them all together. Also, remove the `[sst]` section completely. If you want to do PXC without SSL, which might be easier but less secure, just remove all SSL options everywhere and set `pxc-encrypt-cluster-traffic=OFF`