Hello, I have a 2 centos 7 ec2 instances and I am trying to setup percona 8.0 cluster. I have the mysql service running on my first node but now I am not able to start my second node. I keep getting an error failed to establish connection. I have confirmed all the necessary ports are open.
Here’s my cnf files
**NODE 1**
** server-id=1**
** datadir=/var/lib/mysql**
** socket=/var/lib/mysql/mysql.sock**
** log-error=/var/log/mysqld.log**
** pid-file=/var/run/mysqld/mysqld.pid**
** # Binary log expiration period is 604800 seconds, which equals 7 days**
** binlog_expire_logs_seconds=604800**
** # Path to Galera library**
** wsrep_provider=/usr/lib64/galera4/libgalera_smm.so**
** # Cluster connection URL contains IPs of nodes**
** #If no IP is found, this implies that a new cluster needs to be created,**
** #in order to do that you need to bootstrap this node**
** wsrep_cluster_address=gcomm://172.31.57.122,172.31.51.173**
** # In order for Galera to work correctly binlog format should be ROW**
** binlog_format=ROW**
** wsrep_provider_options="socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"**
** # Slave thread to use**
** wsrep_slave_threads=8**
** wsrep_log_conflicts**
** # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera**
** innodb_autoinc_lock_mode=2**
** # Node IP address**
** wsrep_node_address=172.31.57.122**
** # Cluster name**
** wsrep_cluster_name=pxc-cluster**
** #If wsrep_node_name is not specified, then system hostname will be used**
** wsrep_node_name=percona-test-1**
** #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER**
** pxc_strict_mode=ENFORCING**
** # SST method**
** wsrep_sst_method=xtrabackup-v2**
** [sst]**
** encrypt=4**
** ssl-key=server-key.pem**
** ssl-ca=ca.pem**
** ssl-cert=server-cert.pem**
** NODE 2**
** [mysqld]**
** server-id=1**
** datadir=/var/lib/mysql**
** socket=/var/lib/mysql/mysql.sock**
** log-error=/var/log/mysqld.log**
** pid-file=/var/run/mysqld/mysqld.pid**
** # Binary log expiration period is 604800 seconds, which equals 7 days**
** binlog_expire_logs_seconds=604800**
** # Path to Galera library**
** wsrep_provider=/usr/lib64/galera4/libgalera_smm.so**
** # Cluster connection URL contains IPs of nodes**
** #If no IP is found, this implies that a new cluster needs to be created,**
** #in order to do that you need to bootstrap this node**
** wsrep_cluster_address=gcomm://172.31.57.122,172.31.51.173**
** # In order for Galera to work correctly binlog format should be ROW**
** binlog_format=ROW**
** wsrep_provider_options="socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"**
** # Slave thread to use**
** wsrep_slave_threads=8**
** wsrep_log_conflicts**
** # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera**
** innodb_autoinc_lock_mode=2**
** # Node IP address**
** wsrep_node_address=172.31.51.173**
** # Cluster name**
** wsrep_cluster_name=pxc-cluster**
** #If wsrep_node_name is not specified, then system hostname will be used**
** wsrep_node_name=percona-test-2**
** #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER**
** pxc_strict_mode=ENFORCING**
** # SST method**
** wsrep_sst_method=xtrabackup-v2**
** [sst]**
** encrypt=4**
** ssl-key=server-key.pem**
** ssl-ca=ca.pem**
** ssl-cert=server-cert.pem**
ERROR LOG FROM NODE 2
2021-04-23T22:43:30.737687Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://172.31.57.122:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_paddi
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-04-23T22:43:30.739126Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2021-04-23T22:43:30.739166Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2021-04-23T22:43:30.739306Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():161
2021-04-23T22:43:30.739332Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out)
2021-04-23T22:43:31.739655Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2021-04-23T22:43:31.739719Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2021-04-23T22:43:31.739885Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1754: Failed to open channel 'pxc-cluster' at 'gcomm://172.31.57.122,172.31.51.173': -110 (Connection timed out)
2021-04-23T22:43:31.739914Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
2021-04-23T22:43:31.739943Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://172.31.57.122,172.31.51.173) failed to establish connection with cluster (reason: 7)
2021-04-23T22:43:31.739971Z 0 [ERROR] [MY-010119] [Server] Aborting
2021-04-23T22:43:31.740366Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.22-13.1) Percona XtraDB Cluster (GPL), Release rel13, Revision a48e6d5, WSREP version 26.4.3.
2021-04-23T22:43:31.740808Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
2021-04-23T22:43:31.740863Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2021-04-23T22:43:31.743572Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-04-23T22:43:31.746168Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-04-23T22:43:31.748804Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-04-23T22:43:31.748834Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
2021-04-23T22:43:31.748845Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
2021-04-23T22:43:31.748854Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
2021-04-23T22:43:31.748869Z 0 [Note] [MY-000000] [Galera] avg deps dist 0
2021-04-23T22:43:31.748880Z 0 [Note] [MY-000000] [Galera] avg cert interval 0
2021-04-23T22:43:31.748889Z 0 [Note] [MY-000000] [Galera] cert index size 0
2021-04-23T22:43:31.748933Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-04-23T22:43:31.748969Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
2021-04-23T22:43:31.748984Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2021-04-23T22:43:31.749096Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)
2021-04-23T22:43:31.749944Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...