Not able to add secondary nodes to percona 8.0 cluster

Hello, I have a 2 centos 7 ec2 instances and I am trying to setup percona 8.0 cluster. I have the mysql service running on my first node but now I am not able to start my second node. I keep getting an error failed to establish connection. I have confirmed all the necessary ports are open.

Here’s my cnf files

    **NODE 1**
**    server-id=1**
**    datadir=/var/lib/mysql**
**    socket=/var/lib/mysql/mysql.sock**
**    log-error=/var/log/mysqld.log**
**    pid-file=/var/run/mysqld/mysqld.pid**

**    # Binary log expiration period is 604800 seconds, which equals 7 days**
**    binlog_expire_logs_seconds=604800**

**    # Path to Galera library**
**    wsrep_provider=/usr/lib64/galera4/libgalera_smm.so**

**    # Cluster connection URL contains IPs of nodes**
**    #If no IP is found, this implies that a new cluster needs to be created,**
**    #in order to do that you need to bootstrap this node**
**    wsrep_cluster_address=gcomm://172.31.57.122,172.31.51.173**

**    # In order for Galera to work correctly binlog format should be ROW**
**    binlog_format=ROW**
**    wsrep_provider_options="socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"**
**    # Slave thread to use**
**    wsrep_slave_threads=8**

**    wsrep_log_conflicts**

**    # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera**
**    innodb_autoinc_lock_mode=2**

**    # Node IP address**
**    wsrep_node_address=172.31.57.122**
**    # Cluster name**
**    wsrep_cluster_name=pxc-cluster**

**    #If wsrep_node_name is not specified,  then system hostname will be used**
**    wsrep_node_name=percona-test-1**

**    #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER**
**    pxc_strict_mode=ENFORCING**

**    # SST method**
**    wsrep_sst_method=xtrabackup-v2**
**    [sst]**
**    encrypt=4**
**    ssl-key=server-key.pem**
**    ssl-ca=ca.pem**
**    ssl-cert=server-cert.pem**

**    NODE 2**
**    [mysqld]**
**    server-id=1**
**    datadir=/var/lib/mysql**
**    socket=/var/lib/mysql/mysql.sock**
**    log-error=/var/log/mysqld.log**
**    pid-file=/var/run/mysqld/mysqld.pid**

**    # Binary log expiration period is 604800 seconds, which equals 7 days**
**    binlog_expire_logs_seconds=604800**

**    # Path to Galera library**
**    wsrep_provider=/usr/lib64/galera4/libgalera_smm.so**

**    # Cluster connection URL contains IPs of nodes**
**    #If no IP is found, this implies that a new cluster needs to be created,**
**    #in order to do that you need to bootstrap this node**
**    wsrep_cluster_address=gcomm://172.31.57.122,172.31.51.173**

**    # In order for Galera to work correctly binlog format should be ROW**
**    binlog_format=ROW**
**    wsrep_provider_options="socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"**
**    # Slave thread to use**
**    wsrep_slave_threads=8**

**    wsrep_log_conflicts**

**    # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera**
**    innodb_autoinc_lock_mode=2**

**    # Node IP address**
**    wsrep_node_address=172.31.51.173**
**    # Cluster name**
**    wsrep_cluster_name=pxc-cluster**

**    #If wsrep_node_name is not specified,  then system hostname will be used**
**    wsrep_node_name=percona-test-2**

**    #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER**
**    pxc_strict_mode=ENFORCING**

**    # SST method**
**    wsrep_sst_method=xtrabackup-v2**
**    [sst]**
**    encrypt=4**
**    ssl-key=server-key.pem**
**    ssl-ca=ca.pem**
**    ssl-cert=server-cert.pem**


ERROR LOG FROM NODE 2

2021-04-23T22:43:30.737687Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://172.31.57.122:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_paddi
This error is often caused by SSL issues. For more information, please see:
  https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-04-23T22:43:30.739126Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2021-04-23T22:43:30.739166Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2021-04-23T22:43:30.739306Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():161
2021-04-23T22:43:30.739332Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out)
2021-04-23T22:43:31.739655Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2021-04-23T22:43:31.739719Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2021-04-23T22:43:31.739885Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1754: Failed to open channel 'pxc-cluster' at 'gcomm://172.31.57.122,172.31.51.173': -110 (Connection timed out)
2021-04-23T22:43:31.739914Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
2021-04-23T22:43:31.739943Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://172.31.57.122,172.31.51.173) failed to establish connection with cluster (reason: 7)
2021-04-23T22:43:31.739971Z 0 [ERROR] [MY-010119] [Server] Aborting
2021-04-23T22:43:31.740366Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.22-13.1)  Percona XtraDB Cluster (GPL), Release rel13, Revision a48e6d5, WSREP version 26.4.3.
2021-04-23T22:43:31.740808Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
2021-04-23T22:43:31.740863Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2021-04-23T22:43:31.743572Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-04-23T22:43:31.746168Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-04-23T22:43:31.748804Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-04-23T22:43:31.748834Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
2021-04-23T22:43:31.748845Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
2021-04-23T22:43:31.748854Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
2021-04-23T22:43:31.748869Z 0 [Note] [MY-000000] [Galera] avg deps dist 0
2021-04-23T22:43:31.748880Z 0 [Note] [MY-000000] [Galera] avg cert interval 0
2021-04-23T22:43:31.748889Z 0 [Note] [MY-000000] [Galera] cert index size 0
2021-04-23T22:43:31.748933Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-04-23T22:43:31.748969Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
2021-04-23T22:43:31.748984Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2021-04-23T22:43:31.749096Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)
2021-04-23T22:43:31.749944Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...

Hi. I trid to reproduce this issue but without success.
According to your log:

2021-04-23T22:43:30.737687Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://172.31.57.122:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_paddi
This error is often caused by SSL issues. For more information, please see:
  https://per.co.na/pxc/encrypt_cluster_traffic
--------

So please visit https://per.co.na/pxc/encrypt_cluster_traffic and try to configure ssl accordingly.

1 Like