Hi,
First of all, I followed the steps in “Redirecting” web page. I installed “percona-release_latest.generic_all.deb” on my Ubuntu 20.04 LTS server.
I got some errors while adding nodes to the cluster. I do not use encryption or SSL certificate (I am just testing). Can you help me to fix this issue?
The firewall is not enabled and necessary ports are open. I did not install “sudo apt-get install percona-xtrabackup-80 -y” I only installed “sudo apt install -y percona-xtradb-cluster” latest version.
I did not touch file permissions at all. The 2 servers are created from scratch.
Additional question, should I set “pxc-encrypt-cluster-traffic=OFF” under [mysqld] section
On node1, I stopped the mysql service “systemctl stop mysql”
And I configured the " /etc/mysql/mysql.conf.d/mysqld.cnf" file; Content of it;
# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800
######## wsrep ###############
# Path to Galera library
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.1.201,192.168.1.202,192.168.1.203
# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
# Slave thread to use
wsrep_slave_threads=8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2
# Node IP address
#wsrep_node_address=192.168.70.63
# Cluster name
wsrep_cluster_name=pxc-cluster
#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=db01
wsrep_node_address=192.168.1.201
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING
# SST method
wsrep_sst_method=xtrabackup-v2
After configuring the primary node 1, I configured node2. Again I stopped mysql service. And configured “/etc/mysql/mysql.conf.d/mysqld.cnf” file. Content of it;
# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800
######## wsrep ###############
# Path to Galera library
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.1.201,192.168.1.202,192.168.1.203
# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
# Slave thread to use
wsrep_slave_threads=8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2
# Node IP address
#wsrep_node_address=192.168.70.63
# Cluster name
wsrep_cluster_name=pxc-cluster
#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=db02
wsrep_node_address=192.168.1.202
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING
# SST method
wsrep_sst_method=xtrabackup-v2
After configuring both nodes, I ran “systemctl start mysql@bootstrap.service” command on node1. It works and I was able to see result of “show status like ‘wsrep%’;”
Then, I configured node2, I ran “systemctl start mysql”. But it gave an error “Job for mysql.service failed because the control process exited with error code.”
“See “systemctl status mysql.service” and “journalctl -xe” for details.”
Partial Content of error file on node1 (bootsrapped)
2021-12-19T09:54:40.114714Z 0 [Note] [MY-000000] [WSREP] Starting replication
2021-12-19T09:54:40.114880Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1
2021-12-19T09:54:40.115086Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 750dbe9c-60af-11ec-8912-aa183b7e38cf:2
2021-12-19T09:54:40.115355Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2021-12-19T09:54:40.115675Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2021-12-19T09:54:40.115885Z 0 [Note] [MY-000000] [Galera] initializing ssl context
2021-12-19T09:54:40.116329Z 0 [Note] [MY-000000] [Galera] backend: asio
2021-12-19T09:54:40.116600Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2021-12-19T09:54:40.116899Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2021-12-19T09:54:40.117148Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2021-12-19T09:54:40.117695Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2021-12-19T09:54:40.118160Z 0 [Note] [MY-000000] [Galera] (aee2196b-bd60, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2021-12-19T09:54:40.118416Z 0 [Note] [MY-000000] [Galera] (aee2196b-bd60, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2021-12-19T09:54:40.119002Z 0 [Note] [MY-000000] [Galera] EVS version 1
2021-12-19T09:54:40.119366Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group 'pxc-cluster'
2021-12-19T09:54:40.119616Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery
2021-12-19T09:54:40.121536Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2021-12-19T09:54:40.121788Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2021-12-19T09:54:40.122068Z 0 [Note] [MY-000000] [Galera] Node aee2196b-bd60 state primary
2021-12-19T09:54:40.122326Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,aee2196b-bd60,1)
memb {
aee2196b-bd60,0
}
joined {
}
left {
}
partitioned {
}
)
2021-12-19T09:54:40.122574Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2021-12-19T09:54:40.293168Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://192.168.1.201:4567
2021-12-19T09:54:40.294274Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x55e3606721a0
2021-12-19T09:54:40.295767Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://192.168.1.202:4567
2021-12-19T09:54:40.296867Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x55e360672080
2021-12-19T09:54:40.298066Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://192.168.1.203:4567
2021-12-19T09:54:40.299176Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x55e360783e50
2021-12-19T09:54:40.300613Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2021-12-19T09:54:40.302497Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2021-12-19T09:54:40.303770Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2021-12-19T09:54:40.304586Z 0 [Note] [MY-000000] [Galera] Opened channel 'pxc-cluster'
2021-12-19T09:54:40.305722Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2021-12-19T09:54:40.307246Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: aeff04b8-60b1-11ec-8dce-0e762b2e6b71
2021-12-19T09:54:40.308727Z 1 [Note] [MY-000000] [WSREP] Starting applier thread 1
2021-12-19T09:54:40.309427Z 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2
2021-12-19T09:54:40.310121Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: aeff04b8-60b1-11ec-8dce-0e762b2e6b71
2021-12-19T09:54:40.310833Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: aeff04b8-60b1-11ec-8dce-0e762b2e6b71 from 0 (db01)
2021-12-19T09:54:40.311604Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 0,
members = 1/1 (primary/total),
act_id = 2,
last_appl. = 2,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 750dbe9c-60af-11ec-8912-aa183b7e38cf
2021-12-19T09:54:40.312360Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2021-12-19T09:54:40.313027Z 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (3)
2021-12-19T09:54:40.313752Z 1 [Note] [MY-000000] [Galera] ####### processing CC 3, local, ordered
2021-12-19T09:54:40.314355Z 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 2 upto current CC event 3 upto:2
2021-12-19T09:54:40.314932Z 1 [Note] [MY-000000] [Galera] Drain monitors from 2 up to 2
2021-12-19T09:54:40.315615Z 1 [Note] [MY-000000] [Galera] Process first view: 750dbe9c-60af-11ec-8912-aa183b7e38cf my uuid: aee2196b-60b1-11ec-bd60-731b3c3d900f
2021-12-19T09:54:40.316250Z 1 [Note] [MY-000000] [Galera] Server db01 connected to cluster at position 750dbe9c-60af-11ec-8912-aa183b7e38cf:3 with ID aee2196b-60b1-11ec-bd60-731b3c3d900f
2021-12-19T09:54:40.316839Z 1 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2021-12-19T09:54:40.317497Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-12-19T09:54:40.318184Z 0 [Note] [MY-000000] [Galera] Member 0.0 (db01) synced with group.
2021-12-19T09:54:40.318756Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 3)
2021-12-19T09:54:40.319373Z 1 [Note] [MY-000000] [Galera] ####### My UUID: aee2196b-60b1-11ec-bd60-731b3c3d900f
2021-12-19T09:54:40.319920Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no
2021-12-19T09:54:40.320525Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-12-19T09:54:40.321127Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2021-12-19T09:54:40.321686Z 1 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2021-12-19T09:54:40.322271Z 1 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 3
2021-12-19T09:54:40.322862Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-12-19T09:54:40.647932Z 1 [Note] [MY-000000] [Galera] ================================================
View:
id: 750dbe9c-60af-11ec-8912-aa183b7e38cf:3
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: aee2196b-60b1-11ec-bd60-731b3c3d900f, db01
=================================================
2021-12-19T09:54:40.649850Z 1 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2021-12-19T09:54:40.651747Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-12-19T09:54:40.653530Z 1 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing
2021-12-19T09:54:40.654748Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-12-19T09:54:40.658941Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
2021-12-19T09:54:40.667829Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-12-19T09:54:41.668048Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-12-19T09:54:41.939467Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2021-12-19T09:54:41.960682Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2021-12-19T09:54:41.974165Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-12-19T09:54:42.066164Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-12-19T09:54:42.066783Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-12-19T09:54:42.087981Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2021-12-19T09:54:42.088701Z 0 [Note] [MY-000000] [Galera] Server initialized
2021-12-19T09:54:42.089334Z 0 [Note] [MY-000000] [WSREP] Server status change initializing -> initialized
2021-12-19T09:54:42.089958Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-12-19T09:54:42.090622Z 1 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
2021-12-19T09:54:42.104112Z 10 [Note] [MY-000000] [WSREP] Starting applier thread 10
2021-12-19T09:54:42.104262Z 11 [Note] [MY-000000] [WSREP] Starting applier thread 11
2021-12-19T09:54:42.104386Z 12 [Note] [MY-000000] [WSREP] Starting applier thread 12
2021-12-19T09:54:42.104488Z 13 [Note] [MY-000000] [WSREP] Starting applier thread 13
2021-12-19T09:54:42.104725Z 14 [Note] [MY-000000] [WSREP] Starting applier thread 14
2021-12-19T09:54:42.104867Z 15 [Note] [MY-000000] [WSREP] Starting applier thread 15
2021-12-19T09:54:42.105444Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.25-15.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel15, Revision 8638bb0, WSREP version 26.4.3.
2021-12-19T09:54:42.105627Z 17 [Note] [MY-000000] [WSREP] Starting applier thread 17
2021-12-19T09:54:42.116203Z 8 [Note] [MY-000000] [WSREP] Recovered cluster id 750dbe9c-60af-11ec-8912-aa183b7e38cf
2021-12-19T09:54:42.140283Z 1 [Note] [MY-000000] [WSREP] Server status change initialized -> joined
2021-12-19T09:54:42.140308Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-12-19T09:54:42.140325Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-12-19T09:54:42.158625Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 3
2021-12-19T09:54:42.158651Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 3
2021-12-19T09:54:42.158663Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2021-12-19T09:54:42.158687Z 1 [Note] [MY-000000] [Galera] Server db01 synced with group
2021-12-19T09:54:42.158697Z 1 [Note] [MY-000000] [WSREP] Server status change joined -> synced
2021-12-19T09:54:42.158705Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2021-12-19T09:54:42.158712Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
Partial Content of error file on node2
2021-12-19T09:54:52.325378Z 0 [Note] [MY-000000] [WSREP] Starting replication
2021-12-19T09:54:52.325525Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0
2021-12-19T09:54:52.325659Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to ed41a615-60ae-11ec-8155-7b1ae29c3096:2
2021-12-19T09:54:52.325831Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2021-12-19T09:54:52.326068Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2021-12-19T09:54:52.326225Z 0 [Note] [MY-000000] [Galera] initializing ssl context
2021-12-19T09:54:52.326580Z 0 [Note] [MY-000000] [Galera] backend: asio
2021-12-19T09:54:52.326788Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2021-12-19T09:54:52.327015Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2021-12-19T09:54:52.327178Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2021-12-19T09:54:52.327607Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2021-12-19T09:54:52.327961Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2021-12-19T09:54:52.328135Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2021-12-19T09:54:52.328654Z 0 [Note] [MY-000000] [Galera] EVS version 1
2021-12-19T09:54:52.328937Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-cluster', peer '192.168.1.201:,192.168.1.202:,192.168.1.203:'
2021-12-19T09:54:52.365001Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://192.168.1.202:4567 local endpoint ssl://192.168.1.202:52472 cipher: TLS_AES_256_GCM_SHA384 compression: none
2021-12-19T09:54:52.366809Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:54:52.372606Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://192.168.1.202:52472 local endpoint ssl://192.168.1.202:4567 cipher: TLS_AES_256_GCM_SHA384 compression: none
2021-12-19T09:54:52.373137Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address ssl://192.168.1.202:4567
2021-12-19T09:54:55.332667Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.203:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 729460 cwnd: 1 last_queued_since: 1029582335770 last_delivered_since: 1029582335770 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2021-12-19T09:54:55.332945Z 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout)
2021-12-19T09:54:55.333009Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2021-12-19T09:54:55.333127Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2021-12-19T09:54:55.333161Z 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible
2021-12-19T09:54:55.333190Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,b6293123-9dd5,1)
memb {
b6293123-9dd5,0
}
joined {
}
left {
}
partitioned {
}
)
2021-12-19T09:54:55.334714Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:54:55.833560Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50486S), skipping check
2021-12-19T09:54:59.336234Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.203:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 733464 cwnd: 1 last_queued_since: 1033585904141 last_delivered_since: 1033585904141 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2021-12-19T09:54:59.338175Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:55:03.340470Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.203:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 737468 cwnd: 1 last_queued_since: 1037590091254 last_delivered_since: 1037590091254 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2021-12-19T09:55:03.345892Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:55:07.345509Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.203:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 741472 cwnd: 1 last_queued_since: 1041595178419 last_delivered_since: 1041595178419 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2021-12-19T09:55:07.347428Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:55:11.350696Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.203:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 745476 cwnd: 1 last_queued_since: 1045600347559 last_delivered_since: 1045600347559 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2021-12-19T09:55:11.354240Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:55:15.355408Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.203:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 749480 cwnd: 1 last_queued_since: 1049604998201 last_delivered_since: 1049604998201 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2021-12-19T09:55:15.363453Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:55:19.359430Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.203:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 753484 cwnd: 1 last_queued_since: 1053609088518 last_delivered_since: 1053609088518 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2021-12-19T09:55:19.362440Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:55:23.363121Z 0 [Note] [MY-000000] [Galera] (b6293123-9dd5, 'ssl://0.0.0.0:4567') connection to peer 00000000-0000 with addr ssl://192.168.1.203:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 757488 cwnd: 1 last_queued_since: 1057612777212 last_delivered_since: 1057612777212 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2021-12-19T09:55:23.365920Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://192.168.1.201:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding')
This error is often caused by SSL issues. For more information, please see:
https://per.co.na/pxc/encrypt_cluster_traffic
--------
2021-12-19T09:55:25.364616Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2021-12-19T09:55:25.364669Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2021-12-19T09:55:25.364990Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():161
2021-12-19T09:55:25.365024Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)
2021-12-19T09:55:26.365307Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2021-12-19T09:55:26.365398Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2021-12-19T09:55:26.365665Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1757: Failed to open channel 'pxc-cluster' at 'gcomm://192.168.1.201,192.168.1.202,192.168.1.203': -110 (Connection timed out)
2021-12-19T09:55:26.365725Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
2021-12-19T09:55:26.365791Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.1.201,192.168.1.202,192.168.1.203) failed to establish connection with cluster (reason: 7)
2021-12-19T09:55:26.365834Z 0 [ERROR] [MY-010119] [Server] Aborting
2021-12-19T09:55:26.366266Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.25-15.1) Percona XtraDB Cluster (GPL), Release rel15, Revision 8638bb0, WSREP version 26.4.3.
2021-12-19T09:55:26.367570Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
2021-12-19T09:55:26.367654Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2021-12-19T09:55:26.371518Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-12-19T09:55:26.373479Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-12-19T09:55:26.375406Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2021-12-19T09:55:26.375422Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
2021-12-19T09:55:26.375432Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
2021-12-19T09:55:26.375440Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
2021-12-19T09:55:26.375451Z 0 [Note] [MY-000000] [Galera] avg deps dist 0
2021-12-19T09:55:26.375460Z 0 [Note] [MY-000000] [Galera] avg cert interval 0
2021-12-19T09:55:26.375469Z 0 [Note] [MY-000000] [Galera] cert index size 0
2021-12-19T09:55:26.375500Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-12-19T09:55:26.375517Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
2021-12-19T09:55:26.375528Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2021-12-19T09:55:26.375608Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)
2021-12-19T09:55:26.464526Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...
Content of “/var/lib/mysql/grastate.dat” on node1
# GALERA saved state
version: 2.1
uuid: 750dbe9c-60af-11ec-8912-aa183b7e38cf
seqno: -1
safe_to_bootstrap: 1
Content of “/var/lib/mysql/grastate.dat” on node2
# GALERA saved state
version: 2.1
uuid: ed41a615-60ae-11ec-8155-7b1ae29c3096
seqno: 2
safe_to_bootstrap: 1
Note: I have not configured node3 because of the errors above