Nodes in cluster doesnt join eachother

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

Here is my first node logs

2024-05-02T14:10:34.068419Z 1 [Note] [MY-000000] [WSREP] Server status change connected → joiner
2024-05-02T14:10:34.068506Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:10:34.068571Z 1 [Note] [MY-000000] [WSREP] Server status change joiner → initializing
2024-05-02T14:10:34.068604Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:10:34.074339Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-02T14:10:34.396299Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-05-02T14:10:34.541257Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2024-05-02T14:10:34.551171Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2024-05-02T14:10:34.667864Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location ‘/var/lib/mysql’ in the path is accessible to all OS users. Consider choosing a different directory.
2024-05-02T14:10:34.701408Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2024-05-02T14:10:34.701851Z 0 [Note] [MY-000000] [Galera] Server initialized
2024-05-02T14:10:34.702293Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → initialized
2024-05-02T14:10:34.702696Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:10:34.703178Z 1 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
2024-05-02T14:10:34.707109Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: ‘8.0.36-28.1’ socket: ‘/tmp/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel28, Revision bfb687f, WSREP version 26.1.4.3.
2024-05-02T14:10:34.707559Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: ‘::’ port: 33060, socket: /var/lib/mysql/mysqlx.sock
2024-05-02T14:10:34.708499Z 8 [Note] [MY-000000] [WSREP] Recovered cluster id b3071dca-088d-11ef-b4ee-3e8bf9d03234
2024-05-02T14:10:34.709898Z 1 [Note] [MY-000000] [WSREP] Server status change initialized → joined
2024-05-02T14:10:34.710296Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:10:34.710674Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:10:34.714839Z 11 [Note] [MY-000000] [WSREP] Starting applier thread 11
2024-05-02T14:10:34.715643Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 24
2024-05-02T14:10:34.716060Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 24
2024-05-02T14:10:34.716498Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2024-05-02T14:10:34.716962Z 1 [Note] [MY-000000] [Galera] Server server-1 synced with group
2024-05-02T14:10:34.717357Z 1 [Note] [MY-000000] [WSREP] Server status change joined → synced
2024-05-02T14:10:34.717753Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2024-05-02T14:10:34.718143Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

docker exec -it percona-dockercompose-percona-1 mysql -uroot -p’rootpassword’ -e “SHOW STATUS LIKE ‘wsrep_cluster_size’;”

mysql: [Warning] Using a password on the command line interface can be insecure.
±-------------------±------+
| Variable_name | Value |
±-------------------±------+
| wsrep_cluster_size | 1 |
±-------------------±------+

and here is second

2024-05-02T14:11:27.209778Z 1 [Note] [MY-000000] [WSREP] Server status change connected → joiner
2024-05-02T14:11:27.209834Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:11:27.209899Z 1 [Note] [MY-000000] [WSREP] Server status change joiner → initializing
2024-05-02T14:11:27.212486Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:11:27.217770Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-02T14:11:27.563746Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-05-02T14:11:27.704162Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2024-05-02T14:11:27.713582Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2024-05-02T14:11:27.832412Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location ‘/var/lib/mysql’ in the path is accessible to all OS users. Consider choosing a different directory.
2024-05-02T14:11:27.856204Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2024-05-02T14:11:27.856611Z 0 [Note] [MY-000000] [Galera] Server initialized
2024-05-02T14:11:27.856993Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → initialized
2024-05-02T14:11:27.857385Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:11:27.857793Z 1 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
2024-05-02T14:11:27.861610Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: ‘8.0.36-28.1’ socket: ‘/tmp/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel28, Revision bfb687f, WSREP version 26.1.4.3.
2024-05-02T14:11:27.862120Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: ‘::’ port: 33060, socket: /var/lib/mysql/mysqlx.sock
2024-05-02T14:11:27.862819Z 8 [Note] [MY-000000] [WSREP] Recovered cluster id d2330df8-088d-11ef-a2e6-6b2f64ddc605
2024-05-02T14:11:27.864156Z 1 [Note] [MY-000000] [WSREP] Server status change initialized → joined
2024-05-02T14:11:27.864525Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:11:27.864882Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:11:27.868868Z 11 [Note] [MY-000000] [WSREP] Starting applier thread 11
2024-05-02T14:11:27.869565Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 24
2024-05-02T14:11:27.869929Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 24
2024-05-02T14:11:27.870299Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2024-05-02T14:11:27.870745Z 1 [Note] [MY-000000] [Galera] Server server-3 synced with group
2024-05-02T14:11:27.871086Z 1 [Note] [MY-000000] [WSREP] Server status change joined → synced
2024-05-02T14:11:27.871431Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2024-05-02T14:11:27.871802Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-02T14:13:00.396566Z 12 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ‘‘mysql_native_password’ is deprecated and will be removed in a future release. Please use caching_sha2_password instead’
mbind: Operation not permitted
^C
root@server-3:/home/slave1/Percona-DockerCompose# docker exec -it percona-dockercompose-percona-1 mysql -uroot -p’rootpassword’ -e “SHOW STATUS LIKE ‘wsrep_cluster_size’;”
mysql: [Warning] Using a password on the command line interface can be insecure.
±-------------------±------+
| Variable_name | Value |
±-------------------±------+
| wsrep_cluster_size | 1 |
±-------------------±------+

here is my first node cluseter configuration
docker-compose.yml

version: ‘3.9’

services:
percona:
image: percona/percona-xtradb-cluster:8.0
environment:
- MYSQL_ROOT_PASSWORD=rootpassword
- CLUSTER_NAME=pxc-cluster
- XTRABACKUP_PASSWORD=backuppassword
- MYSQL_DATABASE=exampledb
- MYSQL_USER=exampleuser
- MYSQL_PASSWORD=examplepass
command:
- --wsrep-new-cluster
volumes:
- ./data:/var/lib/mysql
- ./custom.cnf:/etc/mysql/percona-xtradb-cluster.conf.d/custom.cnf
network_mode: host
restart: always

cat custom.cnf

[mysqld]
server-id=1
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
wsrep_on=ON
wsrep_cluster_name=“pxc-cluster”
wsrep_cluster_address=“gcomm://172.31.60.117,172.31.60.118,172.31.60.119”
wsrep_node_address=“172.31.60.117”
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=“xtrabackup:backuppassword”
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
wsrep_node_name=“percona-cluster-db1”
bind-address = 0.0.0.0
pxc_strict_mode=ENFORCING
binlog_format=ROW

and here is my second node configuration

docker-compose.yml

version: ‘3.9’

services:
percona:
image: percona/percona-xtradb-cluster:8.0
environment:
- MYSQL_ROOT_PASSWORD=rootpassword
- CLUSTER_NAME=pxc-cluster
- XTRABACKUP_PASSWORD=backuppassword
- MYSQL_DATABASE=exampledb
- MYSQL_USER=exampleuser
- MYSQL_PASSWORD=examplepass
volumes:
- ./data:/var/lib/mysql
- ./custom.cnf:/etc/mysql/percona-xtradb-cluster.conf.d/custom.cnf
network_mode: host
restart: always

custom.cnf

[mysqld]
server-id=2
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
wsrep_on=ON
wsrep_cluster_name=“pxc-cluster”
wsrep_cluster_address=“gcomm://172.31.60.117,172.31.60.118,172.31.60.119”
wsrep_node_address=“172.31.60.119”
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=“xtrabackup:backuppassword”
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
wsrep_node_name=“percona-cluster-db2”
bind-address = 0.0.0.0
host_cache_size = 0
wsrep_applier_threads = 2
pxc_strict_mode=ENFORCING
binlog_format=ROW

why my nodes are not join each other?!they are on same vlan! and there isnt any firewall or iptables between them!they are on labratoar machines!
also my nc from both node the ports 4568 and 4444 are closed

nc -zv 172.31.60.119 3306 4567 4568 4444
Connection to 172.31.60.119 3306 port [tcp/mysql] succeeded!
Connection to 172.31.60.119 4567 port [tcp/*] succeeded!
nc: connect to 172.31.60.119 port 4568 (tcp) failed: Connection refused
nc: connect to 172.31.60.119 port 4444 (tcp) failed: Connection refused
for example from first node to second node!

2

HI @Sasan_Askari , Welcome to the Percona Community…!

As per the above error logs, seems like you are bootstrapping both nodes. So, the cluster size is command is showing 1 node in the both nodes.

We should always start the first node in bootstrap node and next nodes should be started normally to join with previous nodes.

Regarding ports, PXC cluster needs below default ports which needs to be accessible among all nodes in the PXC Cluster.

  • 3306 is used for MySQL client connections
  • 4444 is used for SST via rsync and Percona XtraBackup.
  • 4567 is used for write-set replication traffic (over TCP) and multicast replication (over TCP and UDP).
  • 4568 is used for IST (Incremental State Transfer).
3

Thank you for your guidance! Can you help me correct my Docker Compose configuration for both nodes?

command:

  • –wsrep-new-cluster

I used the above command in the first node to bootstrap it, so I think I was wrong!

4

I cahnge in my configuraion
Here is my first node

docker-compose.yml

version: ‘3.7’

services:
percona1:
image: percona/percona-xtradb-cluster:8.0
environment:
- CLUSTER_NAME=pxc-cluster
- XTRABACKUP_PASSWORD=xtrabackup_password
- MYSQL_ROOT_PASSWORD=root_password
- MYSQL_DATABASE=mydatabase
- MYSQL_USER=myuser
- MYSQL_PASSWORD=mypassword
- WSREP_NODE_ADDRESS=172.31.60.117
volumes:
- ./data:/var/lib/mysql
- ./custom.cnf:/etc/mysql/conf.d/custom.cnf
network_mode: “host”

cat custom.cnf

[mysqld]
bind-address = 0.0.0.0
wsrep_on=ON
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so
wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://
wsrep_sst_method=xtrabackup-v2
wsrep_node_name=“percona-cluster-db1”
wsrep_node_address=“172.31.60.117”
server-id=1
wsrep_provider_options=“socket.ssl=NO”

and here is my second node configuration

docker-compose.yml

version: ‘3.7’

services:
percona2:
image: percona/percona-xtradb-cluster:8.0
environment:
- CLUSTER_NAME=pxc-cluster
- XTRABACKUP_PASSWORD=xtrabackup_password
- MYSQL_ROOT_PASSWORD=root_password
- MYSQL_DATABASE=mydatabase
- MYSQL_USER=myuser
- MYSQL_PASSWORD=mypassword
- WSREP_NODE_ADDRESS=172.31.60.119
volumes:
- ./data:/var/lib/mysql
- ./custom.cnf:/etc/mysql/conf.d/custom.cnf
network_mode: “host”

cat custom.cnf

[mysqld]
bind-address = 0.0.0.0
wsrep_on=ON
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so
wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://172.31.60.117:4567
wsrep_sst_method=xtrabackup-v2
wsrep_node_name=“percona-cluster-db2”
wsrep_node_address=“172.31.60.119”
server-id=2
wsrep_provider_options=“socket.ssl=NO”

but i get these logs from first node container:

2024-05-03T11:56:46.994670Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-03T11:56:46.998437Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 25
2024-05-03T11:56:46.998900Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 25
2024-05-03T11:56:46.999347Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2024-05-03T11:56:48.298364Z 0 [Note] [MY-000000] [Galera] Member 1.0 (percona-cluster-db2) requested state transfer from ‘any’. Selected 0.0 (percona-cluster-db1)(SYNCED) as donor.
2024-05-03T11:56:48.299065Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED → DONOR/DESYNCED (TO: 25)
2024-05-03T11:56:48.299471Z 1 [Note] [MY-000000] [Galera] Detected STR version: 1, req_len: 133, req: STRv1
2024-05-03T11:56:48.299864Z 1 [Note] [MY-000000] [Galera] Cert index preload: 25 → 25
2024-05-03T11:56:48.300611Z 1 [Note] [MY-000000] [WSREP] Server status change synced → donor
2024-05-03T11:56:48.300922Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-03T11:56:48.301427Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘172.31.60.119:4444/xtrabackup_sst//1’ --socket ‘/tmp/mysql.sock’ --datadir ‘/var/lib/mysql/’ --basedir ‘/usr/’ --plugindir ‘/usr/lib64/mysql/plugin/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘8.0.36-28.1’ ‘’ --gtid ‘207ee875-0944-11ef-b35d-f6daf1e1f7e8:25’ )
2024-05-03T11:56:48.304418Z 0 [Note] [MY-000000] [Galera] async IST sender starting to serve tcp://172.31.60.119:4568 sending 25-25, preload starts from 25
2024-05-03T11:56:48.304916Z 0 [Note] [MY-000000] [Galera] IST sender 25 → 25
2024-05-03T11:56:48.311069Z 12 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ‘‘mysql_native_password’ is deprecated and will be removed in a future release. Please use caching_sha2_password instead’
2024-05-03T11:56:48.313052Z 1 [Note] [MY-000000] [WSREP] DONOR thread signaled with 0
2024-05-03T11:56:48.453845Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* ERROR **********************
2024-05-03T11:56:48.454201Z 0 [ERROR] [MY-000000] [WSREP-SST] Missing version string in comparison
2024-05-03T11:56:48.454498Z 0 [ERROR] [MY-000000] [WSREP-SST] left-side: operation:< right-side:2.4.29
2024-05-03T11:56:48.454790Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* ERROR **********************
2024-05-03T11:56:49.206508Z 0 [Note] [MY-000000] [Galera] (2bf9160d-9bfb, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2024-05-03T11:56:49.306498Z 13 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ‘‘mysql_native_password’ is deprecated and will be removed in a future release. Please use caching_sha2_password instead’
2024-05-03T11:56:49.310960Z 13 [Warning] [MY-013712] [Server] No suitable ‘keyring_component_metadata_query’ service implementation found to fulfill the request.
2024-05-03T11:56:49.347607Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:49 socat[718] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:50.365377Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:50 socat[721] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:51.384731Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:51 socat[724] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:52.404129Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:52 socat[727] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:53.422993Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:53 socat[730] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:54.440124Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:54 socat[733] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:55.456840Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:55 socat[736] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:56.474493Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:56 socat[739] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:57.492080Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:57 socat[742] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:58.508463Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:58 socat[745] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:56:59.524905Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:56:59 socat[748] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:57:00.542288Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:57:00 socat[751] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:57:01.559595Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:57:01 socat[754] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:57:02.577415Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:57:02 socat[757] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:57:03.594968Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:57:03 socat[760] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
2024-05-03T11:57:04.611778Z 0 [Note] [MY-000000] [WSREP-SST] 2024/05/03 11:57:04 socat[763] E SSL_connect(): error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus

and here is second nod log container

2024-05-03T11:56:46.509125Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-05-03T11:56:46.999479Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2024-05-03T11:56:46.999692Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2024-05-03T11:56:47.000042Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)
2024-05-03T11:56:47.000221Z 0 [Note] [MY-000000] [Galera] Opened channel ‘pxc-cluster’
2024-05-03T11:56:47.001048Z 1 [Note] [MY-000000] [WSREP] Starting applier thread 1
2024-05-03T11:56:47.001380Z 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2
2024-05-03T11:56:47.001687Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2024-05-03T11:56:47.001954Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: Waiting for state UUID.
2024-05-03T11:56:47.002272Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 377907e1-0944-11ef-ad3e-3635768b8a24
2024-05-03T11:56:47.002670Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 377907e1-0944-11ef-ad3e-3635768b8a24 from 0 (percona-cluster-db1)
2024-05-03T11:56:47.002911Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 377907e1-0944-11ef-ad3e-3635768b8a24 from 1 (percona-cluster-db2)
2024-05-03T11:56:47.003093Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 1,
members = 1/2 (primary/total),
act_id = 24,
last_appl. = 23,
protocols = 2/11/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 207ee875-0944-11ef-b35d-f6daf1e1f7e8
2024-05-03T11:56:47.003486Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2024-05-03T11:56:47.003668Z 0 [Note] [MY-000000] [Galera] Shifting OPEN → PRIMARY (TO: 25)
2024-05-03T11:56:47.003941Z 1 [Note] [MY-000000] [Galera] ####### processing CC 25, local, ordered
2024-05-03T11:56:47.004147Z 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 1 upto current CC event 25 upto:1
2024-05-03T11:56:47.004355Z 1 [Note] [MY-000000] [Galera] Drain monitors from 1 up to 1
2024-05-03T11:56:47.004598Z 1 [Note] [MY-000000] [Galera] Process first view: 207ee875-0944-11ef-b35d-f6daf1e1f7e8 my uuid: 372e3f3b-0944-11ef-8259-27f90f2bc570
2024-05-03T11:56:47.004792Z 1 [Note] [MY-000000] [Galera] Server percona-cluster-db2 connected to cluster at position 207ee875-0944-11ef-b35d-f6daf1e1f7e8:25 with ID 372e3f3b-0944-11ef-8259-27f90f2bc570
2024-05-03T11:56:47.004986Z 1 [Note] [MY-000000] [WSREP] Server status change disconnected → connected
2024-05-03T11:56:47.005251Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-03T11:56:47.005509Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 372e3f3b-0944-11ef-8259-27f90f2bc570
2024-05-03T11:56:47.005696Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 11), state transfer needed: yes
2024-05-03T11:56:47.005957Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-05-03T11:56:47.006186Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-05-03T11:56:47.006388Z 1 [Note] [MY-000000] [Galera] State transfer required:
Group state: 207ee875-0944-11ef-b35d-f6daf1e1f7e8:25
Local state: 85c72e56-0943-11ef-b61d-772502bcc1ab:1
2024-05-03T11:56:47.006606Z 1 [Note] [MY-000000] [WSREP] Server status change connected → joiner
2024-05-03T11:56:47.006791Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-05-03T11:56:47.007149Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘172.31.60.119’ --datadir ‘/var/lib/mysql/’ --basedir ‘/usr/’ --plugindir ‘/usr/lib64/mysql/plugin/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘1’ --mysqld-version ‘8.0.36-28.1’ ‘’ )
2024-05-03T11:56:47.157768Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* ERROR **********************
2024-05-03T11:56:47.157956Z 0 [ERROR] [MY-000000] [WSREP-SST] Missing version string in comparison
2024-05-03T11:56:47.158123Z 0 [ERROR] [MY-000000] [WSREP-SST] left-side: operation:< right-side:2.4.29
2024-05-03T11:56:47.158270Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* ERROR **********************
2024-05-03T11:56:47.881463Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2024-05-03T11:56:48.315749Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|172.31.60.119:4444/xtrabackup_sst//1
2024-05-03T11:56:48.316002Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2024-05-03T11:56:48.316185Z 1 [Note] [MY-000000] [Galera] Local UUID: 85c72e56-0943-11ef-b61d-772502bcc1ab != Group UUID: 207ee875-0944-11ef-b35d-f6daf1e1f7e8
2024-05-03T11:56:48.316362Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:85c72e56-0943-11ef-b61d-772502bcc1ab f: 0, l: 25, STRv: 3
2024-05-03T11:56:48.316578Z 1 [Note] [MY-000000] [Galera] IST receiver addr using tcp://172.31.60.119:4568
2024-05-03T11:56:48.316916Z 1 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-25, listening at: tcp://172.31.60.119:4568
2024-05-03T11:56:48.318542Z 0 [Note] [MY-000000] [Galera] Member 1.0 (percona-cluster-db2) requested state transfer from ‘any’. Selected 0.0 (percona-cluster-db1)(SYNCED) as donor.
2024-05-03T11:56:48.318709Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY → JOINER (TO: 25)
2024-05-03T11:56:48.318890Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
2024-05-03T11:56:48.319057Z 1 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2024-05-03T11:56:48.319208Z 1 [Note] [MY-000000] [Galera] GCache history reset: 85c72e56-0943-11ef-b61d-772502bcc1ab:1 → 207ee875-0944-11ef-b35d-f6daf1e1f7e8:25
2024-05-03T11:56:48.319570Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 18446744073709551600 bytes
2024-05-03T11:56:48.319718Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 1/2 locked buffers
2024-05-03T11:56:49.499832Z 0 [Note] [MY-000000] [Galera] (372e3f3b-8259, ‘tcp://0.0.0.0:4567’) turning message relay requesting off

2024-05-03T11:57:19.889523Z 0 [Warning] [MY-000000] [Galera] 0.0 (percona-cluster-db1): State transfer to 1.0 (percona-cluster-db2) failed: -32 (Broken pipe)
2024-05-03T11:57:19.889985Z 0 [ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1285: Will never receive state. Need to abort.
2024-05-03T11:57:19.891319Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2024-05-03T11:57:19.891818Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2024-05-03T11:57:19.892427Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2024-05-03T11:57:20.896114Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,2bf9160d-9bfb,2)
memb {
372e3f3b-8259,0
}
joined {
}
left {
}
partitioned {
2bf9160d-9bfb,0
}
)
2024-05-03T11:57:20.898604Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 → 0
2024-05-03T11:57:20.898943Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2024-05-03T11:57:20.899642Z 0 [Note] [MY-000000] [Galera] gcomm: closed
2024-05-03T11:57:20.899966Z 0 [Note] [MY-000000] [Galera] mysqld: Terminated.
2024-05-03T11:57:20.900248Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2024-05-03T11:57:20.900524Z 0 [Note] [MY-000000] [WSREP] Terminating SST process
2024-05-03T11:57:20.904117Z 0 [ERROR] [MY-000000] [WSREP-SST] Removing /var/lib/mysql//xtrabackup_galera_info file due to signal
2024-05-03T11:57:20Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=8849a9bea52d6d7129f6c4910d67e380da94c3ed
Server Version: 8.0.36-28.1 Percona XtraDB Cluster (GPL), Release rel28, Revision bfb687f, WSREP version 26.1.4.3, wsrep_26.1.4.3

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x2283e41]
2024-05-03T11:57:20.912142Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2024-05-03T11:57:20.912430Z 0 [ERROR] [MY-000000] [WSREP-SST] SST script interrupted
2024-05-03T11:57:20.912694Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************************************************
2024-05-03T11:57:20.912948Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32

is there anyone to help me and solve this problem?!