I have face problem still one week please help me

Greetings,

i am using Percona cluster 8.0.32 and xtrabackup 8.0.33 galere is 4.14 and wrep version 26.4
i got this error in node2 and node3
ing loop body : 2
2023-06-09T09:02:30.944897Z 0 [Note] [MY-000000] [WSREP-SST] (debug) 1235: Examining pid: 1143577
2023-06-09T09:02:31.070900Z 0 [Note] [MY-000000] [WSREP-SST] (debug) 1252: sockets: 7274902|7276591|7276592|7276593
2023-06-09T09:02:31.076047Z 0 [Note] [MY-000000] [WSREP-SST] (debug) 1276: found a match for pid: 1143577
2023-06-09T09:02:31.076235Z 0 [Note] [MY-000000] [WSREP-SST] (debug) 1294: wait_for_listen() exiting
2023-06-09T09:02:31.076354Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|20.212.201.14:4444/xtrabackup_sst//1
2023-06-09T09:02:31.076461Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2023-06-09T09:02:31.076542Z 1 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: ddd869f1-044f-11ee-8d63-0311dc43e2f4
2023-06-09T09:02:31.076574Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 112, STRv: 3
2023-06-09T09:02:31.076781Z 1 [Note] [MY-000000] [Galera] IST receiver addr using tcp://20.212.201.14:4568
2023-06-09T09:02:31.077010Z 1 [Note] [MY-000000] [Galera] State gap can’t be serviced using IST. Switching to SST
2023-06-09T09:02:31.077036Z 1 [Warning] [MY-000000] [Galera] Failed to prepare for incremental state transfer: Failed to open IST listener at tcp://20.212.201.14:4568’, asio error ‘Failed to listen: bind: Cannot assign requested address: 99 (Cannot assign requested address)
at galerautils/src/gu_asio_stream_react.cpp:listen():788’: 99 (Cannot assign requested address)
at galera/src/ist.cpp:prepare():349. IST will be unavailable.
2023-06-09T09:02:31.078180Z 0 [Note] [MY-000000] [Galera] Member 1.0 (dev-pxc-cluster-node-2) requested state transfer from ‘any’. Selected 0.0 (dev-pxc-cluster-node-1)(SYNCED) as donor.
2023-06-09T09:02:31.078251Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY → JOINER (TO: 112)
2023-06-09T09:02:31.078315Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
2023-06-09T09:02:31.078362Z 1 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2023-06-09T09:02:31.078417Z 1 [Note] [MY-000000] [Galera] GCache history reset: ddd869f1-044f-11ee-8d63-0311dc43e2f4:0 → ddd869f1-044f-11ee-8d63-0311dc43e2f4:112
2023-06-09T09:02:31.079065Z 0 [Warning] [MY-000000] [Galera] 0.0 (dev-pxc-cluster-node-1): State transfer to 1.0 (dev-pxc-cluster-node-2) failed: -42 (No message of desired type)
2023-06-09T09:02:31.079134Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1216: Will never receive state. Need to abort.
2023-06-09T09:02:31.079180Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2023-06-09T09:02:31.079231Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2023-06-09T09:02:31.079386Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2023-06-09T09:02:32.085617Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,2f98ea99-b381,6)
memb {
5c802700-9c1d,0
}
joined {
}
left {
}
partitioned {
2f98ea99-b381,0
}
)
2023-06-09T09:02:32.085726Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 → 0
2023-06-09T09:02:32.085748Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2023-06-09T09:02:32.085954Z 0 [Note] [MY-000000] [Galera] gcomm: closed
2023-06-09T09:02:32.085992Z 0 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2023-06-09T09:02:32.086007Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2023-06-09T09:02:32.086030Z 0 [Note] [MY-000000] [WSREP] Terminating SST process

MY CONFIG FILE

The Percona Server 8.0 configuration file.

For explanations see

http://dev.mysql.com/doc/mysql/en/server-system-variables.html

* IMPORTANT: Additional settings that can override those from this file!

The files must end with ‘.cnf’, otherwise they’ll be ignored.

!includedir /etc/mysql/mysql.conf.d/
!includedir /etc/mysql/conf.d/
[client]
#socket=/var/lib/mysql/mysql.sock
protocol=TCP

SSL certificates

#ssl_ca=/etc/mysql/my.cnf.d/certificates/ca.pem
#ssl_cert=/etc/mysql/my.cnf.d/certificates/server-cert.pem
#ssl_key=/etc/mysql/my.cnf.d/certificates/server-key.pem

[mysqld]
server_id=96
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log_error=/var/log/mysql/error.log
pid_file=/var/run/mysqld/mysqld.pid
pxc-encrypt-cluster-traffic=OFF
######## wsrep ###############

Path to Galera library

wsrep_provider=/usr/lib/galera4/libgalera_smm.so

Cluster connection URL contains IPs of nodes

#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://20.212.157.189,20.212.201.14,20.164.218.83

In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

Slave threads to use - start with four slave threads per CPU core

#wsrep_slave_threads=8
wsrep_applier_threads=8

Log additional information about node conflicts

wsrep_log_conflicts=ON

This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

Node IP address

#wsrep_node_address=192.168.164.95
#wsrep_node_address=10.2.2.95
wsrep_node_address=20.212.201.14
wsrep_on=ON
wsrep_sst_xtrabackup =/var/lib/mysql
wsrep_sst_receive_address=20.212.201.14

Cluster name

wsrep_cluster_name=dev-pxc-cluster

#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=dev-pxc-cluster-node-2

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

SSL certificates for pxc_encrypt_cluster_traffic

#ssl_ca=/etc/mysql/my.cnf.d/certificates/ca.pem
#ssl_cert=/etc/mysql/my.cnf.d/certificates/server-cert.pem
#ssl_key=/etc/mysql/my.cnf.d/certificates/server-key.pem
#pxc_encrypt_cluster_traffic=ON

SST method

wsrep_sst_method=xtrabackup-v2
#wsrep_sst_method=rsync
#wsrep_sst_method=skip
wsrep-debug=1

Increasing the cache size inceases the chance of IST rather than SST

wsrep_provider_options=“gcache.size=5G”

This is best placed on a fast IO disk rather than in the /var/lib/mysql directory

#wsrep_provider_options=“gcache.dir=/mnt/disk3/cache”

Bind all network cards

bind_address=0.0.0.0

Event scheduler

event_scheduler=ON

Necessary for Percona

default_storage_engine=InnoDB

Security issue

symbolic_links=0

All tables names are lower case

#lower_case_table_names=1

Default SQl mode

#sql_mode=“”
#sql_mode=“ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION”

Enable federated engine

federated

Allow import local files

local_infile=ON

Fine Tuning

key_buffer_size=2048M
#sort_buffer_size=2M
#join_buffer_size=2M
#tmp_table_size=10M
max_allowed_packet=1G
#table_open_cache=4000
#table_definition_cache=2000

Connections

max_connections=250
max_user_connections=20
thread_cache_size=100

Logging and Replication

#log_error=/var/log/mysql/error.log
log_output=FILE
#general_log=ON
#general_log_file=/var/log/mysql/general.log
slow_query_log=ON
#slow_query_log_file=/var/log/mysql/slow.log
log_slow_rate_type=QUERY
log_slow_verbosity=FULL
log_slow_rate_limit=1
slow_query_log_always_write_time=1
log_slow_admin_statements=ON
log_queries_not_using_indexes=ON
long_query_time=0
min_examined_row_limit=0
userstat=1
slow_query_log_use_global_control=all

Enabling the InnoDB Lock Monitor

innodb_status_output=ON
innodb_status_output_locks=ON

timeouts and limits

wait_timeout=1200
interactive_timeout=1200
open_files_limit=65535

InnoDB Parameters

buffer pool should be sized to 50% to 75% of system memory

innodb_buffer_pool_size=5G
#innodb_buffer_pool_instances=1
#innodb_buffer_pool_chunk_size=128M
#innodb_log_buffer_size=16M
#innodb_log_file_size=512M
#innodb_file_per_table=ON
#innodb_data_file_path=ibdata1:12M:autoextend
#innodb_io_capacity=200
#innodb_flush_log_at_trx_commit=1
innodb_flush_method=O_DIRECT
#innodb_write_io_threads=4
#innodb_read_io_threads=4

Binlogs

log_bin=ON
#log_bin_basename=/var/log/mysql/binlog

7 days binlogs

#expire_logs_days=7
binlog_expire_logs_seconds=604800

30 days binlogs

#expire_logs_days=30
#binlog_expire_logs_seconds=2592000
#max_binlog_size=100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name

how to solve this isuue ,help me

Hi, this message points to the issue:

do you have any process already using port 4568?

No, dont use this port

netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:6032 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6033 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6033 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6033 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6033 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:7777 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:2019 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:42315 0.0.0.0:* LISTEN
tcp6 0 0 ::1:3350 :::* LISTEN
tcp6 0 0 :::80 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 :::443 :::* LISTEN
tcp6 0 0 :::3389 :::* LISTEN
tcp6 0 0 ::1:631 :::* LISTEN
udp 0 0 127.0.0.53:53 0.0.0.0:*
udp 0 0 10.15.0.4:68 0.0.0.0:*
udp 0 0 127.0.0.1:323 0.0.0.0:*
udp 0 0 0.0.0.0:631 0.0.0.0:*
udp 0 0 0.0.0.0:5353 0.0.0.0:*
udp 0 0 0.0.0.0:39104 0.0.0.0:*
udp6 0 0 ::1:323 :::*
udp6 0 0 :::5353 :::*
udp6 0 0 :::59166 what is error i can’t find still now

I’m facing literally the same issue, when using docker image.
So definitely nothing will be listening on that port since image should not be configured for anything to listen there.

It’s strange that this happens when pxc-encrypt-cluster-traffic is set to OFF. Haven’t fully tested the other way since i haven’t (yet) configured ca and rest of certificates. But this does seem like a bug.