Failed to open IST listener

2023-11-19T03:28:28.076301Z 1 [Note] [MY-000000] [WSREP] Server status change connected → joiner

2023-11-19T03:28:28.076307Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2023-11-19T03:28:28.076471Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘34.143.208.100’ --datadir ‘/data/mysql/’ --basedir ‘/usr/’ --plugindir ‘/usr/lib/mysql/plugin/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘2699’ --mysqld-version ‘8.0.34-26.1’ ‘’ )

2023-11-19T03:28:28.570522Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /data/mysql//sst_in_progress

2023-11-19T03:28:28.951380Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|34.143.208.100:4444/xtrabackup_sst//1

2023-11-19T03:28:28.951426Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST

2023-11-19T03:28:28.951439Z 1 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: a248e4ef-83e3-11ee-a9f6-3e8ed666fdd7

2023-11-19T03:28:28.951449Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 5092976, STRv: 3

2023-11-19T03:28:28.951511Z 1 [Note] [MY-000000] [Galera] IST receiver addr using ssl://34.143.208.100:4568

2023-11-19T03:28:28.951572Z 1 [Note] [MY-000000] [Galera] IST receiver using ssl

2023-11-19T03:28:28.951673Z 1 [Note] [MY-000000] [Galera] State gap can’t be serviced using IST. Switching to SST

2023-11-19T03:28:28.951692Z 1 [Warning] [MY-000000] [Galera] Failed to prepare for incremental state transfer: Failed to open IST listener at ssl://34.143.208.100:4568’, asio error 'Failed to listen: bind: Cannot assign requested address: 99 (Cannot assign requested address)

at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.34-26/percona-xtradb-cluster-galera/galerautils/src/gu_asio_stream_react.cpp:listen():850’: 99 (Cannot assign requested address)

at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.34-26/percona-xtradb-cluster-galera/galera/src/ist.cpp:prepare():349. IST will be unavailable.

2023-11-19T03:28:28.955787Z 0 [Note] [MY-000000] [Galera] Member 2.0 (pxc-cluster-node-210) requested state transfer from ‘any’. Selected 0.0 (pxc-cluster-node-4)(SYNCED) as donor.

2023-11-19T03:28:28.955829Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY → JOINER (TO: 5093000)

2023-11-19T03:28:28.955894Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0

2023-11-19T03:28:28.955912Z 1 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.

2023-11-19T03:28:28.955924Z 1 [Note] [MY-000000] [Galera] GCache history reset: a248e4ef-83e3-11ee-a9f6-3e8ed666fdd7:0 → a248e4ef-83e3-11ee-a9f6-3e8ed666fdd7:5092976

2023-11-19T03:28:28.960593Z 0 [Warning] [MY-000000] [Galera] 0.0 (pxc-cluster-node-4): State transfer to 2.0 (pxc-cluster-node-210) failed: -42 (No message of desired type)

2023-11-19T03:28:28.960613Z 0 [ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.34-26/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1216: Will never receive state. Need to abort.

2023-11-19T03:28:28.960622Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread

2023-11-19T03:28:28.960632Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread

2023-11-19T03:28:28.960704Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend

2023-11-19T03:28:28.963465Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

This issue has been resolved. It seems to be a bug where Google Cloud servers are not assigned public IPs, making it impossible to bind an IP directly. In the end, I resolved it by using a domain name and updating the hosts file to point to the private IP for binding. Other nodes connect to this node through the domain name.

Using domain names is not recommended. You don’t need public IPs. You can use the private IPs assigned by google and have all intra-node communication travel via private network (recommended for security). Then only the load-balancer, or proxy has a public IP and a private in the same network as the nodes. This greatly reduces the number of attack vectors.