Hello,
I really need help here. I’ve 3 nodes and The 1st node started with bootstrap and the cluster is up and running with 1 node. Now, while the 2nd node trying to join the cluster the SST step is failing. I am trying to install cluster using helm chart and its not make POD up due to error -
pod/mysql-0130-pxc-db-pxc-0 2/2 Running
pod/mysql-0130-pxc-db-pxc-1 1/2 CrashLoopBackOff
When i checked logs for pxc container on “mysql-0130-pxc-db-pxc-1” it shows below error -
2022-01-09T11:29:08.689230Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2022-01-09T11:29:08.689807Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T11:29:08.690408Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 5baf313b-713f-11ec-a2a9-c79a0c530352
2022-01-09T11:29:08.690948Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2022-01-09T11:29:08.691652Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-01-09T11:29:08.692370Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2022-01-09T11:29:08.692947Z 2 [Note] [MY-000000] [Galera] State transfer required:
Group state: 15efe54f-7121-11ec-9225-1b13c8eeac3c:103
Local state: 00000000-0000-0000-0000-000000000000:-1
2022-01-09T11:29:08.693516Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-01-09T11:29:08.694134Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T11:29:08.695120Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '[240b:c0e0:202:55d8:b484:2:0:4a63]' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '8.0.22-13.1' --binlog 'binlog' )
2022-01-09T11:29:09.716950Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2022-01-09T11:29:10.226432Z 2 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|[240b:c0e0:202:55d8:b484:2:0:4a63]:4444/xtrabackup_sst//1
2022-01-09T11:29:10.227360Z 2 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2022-01-09T11:29:10.228001Z 2 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 15efe54f-7121-11ec-9225-1b13c8eeac3c
2022-01-09T11:29:10.228722Z 2 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 103, STRv: 3
2022-01-09T11:29:10.229509Z 2 [Note] [MY-000000] [Galera] IST receiver addr using ssl://[240b:c0e0:202:55d8:b484:2:0:4a63]:4568
2022-01-09T11:29:10.230258Z 2 [Note] [MY-000000] [Galera] IST receiver using ssl
2022-01-09T11:29:10.231634Z 2 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-103, listening at: ssl://[240b:c0e0:202:55d8:b484:2:0:4a63]:4568
2022-01-09T11:29:10.233105Z 0 [Note] [MY-000000] [Galera] Member 1.0 (mysql-0130-pxc-db-pxc-1) requested state transfer from '*any*'. Selected 0.0 (mysql-0130-pxc-db-pxc-0)(SYNCED) as donor.
2022-01-09T11:29:10.233913Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 103)
2022-01-09T11:29:10.234652Z 2 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
2022-01-09T11:29:10.235368Z 2 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2022-01-09T11:29:10.236093Z 2 [Note] [MY-000000] [Galera] GCache history reset: 15efe54f-7121-11ec-9225-1b13c8eeac3c:0 -> 15efe54f-7121-11ec-9225-1b13c8eeac3c:103
2022-01-09T11:29:11.180691Z 0 [Note] [MY-000000] [Galera] (5baf313b-a2a9, 'ssl://[::]:4567') turning message relay requesting off
2022-01-09T11:29:11.272889Z 0 [Warning] [MY-000000] [Galera] 0.0 (mysql-0130-pxc-db-pxc-0): State transfer to 1.0 (mysql-0130-pxc-db-pxc-1) failed: -32 (Broken pipe)
2022-01-09T11:29:11.275357Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receive state. Need to abort.
2022-01-09T11:29:11.275412Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-01-09T11:29:11.275469Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-01-09T11:29:11.275602Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-01-09T11:29:12.280052Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,27d97bde-b4ff,90)
memb {
5baf313b-a2a9,0
}
joined {
}
left {
}
partitioned {
27d97bde-b4ff,0
}
)
2022-01-09T11:29:12.280149Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-01-09T11:29:12.280181Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-01-09T11:29:12.280460Z 0 [Note] [MY-000000] [Galera] gcomm: closed
2022-01-09T11:29:12.280497Z 0 [Note] [MY-000000] [Galera] mysqld: Terminated.
2022-01-09T11:29:12.280518Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2022-01-09T11:29:12.280533Z 0 [Note] [MY-000000] [WSREP] Terminating SST process
11:29:12 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Build ID: 23c893872480ac27eda9bb248bcd29e2ceafa7b4
Server Version: 8.0.22-13.1 Percona XtraDB Cluster (GPL), Release rel13, Revision a48e6d5, WSREP version 26.4.3, wsrep_26.4.3
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x46000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x210e8d1]
/usr/sbin/mysqld(handle_fatal_signal+0x3c3) [0x12c47f3]
/lib64/libpthread.so.0(+0x12b20) [0x7fc91dfb8b20]
/lib64/libc.so.6(abort+0x203) [0x7fc91be85e91]
/usr/lib64/galera4/libgalera_smm.so(+0x143e5) [0x7fc90fd483e5]
/usr/lib64/galera4/libgalera_smm.so(+0x179d66) [0x7fc90feadd66]
/usr/lib64/galera4/libgalera_smm.so(+0x181e6e) [0x7fc90feb5e6e]
/lib64/libpthread.so.0(+0x814a) [0x7fc91dfae14a]
/lib64/libc.so.6(clone+0x43) [0x7fc91bf60dc3]
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file using lib coredumper
PATH: (null)
2022-01-09T11:29:12.291805Z 0 [ERROR] [MY-000000] [WSREP-SST] Removing /var/lib/mysql//xtrabackup_galera_info file due to signal
2022-01-09T11:29:12.974493Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2022-01-09T11:29:12.974531Z 0 [ERROR] [MY-000000] [WSREP-SST] SST script interrupted
2022-01-09T11:29:12.974557Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '[240b:c0e0:202:55d8:b484:2:0:4a63]' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '8.0.22-13.1' --binlog 'binlog' : 32 (Broken pipe)
2022-01-09T11:29:12.974606Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2022-01-09T11:29:12.974638Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32
Error writting coredump: -1 Signal: 11
This is conf file content on this node -
[mysqld]
pxc-encrypt-cluster-traffic=ON
ssl-ca=/etc/mysql/ssl-internal/ca.crt
ssl-key=/etc/mysql/ssl-internal/tls.key
ssl-cert=/etc/mysql/ssl-internal/tls.crt
wsrep_provider_options="pc.weight=10"
wsrep_sst_donor=
log_error_suppression_list="MY-010055"
admin-address=240b:c0e0:202:55d8:b484:2:0:4a63
enforce-gtid-consistency
gtid-mode=ON
plugin_load="binlog_utils_udf=binlog_utils_udf.so"
default_authentication_plugin=mysql_native_password
datadir=/var/lib/mysql
socket=/tmp/mysql.sock
skip-host-cache
coredumper
server_id=1
binlog_format=ROW
default_storage_engine=InnoDB
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_autoinc_lock_mode=2
bind_address = 0.0.0.0
wsrep_slave_threads=2
wsrep_cluster_address=gcomm://mysql-0130-pxc-db-pxc-0.mysql-0130-pxc-db-pxc
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so
wsrep_cluster_name=mysql-0130-pxc-db-pxc
wsrep_node_address=[240b:c0e0:202:55d8:b484:2:0:4a63]
wsrep_node_incoming_address=mysql-0130-pxc-db-pxc-1.mysql-0130-pxc-db-pxc.ns1.svc.cluster.local:3306
wsrep_sst_method=xtrabackup-v2
+ wsrep_start_position_opt=
[client]
socket=/tmp/mysql.sock
[sst]
cpat=.*\.pem$\|.*init\.ok$\|.*galera\.cache$\|.*wsrep_recovery_verbose\.log$\|.*readiness-check\.sh$\|.*liveness-check\.sh$\|.*sst_in_progress$\|.*sst-xb-tmpdir$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$\|.*pxc-entrypoint\.sh$\|.*unsafe-bootstrap\.sh$\|.*pxc-configure-pxc\.sh\|.*peer-list$
progress=1
This is the error i am seeing on pxc-0 node -
2022-01-09T07:52:55.843143Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-01-09T07:52:56.637409Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-01-09T07:52:56.967043Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2022-01-09T07:52:57.002598Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2022-01-09T07:52:57.020085Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql/mysqlx.sock
2022-01-09T07:52:57.235893Z 0 [Warning] [MY-010068] [Server] CA certificate /etc/mysql/ssl-internal/ca.crt is self signed.
2022-01-09T07:52:57.236551Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-01-09T07:52:57.248606Z 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_admin. See below for the description of exact issue.
2022-01-09T07:52:57.249279Z 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2022-01-09T07:52:57.249926Z 0 [System] [MY-013603] [Server] No TLS configuration was given for channel mysql_admin; re-using TLS configuration of channel mysql_main.
2022-01-09T07:52:57.329153Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2022-01-09T07:52:57.331613Z 0 [Note] [MY-000000] [Galera] Server initialized
2022-01-09T07:52:57.332817Z 0 [Note] [MY-000000] [WSREP] Server status change initializing -> initialized
2022-01-09T07:52:57.334021Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T07:52:57.335565Z 1 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
2022-01-09T07:52:57.343084Z 10 [Note] [MY-000000] [WSREP] Starting applier thread 10
2022-01-09T07:52:57.345104Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22-13.1' socket: '/tmp/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel13, Revision a48e6d5, WSREP version 26.4.3.
2022-01-09T07:52:57.345138Z 0 [System] [MY-013292] [Server] Admin interface ready for connections, address: '240b:c0e0:202:55d8:b484:2:0:4574' port: 33062
2022-01-09T07:52:57.346952Z 9 [Note] [MY-000000] [WSREP] Recovered cluster id 15efe54f-7121-11ec-9225-1b13c8eeac3c
2022-01-09T07:52:57.350949Z 1 [Note] [MY-000000] [WSREP] Server status change initialized -> joined
2022-01-09T07:52:57.350984Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T07:52:57.351019Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T07:52:57.360938Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 7
2022-01-09T07:52:57.361007Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 7
2022-01-09T07:52:57.361049Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2022-01-09T07:52:57.361123Z 1 [Note] [MY-000000] [Galera] Server mysql-0130-pxc-db-pxc-0 synced with group
2022-01-09T07:52:57.361152Z 1 [Note] [MY-000000] [WSREP] Server status change joined -> synced
2022-01-09T07:52:57.361184Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2022-01-09T07:52:57.361217Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T07:53:43.108861Z 23 [Warning] [MY-010235] [Server] Following users were specified in CREATE USER IF NOT EXISTS but they already exist. Corresponding entry in binary log used default authentication plugin 'mysql_native_password' to rewrite authentication information (if any) for them: 'moni****'
2022-01-09T07:53:43.177391Z 24 [Warning] [MY-010235] [Server] Following users were specified in CREATE USER IF NOT EXISTS but they already exist. Corresponding entry in binary log used default authentication plugin 'mysql_native_password' to rewrite authentication information (if any) for them: 'xtra*****'
2022-01-09T07:54:33.302902Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://[240b:c0e0:202:55d8:b484:2:0:4a63]:44726 local endpoint ssl://[240b:c0e0:202:55d8:b484:2:0:4574]:4567 cipher: TLS_AES_256_GCM_SHA384 compression: none
2022-01-09T07:54:33.303952Z 0 [Note] [MY-000000] [Galera] (27d97bde-b4ff, 'ssl://[::]:4567') connection established to 61f384c9-8e12 ssl://[240b:c0e0:202:55d8:b484:2:0:4a63]:4567
2022-01-09T07:54:33.305006Z 0 [Note] [MY-000000] [Galera] (27d97bde-b4ff, 'ssl://[::]:4567') turning message relay requesting on, nonlive peers:
2022-01-09T07:54:33.792544Z 0 [Note] [MY-000000] [Galera] declaring 61f384c9-8e12 at ssl://[240b:c0e0:202:55d8:b484:2:0:4a63]:4567 stable
2022-01-09T07:54:33.794423Z 0 [Note] [MY-000000] [Galera] Node 27d97bde-b4ff state primary
2022-01-09T07:54:33.794976Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,27d97bde-b4ff,2)
memb {
27d97bde-b4ff,0
61f384c9-8e12,0
}
joined {
}
left {
}
partitioned {
}
)
2022-01-09T07:54:33.795031Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-01-09T07:54:33.797558Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-01-09T07:54:33.797780Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 6241b714-7121-11ec-9c94-1ff4a21571ef
2022-01-09T07:54:33.801477Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 6241b714-7121-11ec-9c94-1ff4a21571ef
2022-01-09T07:54:33.801871Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 6241b714-7121-11ec-9c94-1ff4a21571ef from 0 (mysql-0130-pxc-db-pxc-0)
2022-01-09T07:54:34.294843Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 6241b714-7121-11ec-9c94-1ff4a21571ef from 1 (mysql-0130-pxc-db-pxc-1)
2022-01-09T07:54:34.294906Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 1,
members = 1/2 (primary/total),
act_id = 14,
last_appl. = 6,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 15efe54f-7121-11ec-9225-1b13c8eeac3c
2022-01-09T07:54:34.295033Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2022-01-09T07:54:34.295187Z 1 [Note] [MY-000000] [Galera] ####### processing CC 15, local, ordered
2022-01-09T07:54:34.295242Z 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 14 upto current CC event 15 upto:14
2022-01-09T07:54:34.295266Z 1 [Note] [MY-000000] [Galera] Drain monitors from 14 up to 14
2022-01-09T07:54:34.295292Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 27d97bde-7121-11ec-b4ff-8b12ad02da43
2022-01-09T07:54:34.295355Z 1 [Note] [MY-000000] [Galera] Skipping cert index reset
2022-01-09T07:54:34.295386Z 1 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2022-01-09T07:54:34.295405Z 1 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 14 -> 15
2022-01-09T07:54:34.295460Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-01-09T07:54:34.300425Z 1 [Note] [MY-000000] [Galera] ================================================
View:
id: 15efe54f-7121-11ec-9225-1b13c8eeac3c:15
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(2):
0: 27d97bde-7121-11ec-b4ff-8b12ad02da43, mysql-0130-pxc-db-pxc-0
1: 61f384c9-7121-11ec-8e12-cb5002945df0, mysql-0130-pxc-db-pxc-1
=================================================
2022-01-09T07:54:34.300483Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T07:54:34.304631Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 15
2022-01-09T07:54:34.304673Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 8
2022-01-09T07:54:34.304693Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2022-01-09T07:54:35.789086Z 0 [Note] [MY-000000] [Galera] Member 1.0 (mysql-0130-pxc-db-pxc-1) requested state transfer from '*any*'. Selected 0.0 (mysql-0130-pxc-db-pxc-0)(SYNCED) as donor.
2022-01-09T07:54:35.789142Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> DONOR/DESYNCED (TO: 15)
2022-01-09T07:54:35.789242Z 1 [Note] [MY-000000] [Galera] Detected STR version: 1, req_len: 175, req: STRv1
2022-01-09T07:54:35.789364Z 1 [Note] [MY-000000] [Galera] Cert index preload: 8 -> 15
2022-01-09T07:54:35.790353Z 1 [Note] [MY-000000] [Galera] IST sender using ssl
2022-01-09T07:54:35.802193Z 1 [Note] [MY-000000] [WSREP] Server status change synced -> donor
2022-01-09T07:54:35.802288Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T07:54:35.802285Z 0 [Note] [MY-000000] [Galera] async IST sender starting to serve ssl://[240b:c0e0:202:55d8:b484:2:0:4a63]:4568 sending 8-15, preload starts from 8
2022-01-09T07:54:35.802772Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '[240b:c0e0:202:55d8:b484:2:0:4a63]:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.22-13.1' --binlog 'binlog' --gtid '15efe54f-7121-11ec-9225-1b13c8eeac3c:15' )
2022-01-09T07:54:35.803824Z 0 [Note] [MY-000000] [Galera] IST sender 8 -> 15
2022-01-09T07:54:35.811479Z 1 [Note] [MY-000000] [WSREP] DONOR thread signaled with 0
2022-01-09T07:54:36.321525Z 0 [Note] [MY-000000] [Galera] (27d97bde-b4ff, 'ssl://[::]:4567') turning message relay requesting off
2022-01-09T07:54:36.801232Z 0 [Note] [MY-000000] [WSREP-SST] 2022/01/09 07:54:36 socat[878] E Failed to set SNI host ""
2022-01-09T07:54:36.803283Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2022-01-09T07:54:36.803481Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node: exit codes: 0 1
2022-01-09T07:54:36.803562Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296
2022-01-09T07:54:36.803636Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************************************************
2022-01-09T07:54:36.803810Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32
2022-01-09T07:54:36.827423Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '[240b:c0e0:202:55d8:b484:2:0:4a63]:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.22-13.1' --binlog 'binlog' --gtid '15efe54f-7121-11ec-9225-1b13c8eeac3c:15' : 32 (Broken pipe)
2022-01-09T07:54:36.831906Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32
2022-01-09T07:54:36.831951Z 0 [Note] [MY-000000] [WSREP] Server status change donor -> joined
2022-01-09T07:54:36.831988Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T07:54:36.832081Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '[240b:c0e0:202:55d8:b484:2:0:4a63]:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.22-13.1' --binlog 'binlog' --gtid '15efe54f-7121-11ec-9225-1b13c8eeac3c:15'
2022-01-09T07:54:36.832805Z 0 [Warning] [MY-000000] [Galera] 0.0 (mysql-0130-pxc-db-pxc-0): State transfer to 1.0 (mysql-0130-pxc-db-pxc-1) failed: -32 (Broken pipe)
2022-01-09T07:54:36.832857Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 15)
2022-01-09T07:54:36.833449Z 0 [Note] [MY-000000] [Galera] Member 0.0 (mysql-0130-pxc-db-pxc-0) synced with group.
2022-01-09T07:54:36.833478Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 15)
2022-01-09T07:54:36.833587Z 1 [Note] [MY-000000] [Galera] Server mysql-0130-pxc-db-pxc-0 synced with group
2022-01-09T07:54:36.833641Z 1 [Note] [MY-000000] [WSREP] Server status change joined -> synced
2022-01-09T07:54:36.833664Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2022-01-09T07:54:36.833681Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-09T07:54:37.836357Z 0 [Note] [MY-000000] [Galera] forgetting 61f384c9-8e12 (ssl://[240b:c0e0:202:55d8:b484:2:0:4a63]:4567)
2022-01-09T07:54:37.836444Z 0 [Note] [MY-000000] [Galera] Node 27d97bde-b4ff state primary
Please help to resolve issue !!