[root@prod-mysql-node01 log]# tail -f mysqld.log
2024-02-27T11:04:26.750712Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 31
2024-02-27T11:04:26.750742Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 31
2024-02-27T11:04:26.750751Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2024-02-27T11:04:26.758920Z 0 [Note] [MY-000000] [Galera] async IST sender served
2024-02-27T11:04:31.931985Z 0 [Note] [MY-000000] [Galera]  cleaning up f6f9700a-8e5f (tcp://10.185.60.131:4567)
2024-02-27T20:59:44.640038Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.90561S), skipping check
2024-02-27T22:28:44.338342Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT5.34577S), skipping check
2024-02-27T22:53:27.361564Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT2.93416S), skipping check
2024-02-28T00:37:44.753322Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.19042S), skipping check
2024-02-28T00:37:47.743922Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT1.99047S), skipping check
2024-02-28T05:35:31.963034Z 0 [Note] [MY-000000] [Galera] (42727aa1-bb56, 'tcp://0.0.0.0:4567') connection established to 305779ab-9be3 tcp://10.185.60.131:4567
2024-02-28T05:35:31.963393Z 0 [Note] [MY-000000] [Galera] (42727aa1-bb56, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2024-02-28T05:35:32.278402Z 0 [Note] [MY-000000] [Galera] declaring 305779ab-9be3 at tcp://10.185.60.131:4567 stable
2024-02-28T05:35:32.278659Z 0 [Note] [MY-000000] [Galera] Node 42727aa1-bb56 state primary
2024-02-28T05:35:32.285185Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,305779ab-9be3,34)
memb {
        305779ab-9be3,0
        42727aa1-bb56,0
        }
joined {
        }
left {
        }
partitioned {
        }
)
2024-02-28T05:35:32.285213Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-02-28T05:35:32.413871Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2024-02-28T05:35:32.413914Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: Waiting for state UUID.
2024-02-28T05:35:32.464519Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 30a426f7-d5fb-11ee-98e9-5bbd12b6fcb1
2024-02-28T05:35:32.464802Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 30a426f7-d5fb-11ee-98e9-5bbd12b6fcb1 from 0 (prod-mysql-node02)
2024-02-28T05:35:32.464846Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 30a426f7-d5fb-11ee-98e9-5bbd12b6fcb1 from 1 (prod-mysql-node01.ipsl.co.ke)
2024-02-28T05:35:32.464858Z 0 [Note] [MY-000000] [Galera] Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 31,
        members    = 1/2 (primary/total),
        act_id     = 31,
        last_appl. = 0,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = e0b97096-d542-11ee-b5c5-324197619ff9
2024-02-28T05:35:32.464913Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2024-02-28T05:35:32.464951Z 1 [Note] [MY-000000] [Galera] ####### processing CC 32, local, ordered
2024-02-28T05:35:32.464964Z 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 31 upto current CC event 32 upto:31
2024-02-28T05:35:32.464972Z 1 [Note] [MY-000000] [Galera] Drain monitors from 31 up to 31
2024-02-28T05:35:32.464981Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 42727aa1-d541-11ee-bb56-9bd874420b61
2024-02-28T05:35:32.464988Z 1 [Note] [MY-000000] [Galera] Skipping cert index reset
2024-02-28T05:35:32.464994Z 1 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-02-28T05:35:32.465002Z 1 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 31 -> 32
2024-02-28T05:35:32.465080Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-28T05:35:32.477332Z 1 [Note] [MY-000000] [Galera] ================================================
View:
  id: e0b97096-d542-11ee-b5c5-324197619ff9:32
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 1
  members(2):
        0: 305779ab-d5fb-11ee-9be3-9efff664fe2d, prod-mysql-node02
        1: 42727aa1-d541-11ee-bb56-9bd874420b61, prod-mysql-node01.ipsl.co.ke
=================================================
2024-02-28T05:35:32.477360Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-28T05:35:32.482773Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 32
2024-02-28T05:35:32.482800Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 32
2024-02-28T05:35:32.482809Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2024-02-28T05:35:33.475226Z 0 [Note] [MY-000000] [Galera] Member 0.0 (prod-mysql-node02) requested state transfer from '*any*'. Selected 1.0 (prod-mysql-node01.ipsl.co.ke)(SYNCED) as donor.
2024-02-28T05:35:33.475286Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> DONOR/DESYNCED (TO: 32)
2024-02-28T05:35:33.475315Z 1 [Note] [MY-000000] [Galera] Detected STR version: 1, req_len: 133, req: STRv1
2024-02-28T05:35:33.475344Z 1 [Note] [MY-000000] [Galera] Cert index preload: 32 -> 32
2024-02-28T05:35:33.475849Z 1 [Note] [MY-000000] [WSREP] Server status change synced -> donor
2024-02-28T05:35:33.475888Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-28T05:35:33.475915Z 0 [Note] [MY-000000] [Galera] async IST sender starting to serve tcp://10.185.60.131:4568 sending 32-32, preload starts from 32
2024-02-28T05:35:33.476034Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.185.60.131:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/data01/mysql_data/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.32-24.1'  --binlog 'binlog' --gtid 'e0b97096-d542-11ee-b5c5-324197619ff9:32' )
2024-02-28T05:35:33.476474Z 1 [Note] [MY-000000] [WSREP] DONOR thread signaled with -1290
2024-02-28T05:35:33.476516Z 1 [Warning] [MY-000000] [Galera] SST preparation failed
2024-02-28T05:35:33.476536Z 1 [Note] [MY-000000] [WSREP] Server status change donor -> joined
2024-02-28T05:35:33.476548Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-28T05:35:33.476592Z 1 [ERROR] [MY-000000] [Galera] SST failed: 1
2024-02-28T05:35:33.476479Z 0 [Note] [MY-000000] [Galera] IST sender 32 -> 32
2024-02-28T05:35:33.476991Z 0 [Warning] [MY-000000] [Galera] 1.0 (prod-mysql-node01.ipsl.co.ke): State transfer to 0.0 (prod-mysql-node02) failed: -125 (Operation canceled)
2024-02-28T05:35:33.477007Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 32)
2024-02-28T05:35:33.477047Z 0 [Note] [MY-000000] [Galera] Processing event queue:... -nan% (0/0 events) complete.
2024-02-28T05:35:34.477914Z 0 [Note] [MY-000000] [Galera] Deferred close timer started for socket with remote endpoint: tcp://10.185.60.131:35760
2024-02-28T05:35:34.477915Z 0 [Note] [MY-000000] [Galera] Member 1.0 (prod-mysql-node01.ipsl.co.ke) synced with group.
2024-02-28T05:35:34.477986Z 0 [Note] [MY-000000] [Galera] forgetting 305779ab-9be3 (tcp://10.185.60.131:4567)
2024-02-28T05:35:34.478010Z 0 [Note] [MY-000000] [Galera] Processing event queue:...100.0% (1/1 events) complete.
2024-02-28T05:35:34.478017Z 0 [Note] [MY-000000] [Galera] Node 42727aa1-bb56 state primary
2024-02-28T05:35:34.478023Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 32)
2024-02-28T05:35:34.478038Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,42727aa1-bb56,35)
memb {
        42727aa1-bb56,0
        }
joined {
        }
left {
        }
partitioned {
        305779ab-9be3,0
        }
)
2024-02-28T05:35:34.478050Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-02-28T05:35:34.478049Z 1 [Note] [MY-000000] [Galera] Server prod-mysql-node01.ipsl.co.ke synced with group
2024-02-28T05:35:34.478065Z 1 [Note] [MY-000000] [WSREP] Server status change joined -> synced
2024-02-28T05:35:34.478071Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2024-02-28T05:35:34.478077Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-28T05:35:34.480941Z 0 [Note] [MY-000000] [Galera] forgetting 305779ab-9be3 (tcp://10.185.60.131:4567)
2024-02-28T05:35:34.480962Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2024-02-28T05:35:34.481050Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 31d7fb32-d5fb-11ee-ae2b-eb323a209fe2
2024-02-28T05:35:34.481107Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 31d7fb32-d5fb-11ee-ae2b-eb323a209fe2
2024-02-28T05:35:34.481113Z 0 [Note] [MY-000000] [Galera] Deferred close timer handle_wait Operation aborted. for 0x7c65250
2024-02-28T05:35:34.481127Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 31d7fb32-d5fb-11ee-ae2b-eb323a209fe2 from 0 (prod-mysql-node01.ipsl.co.ke)
2024-02-28T05:35:34.481130Z 0 [Note] [MY-000000] [Galera] Deferred close timer destruct
2024-02-28T05:35:34.481139Z 0 [Note] [MY-000000] [Galera] Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 32,
        members    = 1/1 (primary/total),
        act_id     = 32,
        last_appl. = 0,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = e0b97096-d542-11ee-b5c5-324197619ff9
2024-02-28T05:35:34.481169Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-02-28T05:35:34.481214Z 1 [Note] [MY-000000] [Galera] ####### processing CC 33, local, ordered
2024-02-28T05:35:34.481228Z 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 32 upto current CC event 33 upto:32
2024-02-28T05:35:34.481235Z 1 [Note] [MY-000000] [Galera] Drain monitors from 32 up to 32
2024-02-28T05:35:34.481243Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 42727aa1-d541-11ee-bb56-9bd874420b61
2024-02-28T05:35:34.481250Z 1 [Note] [MY-000000] [Galera] Skipping cert index reset
2024-02-28T05:35:34.481256Z 1 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-02-28T05:35:34.481263Z 1 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 32 -> 33
2024-02-28T05:35:34.481328Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-28T05:35:34.495624Z 1 [Note] [MY-000000] [Galera] ================================================
View:
  id: e0b97096-d542-11ee-b5c5-324197619ff9:33
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: 42727aa1-d541-11ee-bb56-9bd874420b61, prod-mysql-node01.ipsl.co.ke
=================================================
2024-02-28T05:35:34.495657Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-28T05:35:34.497116Z 0 [Note] [MY-000000] [Galera] async IST sender served
2024-02-28T05:35:34.499303Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 33
2024-02-28T05:35:34.499325Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 33
2024-02-28T05:35:34.499334Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2024-02-28T05:35:35.280868Z 0 [Note] [MY-000000] [Galera] (42727aa1-bb56, 'tcp://0.0.0.0:4567') turning message relay requesting off
2024-02-28T05:35:39.781209Z 0 [Note] [MY-000000] [Galera]  cleaning up 305779ab-9be3 (tcp://10.185.60.131:4567)