Hello all,
I have had an existing cluster of 5 nodes for over a year that has been totally solid. The datacenter had some hardware networking issues a few days ago, and since then only 3 of the nodes have been participating in the cluster. I have already verified that all 5 machines can talk to one another over the network.
The following has not worked: reboot the 2 “missing” nodes, manually join the cluster from those nodes, wipe all datadir files on those nodes and try to rejoin, even setting up a new node (same exact machine config script) and attempting to join fails.
The pertinent error seems to be the following:
WSREP_SST: [ERROR] Cleanup after exit with status:1
[ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.225.29' --datadir '/var/lib/percona-xtradb-cluster/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '8950' '' : 1 (Operation not permitted)
[ERROR] WSREP: Failed to read uuid:seqno from joiner script.
[ERROR] WSREP: SST script aborted with error 1 (Operation not permitted)
[ERROR] WSREP: SST failed: 1 (Operation not permitted)
Host is Ubuntu 18.04, Percona XtraDB Cluster mysqld 5.7.20-18-18, WSREP version 29.24, xtrabackup version 2.4.9.
Here is the error log output from the brand new joiner when first starting up mysqld:
2023-03-31T01:09:43.343373Z mysqld_safe Logging to '/var/log/mysqld.log'.
2023-03-31T01:09:43.358300Z mysqld_safe Starting mysqld daemon with databases from /var/lib/percona-xtradb-cluster
2023-03-31T01:09:43.364663Z mysqld_safe Skipping wsrep-recover for 460e0a7e-cf5f-11ed-9e65-eb978b93c37e:0 pair
2023-03-31T01:09:43.365450Z mysqld_safe Assigning 460e0a7e-cf5f-11ed-9e65-eb978b93c37e:0 to wsrep_start_position
2023-03-31T01:09:43.527091Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server optio
n (see documentation for more details).
2023-03-31T01:09:43.527970Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.20-18-18-log) starting as process 8950 ...
2023-03-31T01:09:43.529457Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this My
SQL server acts as a master and has his hostname changed!! Please use '--log-bin=bbm-db5-bin' to avoid this problem.
2023-03-31T01:09:43.529514Z 0 [Note] WSREP: Setting wsrep_ready to false
2023-03-31T01:09:43.529525Z 0 [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.
2023-03-31T01:09:43.529528Z 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera3/libgalera_smm.so'
2023-03-31T01:09:43.532187Z 0 [Note] WSREP: wsrep_load(): Galera 3.24(r) by Codership Oy <info@codership.com> loaded successfully.
2023-03-31T01:09:43.532239Z 0 [Note] WSREP: CRC-32C: using hardware acceleration.
2023-03-31T01:09:43.532618Z 0 [Note] WSREP: Found saved state: 460e0a7e-cf5f-11ed-9e65-eb978b93c37e:0, safe_to_bootstrap: 1
2023-03-31T01:09:43.533420Z 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/percona-xtradb-cluster/; base_host = 192.168.225.29; base_port
= 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period =
PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = P
T1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/percona-xtradb-cluster/; gcache.free
ze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/percona-xtradb-cluster//g
alera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit
= 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_l
imit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; p
c.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.back
end = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; rep
l.proto_max = 7; socket.checksum = 2; socket.recv_buf_size = 212992;
2023-03-31T01:09:43.542931Z 0 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1
2023-03-31T01:09:43.542986Z 0 [Note] WSREP: Preparing to initiate SST/IST
2023-03-31T01:09:43.542989Z 0 [Note] WSREP: Starting replication
2023-03-31T01:09:43.542999Z 0 [Note] WSREP: Setting initial position to 460e0a7e-cf5f-11ed-9e65-eb978b93c37e:0
2023-03-31T01:09:43.543192Z 0 [Note] WSREP: Using CRC-32C for message checksums.
2023-03-31T01:09:43.543316Z 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2023-03-31T01:09:43.543423Z 0 [Warning] WSREP: Fail to access the file (/var/lib/percona-xtradb-cluster//gvwstate.dat) error (No such file or director
y). It is possible if node is booting for first time or re-booting after a graceful shutdown
2023-03-31T01:09:43.543428Z 0 [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a
graceful shutdown
2023-03-31T01:09:43.543586Z 0 [Note] WSREP: GMCast version 0
2023-03-31T01:09:43.544014Z 0 [Note] WSREP: (b85f91fb, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2023-03-31T01:09:43.544023Z 0 [Note] WSREP: (b85f91fb, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2023-03-31T01:09:43.544267Z 0 [Note] WSREP: EVS version 0
2023-03-31T01:09:43.544346Z 0 [Note] WSREP: gcomm: connecting to group 'Cluster', peer 'db1:,db2:,db3:,db4:'
2023-03-31T01:09:43.792728Z 0 [Note] WSREP: (b85f91fb, 'tcp://0.0.0.0:4567') connection established to 9d2ae1ea tcp://192.168.189.232:4567
2023-03-31T01:09:43.792835Z 0 [Note] WSREP: (b85f91fb, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2023-03-31T01:09:43.935579Z 0 [Note] WSREP: (b85f91fb, 'tcp://0.0.0.0:4567') connection established to 183bfb63 tcp://192.168.217.43:4567
2023-03-31T01:09:44.307152Z 0 [Note] WSREP: (b85f91fb, 'tcp://0.0.0.0:4567') connection established to 74e874a5 tcp://192.168.211.207:4567
2023-03-31T01:09:44.307396Z 0 [Note] WSREP: (b85f91fb, 'tcp://0.0.0.0:4567') connection established to 74e874a5 tcp://192.168.211.207:4567
2023-03-31T01:09:45.047230Z 0 [Note] WSREP: declaring 183bfb63 at tcp://192.168.217.43:4567 stable
2023-03-31T01:09:45.047277Z 0 [Note] WSREP: declaring 74e874a5 at tcp://192.168.211.207:4567 stable
2023-03-31T01:09:45.047281Z 0 [Note] WSREP: declaring 9d2ae1ea at tcp://192.168.189.232:4567 stable
2023-03-31T01:09:45.047974Z 0 [Note] WSREP: Node 183bfb63 state primary
2023-03-31T01:09:45.048678Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,183bfb63,605)
memb {
183bfb63,0
74e874a5,0
9d2ae1ea,0
b85f91fb,0
}
joined {
}
left {
}
partitioned {
}
)
2023-03-31T01:09:45.048691Z 0 [Note] WSREP: Save the discovered primary-component to disk
2023-03-31T01:09:45.048833Z 0 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.211.99:4567
2023-03-31T01:09:45.048839Z 0 [Note] WSREP: discarding pending addr proto entry 0x55b79019e660
2023-03-31T01:09:45.545338Z 0 [Note] WSREP: gcomm: connected
2023-03-31T01:09:45.545590Z 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2023-03-31T01:09:45.545653Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 4
2023-03-31T01:09:45.545681Z 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2023-03-31T01:09:45.545688Z 0 [Note] WSREP: Waiting for SST/IST to complete.
2023-03-31T01:09:45.545933Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: b945a355-cf60-11ed-ad72-f2c51290ead4
2023-03-31T01:09:45.545945Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: b945a355-cf60-11ed-ad72-f2c51290ead4 from 0 (db1)
2023-03-31T01:09:45.546033Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: b945a355-cf60-11ed-ad72-f2c51290ead4 from 1 (db3)
2023-03-31T01:09:45.546077Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: b945a355-cf60-11ed-ad72-f2c51290ead4 from 2 (db4)
2023-03-31T01:09:45.546636Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: b945a355-cf60-11ed-ad72-f2c51290ead4 from 3 (db5)
2023-03-31T01:09:45.546652Z 0 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 478,
members = 3/4 (primary/total),
act_id = 66183330,
last_appl. = -1,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 1f59c4f3-9ae3-11e9-acd7-d21d138010ab
2023-03-31T01:09:45.546660Z 0 [Note] WSREP: Flow-control interval: [200, 200]
2023-03-31T01:09:45.546663Z 0 [Note] WSREP: Trying to continue unpaused monitor
2023-03-31T01:09:45.546666Z 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 66183330)
2023-03-31T01:09:45.546737Z 1 [Note] WSREP: State transfer required:
Group state: 1f59c4f3-9ae3-11e9-acd7-d21d138010ab:66183330
Local state: 460e0a7e-cf5f-11ed-9e65-eb978b93c37e:0
2023-03-31T01:09:45.546751Z 1 [Note] WSREP: New cluster view: global state: 1f59c4f3-9ae3-11e9-acd7-d21d138010ab:66183330, view# 479: Primary, number
of nodes: 4, my index: 3, protocol version 3
2023-03-31T01:09:45.546755Z 1 [Note] WSREP: Setting wsrep_ready to true
2023-03-31T01:09:45.546758Z 1 [Warning] WSREP: Gap in state sequence. Need state transfer.
2023-03-31T01:09:45.546760Z 1 [Note] WSREP: Setting wsrep_ready to false
2023-03-31T01:09:45.546896Z 0 [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.225
.29' --datadir '/var/lib/percona-xtradb-cluster/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '8950' '' )
2023-03-31T01:09:46.128281Z 1 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|192.168.225.29:4444/xtrabackup_sst//1
2023-03-31T01:09:46.128328Z 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-03-31T01:09:46.128355Z 1 [Note] WSREP: REPL Protocols: 7 (3, 2)
2023-03-31T01:09:46.128368Z 1 [Note] WSREP: Assign initial position for certification: 66183330, protocol version: 3
2023-03-31T01:09:46.128427Z 0 [Note] WSREP: Service thread queue flushed.
2023-03-31T01:09:46.128459Z 1 [Note] WSREP: Check if state gap can be serviced using IST
2023-03-31T01:09:46.128474Z 1 [Note] WSREP: Local UUID: 460e0a7e-cf5f-11ed-9e65-eb978b93c37e != Group UUID: 1f59c4f3-9ae3-11e9-acd7-d21d138010ab
2023-03-31T01:09:46.128514Z 1 [Note] WSREP: State gap can't be serviced using IST. Switching to SST
2023-03-31T01:09:46.128518Z 1 [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (460e0a7e-cf5f-11ed-9e65-eb978b93c37e)
does not match group state UUID (1f59c4f3-9ae3-11e9-acd7-d21d138010ab): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():538. IST will be unavailable.
2023-03-31T01:09:46.129446Z 0 [Note] WSREP: Member 3.0 (db5) requested state transfer from '*any*'. Selected 0.0 (db1)(SYNCED) as donor.
2023-03-31T01:09:46.129456Z 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 66183333)
2023-03-31T01:09:46.129479Z 1 [Note] WSREP: Requesting state transfer: success, donor: 0
2023-03-31T01:09:46.129504Z 1 [Note] WSREP: GCache history reset: 460e0a7e-cf5f-11ed-9e65-eb978b93c37e:0 -> 1f59c4f3-9ae3-11e9-acd7-d21d138010ab:66183
330
2023-03-31T01:09:46.465943Z WSREP_SST: [ERROR] Cleanup after exit with status:1
2023-03-31T01:09:46.471758Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.225.29' --datadi
r '/var/lib/percona-xtradb-cluster/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '8950' '' : 1 (Operation not permitted)
2023-03-31T01:09:46.471795Z 0 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2023-03-31T01:09:46.471799Z 0 [ERROR] WSREP: SST script aborted with error 1 (Operation not permitted)
2023-03-31T01:09:46.471822Z 0 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2023-03-31T01:09:46.471844Z 0 [ERROR] Aborting
And here is the error log from the donor when that brand new machine attempted to join:
2023-03-31T01:09:43.794240Z 0 [Note] WSREP: (183bfb63, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.225.29:4567
2023-03-31T01:09:43.936484Z 0 [Note] WSREP: (183bfb63, 'tcp://0.0.0.0:4567') connection established to b85f91fb tcp://192.168.225.29:4567
2023-03-31T01:09:45.048460Z 0 [Note] WSREP: declaring 74e874a5 at tcp://192.168.211.207:4567 stable
2023-03-31T01:09:45.048615Z 0 [Note] WSREP: declaring 9d2ae1ea at tcp://192.168.189.232:4567 stable
2023-03-31T01:09:45.048654Z 0 [Note] WSREP: declaring b85f91fb at tcp://192.168.225.29:4567 stable
2023-03-31T01:09:45.049165Z 0 [Note] WSREP: Node 183bfb63 state primary
2023-03-31T01:09:45.049887Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,183bfb63,605)
memb {
183bfb63,0
74e874a5,0
9d2ae1ea,0
b85f91fb,0
}
joined {
}
left {
}
partitioned {
}
)
2023-03-31T01:09:45.049945Z 0 [Note] WSREP: Save the discovered primary-component to disk
2023-03-31T01:09:45.051151Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 4
2023-03-31T01:09:45.051369Z 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: b945a355-cf60-11ed-ad72-f2c51290ead4
2023-03-31T01:09:45.051954Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: b945a355-cf60-11ed-ad72-f2c51290ead4
2023-03-31T01:09:45.052392Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: b945a355-cf60-11ed-ad72-f2c51290ead4 from 0 (db1)
2023-03-31T01:09:45.052446Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: b945a355-cf60-11ed-ad72-f2c51290ead4 from 1 (db3)
2023-03-31T01:09:45.052484Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: b945a355-cf60-11ed-ad72-f2c51290ead4 from 2 (db4)
2023-03-31T01:09:45.547924Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: b945a355-cf60-11ed-ad72-f2c51290ead4 from 3 (db5)
2023-03-31T01:09:45.548050Z 0 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 478,
members = 3/4 (primary/total),
act_id = 66183330,
last_appl. = 66183203,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 1f59c4f3-9ae3-11e9-acd7-d21d138010ab
2023-03-31T01:09:45.548125Z 0 [Note] WSREP: Flow-control interval: [200, 200]
2023-03-31T01:09:45.548227Z 5 [Note] WSREP: REPL Protocols: 7 (3, 2)
2023-03-31T01:09:45.548258Z 5 [Note] WSREP: REPL Protocols: 7 (3, 2)
2023-03-31T01:09:45.548284Z 5 [Note] WSREP: New cluster view: global state: 1f59c4f3-9ae3-11e9-acd7-d21d138010ab:66183330, view# 479: Primary, number of nodes: 4, my index: 0, protocol version 3
2023-03-31T01:09:45.548301Z 5 [Note] WSREP: Setting wsrep_ready to true
2023-03-31T01:09:45.548319Z 5 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 1) (Increment: 3 -> 4)
2023-03-31T01:09:45.548343Z 5 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-03-31T01:09:45.548659Z 5 [Note] WSREP: Assign initial position for certification: 66183330, protocol version: 3
2023-03-31T01:09:45.548734Z 0 [Note] WSREP: Service thread queue flushed.
2023-03-31T01:09:46.130659Z 0 [Note] WSREP: Member 3.0 (db5) requested state transfer from '*any*'. Selected 0.0 (db1)(SYNCED) as donor.
2023-03-31T01:09:46.130785Z 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 66183333)
2023-03-31T01:09:46.130886Z 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-03-31T01:09:46.131100Z 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.225.29:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/percona-xtradb-cluster/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.33-36-57' '' --gtid '1f59c4f3-9ae3-11e9-acd7-d21d138010ab:66183333' )
2023-03-31T01:09:46.131484Z 4 [Note] WSREP: DONOR thread signaled with 0
2023-03-31T01:09:46.415042Z WSREP_SST: [INFO] Streaming with xbstream
2023-03-31T01:09:46.474832Z 0 [Note] WSREP: declaring 74e874a5 at tcp://192.168.211.207:4567 stable
2023-03-31T01:09:46.474952Z 0 [Note] WSREP: declaring 9d2ae1ea at tcp://192.168.189.232:4567 stable
2023-03-31T01:09:46.474980Z 0 [Note] WSREP: forgetting b85f91fb (tcp://192.168.225.29:4567)
2023-03-31T01:09:46.475515Z 0 [Note] WSREP: Node 183bfb63 state primary
2023-03-31T01:09:46.475978Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,183bfb63,606)
memb {
183bfb63,0
74e874a5,0
9d2ae1ea,0
}
joined {
}
left {
}
partitioned {
b85f91fb,0
}
)
2023-03-31T01:09:46.476033Z 0 [Note] WSREP: Save the discovered primary-component to disk
2023-03-31T01:09:46.477200Z 0 [Note] WSREP: forgetting b85f91fb (tcp://192.168.225.29:4567)
2023-03-31T01:09:46.477260Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
2023-03-31T01:09:46.477529Z 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: ba1f4205-cf60-11ed-a760-a34f49cdcb94
2023-03-31T01:09:46.477963Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: ba1f4205-cf60-11ed-a760-a34f49cdcb94
2023-03-31T01:09:46.478297Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: ba1f4205-cf60-11ed-a760-a34f49cdcb94 from 0 (db1)
2023-03-31T01:09:46.478354Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: ba1f4205-cf60-11ed-a760-a34f49cdcb94 from 1 (db3)
2023-03-31T01:09:46.478398Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: ba1f4205-cf60-11ed-a760-a34f49cdcb94 from 2 (db4)
2023-03-31T01:09:46.478429Z 0 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 479,
members = 3/3 (primary/total),
act_id = 66183333,
last_appl. = 66183328,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 1f59c4f3-9ae3-11e9-acd7-d21d138010ab
2023-03-31T01:09:46.478457Z 0 [Note] WSREP: Flow-control interval: [173, 173]
2023-03-31T01:09:46.478572Z 9 [Note] WSREP: REPL Protocols: 7 (3, 2)
2023-03-31T01:09:46.478593Z 9 [Note] WSREP: REPL Protocols: 7 (3, 2)
2023-03-31T01:09:46.478655Z 9 [Note] WSREP: New cluster view: global state: 1f59c4f3-9ae3-11e9-acd7-d21d138010ab:66183333, view# 480: Primary, number of nodes: 3, my index: 0, protocol version 3
2023-03-31T01:09:46.478667Z 9 [Note] WSREP: Setting wsrep_ready to true
2023-03-31T01:09:46.478673Z 9 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 1) (Increment: 4 -> 3)
2023-03-31T01:09:46.478751Z 9 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-03-31T01:09:46.478819Z 9 [Note] WSREP: Assign initial position for certification: 66183333, protocol version: 3
2023-03-31T01:09:46.478853Z 0 [Note] WSREP: Service thread queue flushed.
2023-03-31T01:09:47.298233Z 0 [Note] WSREP: (183bfb63, 'tcp://0.0.0.0:4567') turning message relay requesting off
2023-03-31T01:09:51.799027Z 0 [Note] WSREP: cleaning up b85f91fb (tcp://192.168.225.29:4567)
I really appreciate the help here. Thanks all.