Full Cluster Stop and Start for off-business hours

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

We have implemented a Percona XtraDB MySQL cluster, version 8.0.34-26.1, consisting of 3 nodes.

  • In our case, we want to stop the MySQL cluster every day, first by stopping node-1, then node-2, and finally master writer node-0, to save costs.
  • we need to start the MySQL cluster every day when our shift begins.However, to restart the MySQL cluster, we always have to initiate the MySQL bootstrap service on node-0 (the last node where we stopped the MySQL service), but this bootstrap service takes time to start.
  • After starting the bootstrap service, we initiate the MySQL service on the other nodes to join the cluster. This process takes too long as it goes with SST.

Can you please provide the correct way to bring cluster up again with minimal time like IST? we want to keep the cluster offline for 12 hours a day to save cost.

2

@satya

You don’t need to start mysql with Bootstrap every time, One by one just start and stop MySQL is enough. If you have a long downtime, work on your Gcache to avoid SST, but for this, make sure all nodes are not down. One of the nodes should be running.If you completely stop a PXC, the only way to bring it back online is to bootstrap the first node. If you don’t bootstrap the first node, the node will seek out an existing cluster to join. Since there isn’t one running, this node will wait forever and never come online

You can restart PXC in this way on all nodes one by one:

set global pxc_maint_mode=MAINTENANCE;
show  processlist; # make sure all connections are drained.
sudo systemctl stop MySQL

check the log for a clean shutdown

sudo vim /etc/my.cnf.d/galera.cnf

wsrep_sst_donor="<SET-IP-DONOR>" #ip should be the IP of another reader to avoid laod on primary
pxc_maint_mode=MAINTENANCE

sudo systemctl start MySQL

check MySQL got a clean start, check pxc status all nodes should be synced

set global pxc_maint_mode=DISABLED;

Comment sst-donor and pxc_maint mode lines from the config file

3

Thank you for your quick response and appreciation of your answer.

We have below nodes structure
node-0 = primary node (writer)
node-1 = reader node
node-2 = reader node

  • Actually, we don’t have a “galera.cnf” file; we store all our configurations for Percona in the “my.cnf” file. Will those variables you provided work from the “my.cnf” file?

  • In our case, we are going to stop the reader nodes first, which are node-1 and node-2, and then stop the primary node-0. So, when starting the primary node-0 first, which IP should we set for wsrep_sst_donor=“” in the my.cnf file? Because, we are going to start the cluster from the primary node-0 and then the other reader nodes.

  • And also, when we start the reader nodes of node-1 and node-2, which donor IP should we set in the my.cnf file for the variable “wsrep_sst_donor”?

4
  • Actually, we don’t have a “galera.cnf” file; we store all our configurations for Percona in the “my.cnf” file. Will those variables you provided work from the “my.cnf” file?

yes that is fine

  • In our case, we are going to stop the reader nodes first, which are node-1 and node-2, and then stop the primary node-0. So, when starting the primary node-0 first, which IP should we set for wsrep_sst_donor=“” in the my.cnf file? Because, we are going to start the cluster from the primary node-0 and then the other reader nodes.

Why are you stopping the whole cluster, I don’t find the reason for which you need to stop the whole cluster, please clarify this first. for OS patching, for upgrades, or mainitinace it is always in rolling mode, so it goes like reader1, reader2, the Writer.

  • And also, when we start the reader nodes of node-1 and node-2, which donor IP should we set in the my.cnf file for the variable “wsrep_sst_donor”?

when working on reader1 you should add another reader IP as the donor.

Also I rectified my first response read that again as well . thanks

5

Thank you for your answer.

As per your answer, we will not going to stop our master node-0 (primary node) from our cluster.

We are going to stop the node-1 and node-2. After stopping the nodes, whenever we are start mysql service in the node-1 and node-2, those nodes always join the cluster with the SST transfer. and it takes so much time to join the cluster.

Can you please provide the correct way to join the node-1 and node-2 in the cluster with IST?

6

for how long you guys stops the node1 and node2, please post the logs of starting of node1/node2

7

We stop node-1 and node-2 during our working days for approximately 12 hours each day, and over the weekend, it’s nearly 2 days.

Our node-1 server logs as below,

2024-02-29T13:02:06.869237Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 49c87952-cfba-33dd-88qe-431068216069, offset: 1800
2024-02-29T13:02:06.869344Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% ( 0/536870936 bytes) complete.
2024-02-29T13:02:06.869414Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan…100.0% (536870936/536870936 bytes) complete.
2024-02-29T13:02:06.869422Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 5-93
2024-02-29T13:02:06.869444Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% ( 0/34248 bytes) complete.
2024-02-29T13:02:06.869463Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan…100.0% (34248/34248 bytes) complete.
2024-02-29T13:02:06.869470Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 53/142 locked buffers
2024-02-29T13:02:06.869476Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 536848048/536870912
2024-02-29T13:02:06.874567Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /data/; base_host = 10.10.0.2; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /data/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 512M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto;
2024-02-29T13:02:06.885894Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-29T13:02:06.886581Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 49c87952-cfba-33dd-88qe-431068216069:93, protocol version: -1
2024-02-29T13:02:06.886671Z 0 [Note] [MY-000000] [WSREP] Starting replication
2024-02-29T13:02:06.886686Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0
2024-02-29T13:02:06.886697Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 49c87952-cfba-33dd-88qe-431068216069:93
2024-02-29T13:02:06.886755Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2024-02-29T13:02:06.886803Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2024-02-29T13:02:06.886819Z 0 [Note] [MY-000000] [Galera] backend: asio
2024-02-29T13:02:06.886928Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2024-02-29T13:02:06.887033Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2024-02-29T13:02:06.887044Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2024-02-29T13:02:06.887211Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2024-02-29T13:02:06.887354Z 0 [Note] [MY-000000] [Galera] (bdc67601-a648, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
2024-02-29T13:02:06.887365Z 0 [Note] [MY-000000] [Galera] (bdc67601-a648, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
2024-02-29T13:02:06.887656Z 0 [Note] [MY-000000] [Galera] EVS version 1
2024-02-29T13:02:06.887737Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group ‘pxc-cluster’, peer ‘10.10.0.1:,10.10.0.2:,10.10.0.3:’
2024-02-29T13:02:06.888620Z 0 [Note] [MY-000000] [Galera] (bdc67601-a648, ‘tcp://0.0.0.0:4567’) Found matching local endpoint for a connection, blacklisting address tcp://10.10.0.2:4567
2024-02-29T13:02:06.889150Z 0 [Note] [MY-000000] [Galera] (bdc67601-a648, ‘tcp://0.0.0.0:4567’) connection established to dff0c8d9-a692 tcp://10.10.0.1:4567
2024-02-29T13:02:06.889216Z 0 [Note] [MY-000000] [Galera] (bdc67601-a648, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers:
2024-02-29T13:02:06.889470Z 0 [Note] [MY-000000] [Galera] (bdc67601-a648, ‘tcp://0.0.0.0:4567’) connection established to f3bd8647-abb4 tcp://10.10.0.3:4567
2024-02-29T13:02:07.389812Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 → 1
2024-02-29T13:02:07.389926Z 0 [Note] [MY-000000] [Galera] declaring dff0c8d9-a692 at tcp://10.10.0.1:4567 stable
2024-02-29T13:02:07.389937Z 0 [Note] [MY-000000] [Galera] declaring f3bd8647-abb4 at tcp://10.10.0.3:4567 stable
2024-02-29T13:02:07.389986Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 → 1
2024-02-29T13:02:07.390285Z 0 [Note] [MY-000000] [Galera] Node dff0c8d9-a692 state primary
2024-02-29T13:02:07.390713Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,bdc67601-a648,5)
memb {
bdc67601-a648,0
dff0c8d9-a692,0
f3bd8647-abb4,0
}
joined {
}
left {
}
partitioned {
}
)
2024-02-29T13:02:07.390731Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-02-29T13:02:07.888752Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2024-02-29T13:02:07.888889Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2024-02-29T13:02:07.889900Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)
2024-02-29T13:02:07.889931Z 0 [Note] [MY-000000] [Galera] Opened channel ‘pxc-cluster’
2024-02-29T13:02:07.890065Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
2024-02-29T13:02:07.891051Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: be5f9674-d764-11ee-8c0f-67f1baaa786c
2024-02-29T13:02:07.891324Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2024-02-29T13:02:07.891394Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2024-02-29T13:02:07.891669Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: be5f9674-d764-11ee-8c0f-67f1baaa786c
2024-02-29T13:02:07.891952Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: be5f9674-d764-11ee-8c0f-67f1baaa786c from 0 (pxc-cluster-node-2)
2024-02-29T13:02:07.891980Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: be5f9674-d764-11ee-8c0f-67f1baaa786c from 1 (pxc-cluster-node-1)
2024-02-29T13:02:07.891993Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: be5f9674-d764-11ee-8c0f-67f1baaa786c from 2 (pxc-cluster-node-3)
2024-02-29T13:02:07.892002Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 4,
members = 2/3 (primary/total),
act_id = 94,
last_appl. = 90,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 49c87952-cfba-33dd-88qe-431068216069
2024-02-29T13:02:07.892057Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [173, 173]
2024-02-29T13:02:07.892066Z 0 [Note] [MY-000000] [Galera] Shifting OPEN → PRIMARY (TO: 95)
2024-02-29T13:02:07.892166Z 2 [Note] [MY-000000] [Galera] ####### processing CC 95, local, ordered
2024-02-29T13:02:07.892196Z 2 [Note] [MY-000000] [Galera] Maybe drain monitors from 93 upto current CC event 95 upto:93
2024-02-29T13:02:07.892206Z 2 [Note] [MY-000000] [Galera] Drain monitors from 93 up to 93
2024-02-29T13:02:07.892218Z 2 [Note] [MY-000000] [Galera] Process first view: 49c87952-cfba-33dd-88qe-431068216069 my uuid: bdc67601-d702-11ee-a648-2a3ec3d20519
2024-02-29T13:02:07.892241Z 2 [Note] [MY-000000] [Galera] Server pxc-cluster-node-2 connected to cluster at position 49c87952-cfba-33dd-88qe-431068216069:95 with ID bdc67601-d702-11ee-a648-2a3ec3d20519
2024-02-29T13:02:07.892251Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected → connected
2024-02-29T13:02:07.892295Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-29T13:02:07.892327Z 2 [Note] [MY-000000] [Galera] ####### My UUID: bdc67601-d702-11ee-a648-2a3ec3d20519
2024-02-29T13:02:07.892338Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2024-02-29T13:02:07.892379Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-29T13:02:07.892425Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-02-29T13:02:07.892439Z 2 [Note] [MY-000000] [Galera] State transfer required:
Group state: 49c87952-cfba-33dd-88qe-431068216069:95
Local state: 49c87952-cfba-33dd-88qe-431068216069:93
2024-02-29T13:02:07.892445Z 2 [Note] [MY-000000] [WSREP] Server status change connected → joiner
2024-02-29T13:02:07.892458Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-29T13:02:07.892667Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.10.0.2’ --datadir ‘/data/’ --basedir ‘/usr/’ --plugindir ‘/usr/lib64/mysql/plugin/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘4312’ --mysqld-version ‘8.0.34-26.1’ ‘’ )
2024-02-29T13:02:08.560624Z 2 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|10.10.0.2:4444/xtrabackup_sst//1
2024-02-29T13:02:08.560695Z 2 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2024-02-29T13:02:08.560711Z 2 [Note] [MY-000000] [Galera] ####### IST uuid:49c87952-cfba-33dd-88qe-431068216069 f: 94, l: 95, STRv: 3
2024-02-29T13:02:08.560791Z 2 [Note] [MY-000000] [Galera] IST receiver addr using tcp://10.10.0.2:4568
2024-02-29T13:02:08.560975Z 2 [Note] [MY-000000] [Galera] Prepared IST receiver for 94-95, listening at: tcp://10.10.0.2:4568
2024-02-29T13:02:08.564143Z 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc-cluster-node-2) requested state transfer from ‘pxc-cluster-node-1’. Selected 1.0 (pxc-cluster-node-1)(SYNCED) as donor.
2024-02-29T13:02:08.564180Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY → JOINER (TO: 95)
2024-02-29T13:02:08.564258Z 2 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1
2024-02-29T13:02:09.312818Z 0 [Note] [MY-000000] [WSREP-SST] xtrabackup_ist received from donor: Running IST
2024-02-29T13:02:09.312908Z 0 [Note] [MY-000000] [WSREP-SST] Running post-processing…
2024-02-29T13:02:09.318279Z 0 [Note] [MY-000000] [WSREP-SST] Skipping mysql_upgrade (ist)
2024-02-29T13:02:09.318556Z 0 [Note] [MY-000000] [WSREP-SST] …post-processing done
2024-02-29T13:02:09.320409Z 0 [Note] [MY-000000] [WSREP-SST] Galera co-ords from recovery: 49c87952-cfba-33dd-88qe-431068216069:93
2024-02-29T13:02:09.334673Z 3 [Note] [MY-000000] [Galera] Processing SST received
2024-02-29T13:02:09.334732Z 3 [Note] [MY-000000] [WSREP] Server status change joiner → initializing
2024-02-29T13:02:09.334780Z 3 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-29T13:02:09.340451Z 4 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-02-29T13:02:09.889800Z 0 [Note] [MY-000000] [Galera] (bdc67601-a648, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2024-02-29T13:02:10.013366Z 0 [Note] [MY-000000] [Galera] 1.0 (pxc-cluster-node-1): State transfer to 0.0 (pxc-cluster-node-2) complete.
2024-02-29T13:02:10.013981Z 0 [Note] [MY-000000] [Galera] Member 1.0 (pxc-cluster-node-1) synced with group.
2024-02-29T13:06:01.318511Z 4 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-02-29T13:06:07.489789Z 4 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2024-02-29T13:06:07.497856Z 4 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2024-02-29T13:06:07.665239Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-02-29T13:06:07.665319Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-02-29T13:06:07.689445Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2024-02-29T13:06:07.689531Z 0 [Note] [MY-000000] [Galera] Server initialized
2024-02-29T13:06:07.689548Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → initialized
2024-02-29T13:06:07.689573Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-29T13:06:07.689680Z 3 [Note] [MY-000000] [Galera] Recovered position from storage: 49c87952-cfba-33dd-88qe-431068216069:93
2024-02-29T13:06:07.690343Z 10 [Note] [MY-000000] [WSREP] Starting applier thread 10
2024-02-29T13:06:07.690411Z 11 [Note] [MY-000000] [WSREP] Starting applier thread 11
2024-02-29T13:06:07.690727Z 13 [Note] [MY-000000] [WSREP] Starting applier thread 13
2024-02-29T13:06:07.690811Z 12 [Note] [MY-000000] [WSREP] Starting applier thread 12
2024-02-29T13:06:07.690931Z 15 [Note] [MY-000000] [WSREP] Starting applier thread 15
2024-02-29T13:06:07.690929Z 14 [Note] [MY-000000] [WSREP] Starting applier thread 14
2024-02-29T13:06:07.691076Z 16 [Note] [MY-000000] [WSREP] Starting applier thread 16
2024-02-29T13:06:07.692127Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: ‘::’ port: 33060, socket: /var/lib/mysql/mysqlx.sock
2024-02-29T13:06:07.692326Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: ‘8.0.34-26.1’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel26, Revision 0988772, WSREP version 26.1.4.3.
2024-02-29T13:06:07.726440Z 3 [Note] [MY-000000] [Galera] Recovered view from SST:
id: 49c87952-cfba-33dd-88qe-431068216069:93
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: -1
members(3):
0: 46ee61e3-d701-11ee-8ec2-bf66772c58bb, pxc-cluster-node-2
1: dff0c8d9-d700-11ee-a692-eb0dbccf7012, pxc-cluster-node-1
2: f3bd8647-d701-11ee-abb4-bf981a533e92, pxc-cluster-node-3

2024-02-29T13:06:07.726516Z 3 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-29T13:06:07.726754Z 18 [Note] [MY-000000] [WSREP] Recovered cluster id 49c87952-cfba-33dd-88qe-431068216069
2024-02-29T13:06:07.752399Z 3 [Note] [MY-000000] [Galera] SST received: 49c87952-cfba-33dd-88qe-431068216069:93
2024-02-29T13:06:07.752481Z 3 [System] [MY-000000] [WSREP] SST completed
2024-02-29T13:06:07.752593Z 2 [Note] [MY-000000] [Galera] str_proto_ver_: 3 sst_seqno_: 93 cc_seqno: 95 req->ist_len(): 66
2024-02-29T13:06:07.752654Z 2 [Note] [MY-000000] [Galera] Installed new state from SST: 49c87952-cfba-33dd-88qe-431068216069:93
2024-02-29T13:06:07.755110Z 2 [Note] [MY-000000] [Galera] Receiving IST: 2 writesets, seqnos 94-95
2024-02-29T13:06:07.755196Z 0 [Note] [MY-000000] [Galera] ####### IST applying starts with 94
2024-02-29T13:06:07.755312Z 0 [Note] [MY-000000] [Galera] ####### IST current seqno initialized to 94
2024-02-29T13:06:07.755410Z 0 [Note] [MY-000000] [Galera] Receiving IST… 0.0% (0/2 events) complete.
2024-02-29T13:06:07.755506Z 0 [Note] [MY-000000] [Galera] ####### Passing IST CC 94, must_apply: 1, preload: false
2024-02-29T13:06:07.755659Z 0 [Note] [MY-000000] [Galera] First IST (CC) event for rebuilding cert index
2024-02-29T13:06:07.755725Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-02-29T13:06:07.755764Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-29T13:06:07.755858Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 49c87952-cfba-33dd-88qe-431068216069:93, protocol version: 5
2024-02-29T13:06:07.755895Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-02-29T13:06:07.755910Z 0 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 93 → 94
2024-02-29T13:06:07.755933Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-29T13:06:07.755967Z 0 [Note] [MY-000000] [Galera] Recording CC from ist: 94
2024-02-29T13:06:07.755981Z 0 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from ist: 94
2024-02-29T13:06:07.755999Z 0 [Note] [MY-000000] [Galera] Min available from gcache for CC from ist: 5
2024-02-29T13:06:07.756058Z 10 [Note] [MY-000000] [Galera] ================================================
View:
id: 49c87952-cfba-33dd-88qe-431068216069:94
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: -1
members(2):
0: dff0c8d9-d700-11ee-a692-eb0dbccf7012, pxc-cluster-node-1
1: f3bd8647-d701-11ee-abb4-bf981a533e92, pxc-cluster-node-3

2024-02-29T13:06:07.756098Z 0 [Note] [MY-000000] [Galera] IST preload starting at 95
2024-02-29T13:06:07.756125Z 0 [Note] [MY-000000] [Galera] ####### Passing IST CC 95, must_apply: 1, preload: true
2024-02-29T13:06:07.756121Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-29T13:06:07.762089Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-02-29T13:06:07.762121Z 0 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 94 → 95
2024-02-29T13:06:07.762166Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-29T13:06:07.762261Z 0 [Note] [MY-000000] [Galera] Recording CC from ist: 95
2024-02-29T13:06:07.762295Z 0 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from ist: 95
2024-02-29T13:06:07.762307Z 0 [Note] [MY-000000] [Galera] Min available from gcache for CC from ist: 5
2024-02-29T13:06:07.762365Z 0 [Note] [MY-000000] [Galera] Receiving IST…100.0% (2/2 events) complete.
2024-02-29T13:06:07.762407Z 11 [Note] [MY-000000] [Galera]

View:
id: 49c87952-cfba-33dd-88qe-431068216069:95
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(3):
0: bdc67601-d702-11ee-a648-2a3ec3d20519, pxc-cluster-node-2
1: dff0c8d9-d700-11ee-a692-eb0dbccf7012, pxc-cluster-node-1
2: f3bd8647-d701-11ee-abb4-bf981a533e92, pxc-cluster-node-3

2024-02-29T13:06:07.762443Z 11 [Note] [MY-000000] [WSREP] Server status change initialized → joined
2024-02-29T13:06:07.762496Z 11 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-29T13:06:07.762541Z 11 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-29T13:06:07.762758Z 2 [Note] [MY-000000] [Galera] Draining apply monitors after IST up to 95
2024-02-29T13:06:07.768927Z 2 [Note] [MY-000000] [Galera] IST received: 49c87952-cfba-33dd-88qe-431068216069:95
2024-02-29T13:06:07.770910Z 2 [Note] [MY-000000] [Galera] Recording CC from sst: 95
2024-02-29T13:06:07.770938Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from sst: 95
2024-02-29T13:06:07.770947Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from sst: 5
2024-02-29T13:06:07.771666Z 0 [Note] [MY-000000] [Galera] 0.0 (pxc-cluster-node-2): State transfer from 1.0 (pxc-cluster-node-1) complete.
2024-02-29T13:06:07.771702Z 0 [Note] [MY-000000] [Galera] SST leaving flow control
2024-02-29T13:06:07.771712Z 0 [Note] [MY-000000] [Galera] Shifting JOINER → JOINED (TO: 95)
2024-02-29T13:06:07.771769Z 0 [Note] [MY-000000] [Galera] Processing event queue:… -nan% (0/0 events) complete.
2024-02-29T13:06:07.772191Z 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc-cluster-node-2) synced with group.
2024-02-29T13:06:07.772232Z 0 [Note] [MY-000000] [Galera] Processing event queue:…100.0% (1/1 events) complete.
2024-02-29T13:06:07.772242Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 95)
2024-02-29T13:06:07.772276Z 2 [Note] [MY-000000] [Galera] Server pxc-cluster-node-2 synced with group
2024-02-29T13:06:07.772324Z 2 [Note] [MY-000000] [WSREP] Server status change joined → synced
2024-02-29T13:06:07.772337Z 2 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2024-02-29T13:06:07.772349Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.