Hi There,
I am running Percona MySQL Cluster with 3 Nodes 111,112,113 & ProxySQL 114. For the last 3 months we have been having issues, Percona Nodes Disconnect with each other, sometimes it works because other 2 nodes were alive but sometimes not, ProxySQL is not able to connect any node. After getting these incidents I configured PMM to monitor the cluster. Last two incidents show node 113 completely blackouts when connection timeout happens, PMM shows no TCP network connectivity in 113 node but 111 and 112 node do not have TCP connectivity issues. We also have an Elastic Dashboard for host metrics that also shows the same. I am assuming something is not right with 113 nodes.
My all read & write connection going here is 113 node, I want to change this to 111 or 112. I really don’t know how I can change the writer node.
What I observed when 113 node disconnect with other Nodes, ProxySQL will transfer all connection to 112 node and when 113 node back connection automatically goes to 113 node , I really did not found any setting about this , why ProxySQL transfer all connection to 113 when this node back ?
What I want to achieve…
-
Not to use 113 as Writer node
-
Use either 112 as Writer node and 111 & 113 as reader node.
Can someone please guide me here on how to do this ?
Here are some insights.
I checked proxysql-admin.cnf, here are the settings.
mysql> select * from mysql_servers;
mysql> select * from runtime_mysql_servers;
As you can see both mysql_servers & runtime_mysql_servers have different settings.
mysql> select * from mysql_galera_hostgroups;
1 row in set (0.01 sec)
I had issue today as well, Pmm shows at 8.47 AM Cluster size reduced from 3 to 2
Below are the MysqlD log from 113 Node (Writter Node)
2025-03-10T07:50:44.352837Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT2.61685S), skipping check
2025-03-10T07:50:51.088150Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer b8653e1e-8dff with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 12484 rttvar: 17860 rto: 213000 lost: 0 last_data_recv: 2280 cwnd: 10 last_queued_since: 5280111298 last_delivered_since: 5272798527 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 (gmcast.peer_timeout)
2025-03-10T07:50:51.088706Z 150894 [Warning] [MY-000000] [Galera] client_state: Unallowed state transition: 3 → 4
2025-03-10T07:50:51.091288Z 153094 [Warning] [MY-000000] [Galera] client_state: Unallowed state transition: 3 → 4
2025-03-10T07:50:51.091321Z 152730 [Warning] [MY-000000] [Galera] client_state: Unallowed state transition: 3 → 4
2025-03-10T07:50:51.092796Z 153093 [Warning] [MY-000000] [Galera] client_state: Unallowed state transition: 3 → 4
2025-03-10T07:50:51.092837Z 152375 [Warning] [MY-000000] [Galera] client_state: Unallowed state transition: 3 → 4
2025-03-10T07:50:51.098609Z 152767 [Warning] [MY-000000] [Galera] client_state: Unallowed state transition: 3 → 4
2025-03-10T07:50:52.820172Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 10015 rttvar: 14142 rto: 211000 lost: 0 last_data_recv: 3646 cwnd: 10 last_queued_since: 116003106 last_delivered_since: 7072608242 send_queue_length: 1 send_queue_bytes: 212 segment: 0 messages: 1 (gmcast.peer_timeout)
2025-03-10T07:51:04.371143Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://172.25.30.111:4567 tcp://172.25.30.112:4567 **
2025-03-10T07:51:08.835119Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) reconnecting to b8653e1e-8dff (tcp://172.25.30.111:4567), attempt 0
2025-03-10T07:51:17.302756Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) reconnecting to 8d731cab-ba76 (tcp://172.25.30.112:4567), attempt 0
2025-03-10T07:51:18.072908Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT32.5296S), skipping check
2025-03-10T07:51:24.315090Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 463 rttvar: 231 rto: 201000 lost: 0 last_data_recv: 7008 cwnd: 10 last_queued_since: 425636143298995 last_delivered_since: 425636143298995 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2025-03-10T07:52:10.382846Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT43.1516S), skipping check
2025-03-10T07:52:24.559876Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT20.0633S), skipping check
2025-03-10T07:52:33.077526Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 1423 rttvar: 711 rto: 202000 lost: 0 last_data_recv: 97032 cwnd: 10 last_queued_since: 6534624137 last_delivered_since: 425703055883404 send_queue_length: 1 send_queue_bytes: 44 segment: 0 messages: 1 (gmcast.peer_timeout)
2025-03-10T07:52:48.190140Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 433 rttvar: 216 rto: 201000 lost: 0 last_data_recv: 79180 cwnd: 10 last_queued_since: 15974877569 last_delivered_since: 15974885376 send_queue_length: 1 send_queue_bytes: 140 segment: 0 messages: 1 (gmcast.peer_timeout)
2025-03-10T07:52:51.667245Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer b8653e1e-8dff with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 456 rttvar: 228 rto: 201000 lost: 0 last_data_recv: 50526 cwnd: 10 last_queued_since: 28254514202 last_delivered_since: 28254515981 send_queue_length: 1 send_queue_bytes: 140 segment: 0 messages: 1 (gmcast.peer_timeout)
2025-03-10T07:53:00.024229Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT34.9579S), skipping check
2025-03-10T07:53:46.932309Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT37.19S), skipping check
2025-03-10T07:53:56.113633Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 496 rttvar: 248 rto: 201000 lost: 0 last_data_recv: 20930 cwnd: 10 last_queued_since: 425785675416520 last_delivered_since: 425785675416520 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2025-03-10T07:53:58.229038Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 511 rttvar: 255 rto: 201000 lost: 0 last_data_recv: 62160 cwnd: 10 last_queued_since: 40180968152 last_delivered_since: 40300641768 send_queue_length: 1 send_queue_bytes: 140 segment: 0 messages: 1 (gmcast.peer_timeout)
2025-03-10T07:53:58.269066Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT25.1849S), skipping check
2025-03-10T07:54:43.306770Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT42.5804S), skipping check
2025-03-10T07:54:43.542658Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 400 rttvar: 200 rto: 201000 lost: 0 last_data_recv: 18282 cwnd: 10 last_queued_since: 425839120145076 last_delivered_since: 425839120145076 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2025-03-10T07:54:45.047733Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.35194S), skipping check
2025-03-10T07:54:50.573505Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT4.7756S), skipping check
2025-03-10T07:54:59.459797Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT7.72131S), skipping check
2025-03-10T07:55:05.724868Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 405 rttvar: 202 rto: 201000 lost: 0 last_data_recv: 13135 cwnd: 10 last_queued_since: 5376827393 last_delivered_since: 5376874772 send_queue_length: 1 send_queue_bytes: 140 segment: 0 messages: 1 (gmcast.peer_timeout)
2025-03-10T07:55:09.706754Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer b8653e1e-8dff with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 423 rttvar: 211 rto: 201000 lost: 0 last_data_recv: 19744 cwnd: 10 last_queued_since: 12900907526 last_delivered_since: 13881415993 send_queue_length: 1 send_queue_bytes: 140 segment: 0 messages: 1 (gmcast.peer_timeout)
2025-03-10T07:56:06.964030Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT57.4167S), skipping check
2025-03-10T07:56:45.795429Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT48.7715S), skipping check
2025-03-10T07:56:50.310735Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 488 rttvar: 244 rto: 201000 lost: 0 last_data_recv: 56958 cwnd: 10 last_queued_since: 20424872114 last_delivered_since: 20433797249 send_queue_length: 1 send_queue_bytes: 140 segment: 0 messages: 1 (gmcast.peer_timeout)
2025-03-10T07:56:52.024369Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 613 rttvar: 306 rto: 201000 lost: 0 last_data_recv: 12261 cwnd: 10 last_queued_since: 425967028181237 last_delivered_since: 425967028181237 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
2025-03-10T07:56:53.897912Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT10.7964S), skipping check
2025-03-10T07:57:12.762160Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT19.0792S), skipping check
2025-03-10T07:57:12.765693Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection established to b8653e1e-8dff tcp://172.25.30.111:4567
2025-03-10T07:57:13.001121Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection established to 8d731cab-ba76 tcp://172.25.30.112:4567
2025-03-10T07:57:13.263761Z 0 [Note] [MY-000000] [Galera] declaring node with index 1 suspected, timeout PT5S (evs.suspect_timeout)
2025-03-10T07:57:13.263812Z 0 [Note] [MY-000000] [Galera] declaring node with index 1 inactive (evs.inactive_timeout) **
2025-03-10T07:57:13.263822Z 0 [Note] [MY-000000] [Galera] declaring node with index 2 suspected, timeout PT5S (evs.suspect_timeout)
2025-03-10T07:57:13.263827Z 0 [Note] [MY-000000] [Galera] declaring node with index 2 inactive (evs.inactive_timeout) **
2025-03-10T07:57:13.263839Z 0 [Note] [MY-000000] [Galera] evs::proto(3c8fa886-b224, OPERATIONAL, view_id(REG,3c8fa886-b224,9)) detected inactive node: 8d731cab-ba76
2025-03-10T07:57:13.263847Z 0 [Note] [MY-000000] [Galera] evs::proto(3c8fa886-b224, OPERATIONAL, view_id(REG,3c8fa886-b224,9)) detected inactive node: b8653e1e-8dff
2025-03-10T07:57:14.344500Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,3c8fa886-b224,9)
memb {
** 3c8fa886-b224,0
** }
joined {
** }
left {
** }**
partitioned {
** 8d731cab-ba76,0**
** b8653e1e-8dff,0**
** }**
)
2025-03-10T07:57:14.413650Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,3c8fa886-b224,10)
memb {
** 3c8fa886-b224,0**
** }**
joined {
** }**
left {
** }**
partitioned {
** 8d731cab-ba76,0**
** b8653e1e-8dff,0**
** }**
)
2025-03-10T07:57:14.429545Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2025-03-10T07:57:14.496575Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2025-03-10T07:57:14.496613Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2025-03-10T07:57:14.496620Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED → OPEN (TO: 555280067)
2025-03-10T07:57:14.496785Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2025-03-10T07:57:14.496832Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2025-03-10T07:57:14.496839Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2025-03-10T07:57:14.497877Z 16 [Note] [MY-000000] [Galera] Maybe drain monitors from 555280067 upto current CC event 555280067 upto:555280067
2025-03-10T07:57:14.536800Z 16 [Note] [MY-000000] [Galera] Drain monitors from 555280067 up to 555280067
2025-03-10T07:57:14.541828Z 16 [Note] [MY-000000] [Galera] ================================================
View:
** id: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280067**
** status: non-primary**
** protocol_version: 4**
** capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO**
** final: no**
** own_index: 0**
** members(1):**
** 0: 3c8fa886-f9a7-11ef-b224-4e7992fd2333, fra05v-xdb11**
=================================================
2025-03-10T07:57:14.546536Z 16 [Note] [MY-000000] [Galera] Non-primary view
2025-03-10T07:57:14.556850Z 16 [Note] [MY-000000] [WSREP] Server status change synced → connected
2025-03-10T07:57:14.589083Z 16 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2025-03-10T07:57:14.600978Z 16 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2025-03-10T07:57:14.630170Z 16 [Note] [MY-000000] [Galera] Maybe drain monitors from 555280067 upto current CC event 555280067 upto:555280067
2025-03-10T07:57:14.630208Z 16 [Note] [MY-000000] [Galera] Drain monitors from 555280067 up to 555280067
2025-03-10T07:57:14.630232Z 16 [Note] [MY-000000] [Galera] ================================================
View:
** id: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280067**
** status: non-primary**
** protocol_version: 4**
** capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO**
** final: no**
** own_index: 0**
** members(1):**
** 0: 3c8fa886-f9a7-11ef-b224-4e7992fd2333, fra05v-xdb11**
=================================================
2025-03-10T07:57:14.633961Z 16 [Note] [MY-000000] [Galera] Non-primary view
2025-03-10T07:57:14.633993Z 16 [Note] [MY-000000] [WSREP] Server status change connected → connected
2025-03-10T07:57:14.634005Z 16 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2025-03-10T07:57:15.498908Z 0 [Note] [MY-000000] [Galera] declaring 8d731cab-ba76 at tcp://172.25.30.112:4567 stable
2025-03-10T07:57:15.531905Z 0 [Note] [MY-000000] [Galera] declaring b8653e1e-8dff at tcp://172.25.30.111:4567 stable
2025-03-10T07:57:15.537853Z 0 [Note] [MY-000000] [Galera] Node 8d731cab-ba76 state primary
2025-03-10T07:57:15.538790Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,3c8fa886-b224,11)
memb {
** 3c8fa886-b224,0**
** 8d731cab-ba76,0**
** b8653e1e-8dff,0**
** }**
joined {
** }**
left {
** }**
partitioned {
** }**
)
2025-03-10T07:57:15.538810Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2025-03-10T07:57:15.618659Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
2025-03-10T07:57:15.623106Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 483d4f82-fd85-11ef-a91a-1776245bb33d
2025-03-10T07:57:15.623132Z 0 [Warning] [MY-000000] [Galera] Action message in non-primary configuration from member 1
2025-03-10T07:57:15.623768Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 483d4f82-fd85-11ef-a91a-1776245bb33d
2025-03-10T07:57:15.627219Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 483d4f82-fd85-11ef-a91a-1776245bb33d from 0 (fra05v-xdb11)
2025-03-10T07:57:15.627247Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 483d4f82-fd85-11ef-a91a-1776245bb33d from 1 (fra04v-xdb12)
2025-03-10T07:57:15.627259Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 483d4f82-fd85-11ef-a91a-1776245bb33d from 2 (fra04v-xdb11)
2025-03-10T07:57:15.627267Z 0 [Note] [MY-000000] [Galera] Quorum results:
** version = 6,**
** component = PRIMARY,**
** conf_id = 10,**
** members = 2/3 (primary/total),**
** act_id = 555292256,**
** last_appl. = 555292207,**
** protocols = 2/10/4 (gcs/repl/appl),**
** vote policy= 0,**
** group UUID = 44c2c619-f26a-11ee-b9d2-cae0218742a0**
2025-03-10T07:57:15.654938Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [173, 173]
2025-03-10T07:57:15.654982Z 0 [Note] [MY-000000] [Galera] Shifting OPEN → PRIMARY (TO: 555292257)
2025-03-10T07:57:15.655066Z 16 [Note] [MY-000000] [Galera] ####### processing CC 555292257, local, ordered
2025-03-10T07:57:15.655100Z 16 [Note] [MY-000000] [Galera] Maybe drain monitors from 555280067 upto current CC event 555292257 upto:555280067
2025-03-10T07:57:15.655109Z 16 [Note] [MY-000000] [Galera] Drain monitors from 555280067 up to 555280067
2025-03-10T07:57:15.655117Z 16 [Note] [MY-000000] [Galera] ####### My UUID: 3c8fa886-f9a7-11ef-b224-4e7992fd2333
2025-03-10T07:57:15.655129Z 16 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2025-03-10T07:57:15.667087Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2025-03-10T07:57:15.667152Z 16 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2025-03-10T07:57:15.667185Z 16 [Note] [MY-000000] [Galera] State transfer required: **
** Group state: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555292257
** Local state: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280067**
2025-03-10T07:57:15.675137Z 16 [Note] [MY-000000] [WSREP] Server status change connected → joiner
2025-03-10T07:57:15.675170Z 16 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2025-03-10T07:57:15.675196Z 16 [Note] [MY-000000] [WSREP] WSREP: You have configured ‘xtrabackup-v2’ state snapshot transfer method which cannot be performed on a running server. Wsrep provider won’t be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server.
2025-03-10T07:57:15.675216Z 16 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2025-03-10T07:57:15.675227Z 16 [Note] [MY-000000] [Galera] ####### IST uuid:44c2c619-f26a-11ee-b9d2-cae0218742a0 f: 555280068, l: 555292257, STRv: 3
2025-03-10T07:57:15.712912Z 16 [Note] [MY-000000] [Galera] IST receiver addr using tcp://172.25.30.113:4568
2025-03-10T07:57:15.734271Z 16 [Note] [MY-000000] [Galera] Prepared IST receiver for 555280068-555292257, listening at: tcp://172.25.30.113:4568
2025-03-10T07:57:15.773263Z 0 [Note] [MY-000000] [Galera] Member 0.0 (fra05v-xdb11) requested state transfer from ‘any’. Selected 1.0 (fra04v-xdb12)(SYNCED) as donor.
2025-03-10T07:57:15.778448Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY → JOINER (TO: 555292267)
2025-03-10T07:57:15.778539Z 16 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1
2025-03-10T07:57:15.783888Z 16 [Note] [MY-000000] [Galera] Receiving IST: 12190 writesets, seqnos 555280068-555292257
2025-03-10T07:57:15.792703Z 0 [Note] [MY-000000] [Galera] 1.0 (fra04v-xdb12): State transfer to 0.0 (fra05v-xdb11) complete.
2025-03-10T07:57:15.793307Z 0 [Note] [MY-000000] [Galera] Member 1.0 (fra04v-xdb12) synced with group.
2025-03-10T07:57:15.799061Z 0 [Note] [MY-000000] [Galera] ####### IST applying starts with 555280068
2025-03-10T07:57:15.810217Z 0 [Note] [MY-000000] [Galera] ####### IST current seqno initialized to 555280068
2025-03-10T07:57:15.842386Z 0 [Note] [MY-000000] [Galera] Receiving IST… 0.0% ( 0/12190 events) complete.
2025-03-10T07:57:15.872284Z 0 [Note] [MY-000000] [Galera] ####### Passing IST CC 555280068, must_apply: 1, preload: false
2025-03-10T07:57:15.901558Z 0 [Note] [MY-000000] [Galera] First IST (CC) event for rebuilding cert index
2025-03-10T07:57:15.909148Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2025-03-10T07:57:15.920549Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2025-03-10T07:57:15.926550Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280067, protocol version: 5
2025-03-10T07:57:15.928458Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2025-03-10T07:57:15.928480Z 0 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 555280067 → 555280068
2025-03-10T07:57:15.928553Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2025-03-10T07:57:15.941099Z 0 [Note] [MY-000000] [Galera] Recording CC from ist: 555280068
2025-03-10T07:57:15.946944Z 0 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from ist: 555280068
2025-03-10T07:57:15.946971Z 0 [Note] [MY-000000] [Galera] Min available from gcache for CC from ist: 555169280
2025-03-10T07:57:16.033443Z 13 [Note] [MY-000000] [Galera] ================================================
View:
** id: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280068**
** status: primary**
** protocol_version: 4**
** capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO**
** final: no**
** own_index: -1**
** members(2):**
** 0: 8d731cab-f9b6-11ef-ba76-7b134b929f3d, fra04v-xdb12**
** 1: b8653e1e-f9a8-11ef-8dff-d6fe14f79c27, fra04v-xdb11**
=================================================
2025-03-10T07:57:16.104997Z 13 [Note] [MY-000000] [WSREP] Server status change joiner → joined
2025-03-10T07:57:16.291064Z 13 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2025-03-10T07:57:16.529878Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2025-03-10T07:57:16.493524Z 13 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2025-03-10T07:57:16.860139Z 0 [Note] [MY-000000] [Galera] Receiving IST… 99.6% (12140/12190 events) complete.
2025-03-10T07:57:16.864509Z 0 [Note] [MY-000000] [Galera] IST preload starting at 555292208
2025-03-10T07:57:16.865107Z 0 [Note] [MY-000000] [Galera] ####### Passing IST CC 555292257, must_apply: 1, preload: true
2025-03-10T07:58:22.768653Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2025-03-10T07:58:22.769383Z 0 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 555292256 → 555292257
2025-03-10T07:58:22.769446Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2025-03-10T07:58:22.769499Z 0 [Note] [MY-000000] [Galera] Recording CC from ist: 555292257
2025-03-10T07:58:22.769508Z 0 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from ist: 555292208
2025-03-10T07:58:22.769513Z 0 [Note] [MY-000000] [Galera] Min available from gcache for CC from ist: 555183035
2025-03-10T07:58:22.769596Z 1 [Note] [MY-000000] [Galera] ================================================
View:
** id: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555292257**
** status: primary**
** protocol_version: 4**
** capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO**
** final: no**
** own_index: 0**
** members(3):**
** 0: 3c8fa886-f9a7-11ef-b224-4e7992fd2333, fra05v-xdb11**
** 1: 8d731cab-f9b6-11ef-ba76-7b134b929f3d, fra04v-xdb12**
** 2: b8653e1e-f9a8-11ef-8dff-d6fe14f79c27, fra04v-xdb11**
=================================================
2025-03-10T07:58:22.770431Z 0 [Note] [MY-000000] [Galera] Receiving IST…100.0% (12190/12190 events) complete.
2025-03-10T07:58:22.771934Z 16 [Note] [MY-000000] [Galera] Draining apply monitors after IST up to 555292257
2025-03-10T07:58:22.772355Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2025-03-10T07:58:22.782674Z 16 [Note] [MY-000000] [Galera] IST received: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555292257
2025-03-10T07:58:22.784893Z 16 [Note] [MY-000000] [Galera] Recording CC from sst: 555292257
2025-03-10T07:58:22.784931Z 16 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from sst: 555292208
2025-03-10T07:58:22.784938Z 16 [Note] [MY-000000] [Galera] Min available from gcache for CC from sst: 555183035
2025-03-10T07:58:22.786029Z 0 [Note] [MY-000000] [Galera] 0.0 (fra05v-xdb11): State transfer from 1.0 (fra04v-xdb12) complete.
2025-03-10T07:58:22.786069Z 0 [Note] [MY-000000] [Galera] SST leaving flow control
2025-03-10T07:58:22.786078Z 0 [Note] [MY-000000] [Galera] Shifting JOINER → JOINED (TO: 555293562)
2025-03-10T07:58:29.242901Z 0 [Note] [MY-000000] [Galera] Member 0.0 (fra05v-xdb11) synced with group.
2025-03-10T07:58:29.242947Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 555293645)
2025-03-10T07:58:30.154148Z 15 [Note] [MY-000000] [Galera] Server fra05v-xdb11 synced with group
2025-03-10T07:58:30.154195Z 15 [Note] [MY-000000] [WSREP] Server status change joined → synced
2025-03-10T07:58:30.154202Z 15 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2025-03-10T07:58:30.157263Z 15 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.