# Want to change Writter Node In Percona CLuster **URL:** \ **Category:** Percona XtraDB Cluster 8.x\ **Tags:** community, pmm, mysql, percona\ **Created:** [March 10, 2025, 3:02pm UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113 "2025-03-10T15:02:43Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![sachinjoshi](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@sachinjoshi](https://forums.percona.com/u/sachinjoshi)\ **Post date:** [March 10, 2025, 3:02pm UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/1 "2025-03-10T15:02:43Z")
Hi There, I am running Percona MySQL Cluster with 3 Nodes 111,112,113 & ProxySQL 114. For the last 3 months we have been having issues, Percona Nodes Disconnect with each other, sometimes it works because other 2 nodes were alive but sometimes not, ProxySQL is not able to connect any node. After getting these incidents I configured PMM to monitor the cluster. Last two incidents show node 113 completely blackouts when connection timeout happens, PMM shows no TCP network connectivity in 113 node but 111 and 112 node do not have TCP connectivity issues. We also have an Elastic Dashboard for host metrics that also shows the same. I am assuming something is not right with 113 nodes. My all read & write connection going here is 113 node, I want to change this to 111 or 112. I really don’t know how I can change the writer node. What I observed when 113 node disconnect with other Nodes, ProxySQL will transfer all connection to 112 node and when 113 node back connection automatically goes to 113 node , I really did not found any setting about this , why ProxySQL transfer all connection to 113 when this node back ? What I want to achieve… 1. Not to use 113 as Writer node 2. Use either 112 as Writer node and 111 & 113 as reader node. Can someone please guide me here on how to do this ? Here are some insights. I checked proxysql-admin.cnf, here are the settings. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/5/1/5159f71a83ec982a1a1522eb4ae17886b91fda6a.png) **mysql\> select \* from mysql\_servers;** ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/2/e/2edc34913db25f3c01800ac0daa1ae2389ae4212.png) **mysql\> select \* from runtime\_mysql\_servers;** ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/f/e/fe010a2f4eebd6768303a8fc724e0f04638a11a6.png) As you can see both mysql\_servers & runtime\_mysql\_servers have different settings. **mysql\> select \* from mysql\_galera\_hostgroups;** ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/c/c/ccfe3b853d79e948032025742385c0d3a6e217f2.png) 1 row in set (0.01 sec) I had issue today as well, Pmm shows at 8.47 AM Cluster size reduced from 3 to 2 Below are the MysqlD log from 113 Node (Writter Node) **2025-03-10T07:50:44.352837Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT2.61685S), skipping check** **2025-03-10T07:50:51.088150Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer b8653e1e-8dff with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 12484 rttvar: 17860 rto: 213000 lost: 0 last\_data\_recv: 2280 cwnd: 10 last\_queued\_since: 5280111298 last\_delivered\_since: 5272798527 send\_queue\_length: 0 send\_queue\_bytes: 0 segment: 0 messages: 0 (gmcast.peer\_timeout)** **2025-03-10T07:50:51.088706Z 150894 [Warning] [MY-000000] [Galera] client\_state: Unallowed state transition: 3 → 4** **2025-03-10T07:50:51.091288Z 153094 [Warning] [MY-000000] [Galera] client\_state: Unallowed state transition: 3 → 4** **2025-03-10T07:50:51.091321Z 152730 [Warning] [MY-000000] [Galera] client\_state: Unallowed state transition: 3 → 4** **2025-03-10T07:50:51.092796Z 153093 [Warning] [MY-000000] [Galera] client\_state: Unallowed state transition: 3 → 4** **2025-03-10T07:50:51.092837Z 152375 [Warning] [MY-000000] [Galera] client\_state: Unallowed state transition: 3 → 4** **2025-03-10T07:50:51.098609Z 152767 [Warning] [MY-000000] [Galera] client\_state: Unallowed state transition: 3 → 4** **2025-03-10T07:50:52.820172Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 10015 rttvar: 14142 rto: 211000 lost: 0 last\_data\_recv: 3646 cwnd: 10 last\_queued\_since: 116003106 last\_delivered\_since: 7072608242 send\_queue\_length: 1 send\_queue\_bytes: 212 segment: 0 messages: 1 (gmcast.peer\_timeout)** **2025-03-10T07:51:04.371143Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://172.25.30.111:4567 tcp://172.25.30.112:4567 \*\* 2025-03-10T07:51:08.835119Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) reconnecting to b8653e1e-8dff (tcp://172.25.30.111:4567), attempt 0 2025-03-10T07:51:17.302756Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) reconnecting to 8d731cab-ba76 (tcp://172.25.30.112:4567), attempt 0 2025-03-10T07:51:18.072908Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT32.5296S), skipping check 2025-03-10T07:51:24.315090Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 463 rttvar: 231 rto: 201000 lost: 0 last\_data\_recv: 7008 cwnd: 10 last\_queued\_since: 425636143298995 last\_delivered\_since: 425636143298995 send\_queue\_length: 0 send\_queue\_bytes: 0 (gmcast.peer\_timeout) 2025-03-10T07:52:10.382846Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT43.1516S), skipping check 2025-03-10T07:52:24.559876Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT20.0633S), skipping check 2025-03-10T07:52:33.077526Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 1423 rttvar: 711 rto: 202000 lost: 0 last\_data\_recv: 97032 cwnd: 10 last\_queued\_since: 6534624137 last\_delivered\_since: 425703055883404 send\_queue\_length: 1 send\_queue\_bytes: 44 segment: 0 messages: 1 (gmcast.peer\_timeout) 2025-03-10T07:52:48.190140Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 433 rttvar: 216 rto: 201000 lost: 0 last\_data\_recv: 79180 cwnd: 10 last\_queued\_since: 15974877569 last\_delivered\_since: 15974885376 send\_queue\_length: 1 send\_queue\_bytes: 140 segment: 0 messages: 1 (gmcast.peer\_timeout) 2025-03-10T07:52:51.667245Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer b8653e1e-8dff with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 456 rttvar: 228 rto: 201000 lost: 0 last\_data\_recv: 50526 cwnd: 10 last\_queued\_since: 28254514202 last\_delivered\_since: 28254515981 send\_queue\_length: 1 send\_queue\_bytes: 140 segment: 0 messages: 1 (gmcast.peer\_timeout) 2025-03-10T07:53:00.024229Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT34.9579S), skipping check 2025-03-10T07:53:46.932309Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT37.19S), skipping check 2025-03-10T07:53:56.113633Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 496 rttvar: 248 rto: 201000 lost: 0 last\_data\_recv: 20930 cwnd: 10 last\_queued\_since: 425785675416520 last\_delivered\_since: 425785675416520 send\_queue\_length: 0 send\_queue\_bytes: 0 (gmcast.peer\_timeout) 2025-03-10T07:53:58.229038Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 511 rttvar: 255 rto: 201000 lost: 0 last\_data\_recv: 62160 cwnd: 10 last\_queued\_since: 40180968152 last\_delivered\_since: 40300641768 send\_queue\_length: 1 send\_queue\_bytes: 140 segment: 0 messages: 1 (gmcast.peer\_timeout) 2025-03-10T07:53:58.269066Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT25.1849S), skipping check 2025-03-10T07:54:43.306770Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT42.5804S), skipping check 2025-03-10T07:54:43.542658Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 400 rttvar: 200 rto: 201000 lost: 0 last\_data\_recv: 18282 cwnd: 10 last\_queued\_since: 425839120145076 last\_delivered\_since: 425839120145076 send\_queue\_length: 0 send\_queue\_bytes: 0 (gmcast.peer\_timeout) 2025-03-10T07:54:45.047733Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT3.35194S), skipping check 2025-03-10T07:54:50.573505Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT4.7756S), skipping check 2025-03-10T07:54:59.459797Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT7.72131S), skipping check 2025-03-10T07:55:05.724868Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 405 rttvar: 202 rto: 201000 lost: 0 last\_data\_recv: 13135 cwnd: 10 last\_queued\_since: 5376827393 last\_delivered\_since: 5376874772 send\_queue\_length: 1 send\_queue\_bytes: 140 segment: 0 messages: 1 (gmcast.peer\_timeout) 2025-03-10T07:55:09.706754Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer b8653e1e-8dff with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 423 rttvar: 211 rto: 201000 lost: 0 last\_data\_recv: 19744 cwnd: 10 last\_queued\_since: 12900907526 last\_delivered\_since: 13881415993 send\_queue\_length: 1 send\_queue\_bytes: 140 segment: 0 messages: 1 (gmcast.peer\_timeout) 2025-03-10T07:56:06.964030Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT57.4167S), skipping check 2025-03-10T07:56:45.795429Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT48.7715S), skipping check 2025-03-10T07:56:50.310735Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S, socket stats: rtt: 488 rttvar: 244 rto: 201000 lost: 0 last\_data\_recv: 56958 cwnd: 10 last\_queued\_since: 20424872114 last\_delivered\_since: 20433797249 send\_queue\_length: 1 send\_queue\_bytes: 140 segment: 0 messages: 1 (gmcast.peer\_timeout) 2025-03-10T07:56:52.024369Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection to peer 00000000-0000 with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S, socket stats: rtt: 613 rttvar: 306 rto: 201000 lost: 0 last\_data\_recv: 12261 cwnd: 10 last\_queued\_since: 425967028181237 last\_delivered\_since: 425967028181237 send\_queue\_length: 0 send\_queue\_bytes: 0 (gmcast.peer\_timeout) 2025-03-10T07:56:53.897912Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT10.7964S), skipping check 2025-03-10T07:57:12.762160Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3\*evs.inactive\_check\_period) ago (PT19.0792S), skipping check 2025-03-10T07:57:12.765693Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection established to b8653e1e-8dff tcp://172.25.30.111:4567 2025-03-10T07:57:13.001121Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) connection established to 8d731cab-ba76 tcp://172.25.30.112:4567 2025-03-10T07:57:13.263761Z 0 [Note] [MY-000000] [Galera] declaring node with index 1 suspected, timeout PT5S (evs.suspect\_timeout) 2025-03-10T07:57:13.263812Z 0 [Note] [MY-000000] [Galera] declaring node with index 1 inactive (evs.inactive\_timeout) \*\* 2025-03-10T07:57:13.263822Z 0 [Note] [MY-000000] [Galera] declaring node with index 2 suspected, timeout PT5S (evs.suspect\_timeout) 2025-03-10T07:57:13.263827Z 0 [Note] [MY-000000] [Galera] declaring node with index 2 inactive (evs.inactive\_timeout) \*\* 2025-03-10T07:57:13.263839Z 0 [Note] [MY-000000] [Galera] evs::proto(3c8fa886-b224, OPERATIONAL, view\_id(REG,3c8fa886-b224,9)) detected inactive node: 8d731cab-ba76 2025-03-10T07:57:13.263847Z 0 [Note] [MY-000000] [Galera] evs::proto(3c8fa886-b224, OPERATIONAL, view\_id(REG,3c8fa886-b224,9)) detected inactive node: b8653e1e-8dff 2025-03-10T07:57:14.344500Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view (view\_id(NON\_PRIM,3c8fa886-b224,9) memb { \*\* 3c8fa886-b224,0 \*\* } joined { \*\* }** **left {** \*\* }\*\* **partitioned {** \*\* 8d731cab-ba76,0\*\* \*\* b8653e1e-8dff,0\*\* \*\* }\*\* **)** **2025-03-10T07:57:14.413650Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node** **view (view\_id(NON\_PRIM,3c8fa886-b224,10)** **memb {** \*\* 3c8fa886-b224,0\*\* \*\* }\*\* **joined {** \*\* }\*\* **left {** \*\* }\*\* **partitioned {** \*\* 8d731cab-ba76,0\*\* \*\* b8653e1e-8dff,0\*\* \*\* }\*\* **)** **2025-03-10T07:57:14.429545Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1** **2025-03-10T07:57:14.496575Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]** **2025-03-10T07:57:14.496613Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.** **2025-03-10T07:57:14.496620Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED → OPEN (TO: 555280067)** **2025-03-10T07:57:14.496785Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1** **2025-03-10T07:57:14.496832Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]** **2025-03-10T07:57:14.496839Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.** **2025-03-10T07:57:14.497877Z 16 [Note] [MY-000000] [Galera] Maybe drain monitors from 555280067 upto current CC event 555280067 upto:555280067** **2025-03-10T07:57:14.536800Z 16 [Note] [MY-000000] [Galera] Drain monitors from 555280067 up to 555280067** **2025-03-10T07:57:14.541828Z 16 [Note] [MY-000000] [Galera] ================================================** **View:** \*\* id: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280067\*\* \*\* status: non-primary\*\* \*\* protocol\_version: 4\*\* \*\* capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL\_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL\_READ, INCREMENTAL\_WS, UNORDERED, PREORDERED, STREAMING, NBO\*\* \*\* final: no\*\* \*\* own\_index: 0\*\* \*\* members(1):\*\* \*\* 0: 3c8fa886-f9a7-11ef-b224-4e7992fd2333, fra05v-xdb11\*\* **=================================================** **2025-03-10T07:57:14.546536Z 16 [Note] [MY-000000] [Galera] Non-primary view** **2025-03-10T07:57:14.556850Z 16 [Note] [MY-000000] [WSREP] Server status change synced → connected** **2025-03-10T07:57:14.589083Z 16 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.** **2025-03-10T07:57:14.600978Z 16 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.** **2025-03-10T07:57:14.630170Z 16 [Note] [MY-000000] [Galera] Maybe drain monitors from 555280067 upto current CC event 555280067 upto:555280067** **2025-03-10T07:57:14.630208Z 16 [Note] [MY-000000] [Galera] Drain monitors from 555280067 up to 555280067** **2025-03-10T07:57:14.630232Z 16 [Note] [MY-000000] [Galera] ================================================** **View:** \*\* id: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280067\*\* \*\* status: non-primary\*\* \*\* protocol\_version: 4\*\* \*\* capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL\_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL\_READ, INCREMENTAL\_WS, UNORDERED, PREORDERED, STREAMING, NBO\*\* \*\* final: no\*\* \*\* own\_index: 0\*\* \*\* members(1):\*\* \*\* 0: 3c8fa886-f9a7-11ef-b224-4e7992fd2333, fra05v-xdb11\*\* **=================================================** **2025-03-10T07:57:14.633961Z 16 [Note] [MY-000000] [Galera] Non-primary view** **2025-03-10T07:57:14.633993Z 16 [Note] [MY-000000] [WSREP] Server status change connected → connected** **2025-03-10T07:57:14.634005Z 16 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.** **2025-03-10T07:57:15.498908Z 0 [Note] [MY-000000] [Galera] declaring 8d731cab-ba76 at tcp://172.25.30.112:4567 stable** **2025-03-10T07:57:15.531905Z 0 [Note] [MY-000000] [Galera] declaring b8653e1e-8dff at tcp://172.25.30.111:4567 stable** **2025-03-10T07:57:15.537853Z 0 [Note] [MY-000000] [Galera] Node 8d731cab-ba76 state primary** **2025-03-10T07:57:15.538790Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node** **view (view\_id(PRIM,3c8fa886-b224,11)** **memb {** \*\* 3c8fa886-b224,0\*\* \*\* 8d731cab-ba76,0\*\* \*\* b8653e1e-8dff,0\*\* \*\* }\*\* **joined {** \*\* }\*\* **left {** \*\* }\*\* **partitioned {** \*\* }\*\* **)** **2025-03-10T07:57:15.538810Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk** **2025-03-10T07:57:15.618659Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my\_idx = 0, memb\_num = 3** **2025-03-10T07:57:15.623106Z 0 [Note] [MY-000000] [Galera] STATE\_EXCHANGE: sent state UUID: 483d4f82-fd85-11ef-a91a-1776245bb33d** **2025-03-10T07:57:15.623132Z 0 [Warning] [MY-000000] [Galera] Action message in non-primary configuration from member 1** **2025-03-10T07:57:15.623768Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 483d4f82-fd85-11ef-a91a-1776245bb33d** **2025-03-10T07:57:15.627219Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 483d4f82-fd85-11ef-a91a-1776245bb33d from 0 (fra05v-xdb11)** **2025-03-10T07:57:15.627247Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 483d4f82-fd85-11ef-a91a-1776245bb33d from 1 (fra04v-xdb12)** **2025-03-10T07:57:15.627259Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 483d4f82-fd85-11ef-a91a-1776245bb33d from 2 (fra04v-xdb11)** **2025-03-10T07:57:15.627267Z 0 [Note] [MY-000000] [Galera] Quorum results:** \*\* version = 6,\*\* \*\* component = PRIMARY,\*\* \*\* conf\_id = 10,\*\* \*\* members = 2/3 (primary/total),\*\* \*\* act\_id = 555292256,\*\* \*\* last\_appl. = 555292207,\*\* \*\* protocols = 2/10/4 (gcs/repl/appl),\*\* \*\* vote policy= 0,\*\* \*\* group UUID = 44c2c619-f26a-11ee-b9d2-cae0218742a0\*\* **2025-03-10T07:57:15.654938Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [173, 173]** **2025-03-10T07:57:15.654982Z 0 [Note] [MY-000000] [Galera] Shifting OPEN → PRIMARY (TO: 555292257)** **2025-03-10T07:57:15.655066Z 16 [Note] [MY-000000] [Galera] ####### processing CC 555292257, local, ordered** **2025-03-10T07:57:15.655100Z 16 [Note] [MY-000000] [Galera] Maybe drain monitors from 555280067 upto current CC event 555292257 upto:555280067** **2025-03-10T07:57:15.655109Z 16 [Note] [MY-000000] [Galera] Drain monitors from 555280067 up to 555280067** **2025-03-10T07:57:15.655117Z 16 [Note] [MY-000000] [Galera] ####### My UUID: 3c8fa886-f9a7-11ef-b224-4e7992fd2333** **2025-03-10T07:57:15.655129Z 16 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes** **2025-03-10T07:57:15.667087Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.** **2025-03-10T07:57:15.667152Z 16 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1** **2025-03-10T07:57:15.667185Z 16 [Note] [MY-000000] [Galera] State transfer required: \*\* \*\* Group state: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555292257** \*\* Local state: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280067\*\* **2025-03-10T07:57:15.675137Z 16 [Note] [MY-000000] [WSREP] Server status change connected → joiner** **2025-03-10T07:57:15.675170Z 16 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.** **2025-03-10T07:57:15.675196Z 16 [Note] [MY-000000] [WSREP] WSREP: You have configured ‘xtrabackup-v2’ state snapshot transfer method which cannot be performed on a running server. Wsrep provider won’t be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server.** **2025-03-10T07:57:15.675216Z 16 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST** **2025-03-10T07:57:15.675227Z 16 [Note] [MY-000000] [Galera] ####### IST uuid:44c2c619-f26a-11ee-b9d2-cae0218742a0 f: 555280068, l: 555292257, STRv: 3** **2025-03-10T07:57:15.712912Z 16 [Note] [MY-000000] [Galera] IST receiver addr using tcp://172.25.30.113:4568** **2025-03-10T07:57:15.734271Z 16 [Note] [MY-000000] [Galera] Prepared IST receiver for 555280068-555292257, listening at: tcp://172.25.30.113:4568** **2025-03-10T07:57:15.773263Z 0 [Note] [MY-000000] [Galera] Member 0.0 (fra05v-xdb11) requested state transfer from ‘_any_’. Selected 1.0 (fra04v-xdb12)(SYNCED) as donor.** **2025-03-10T07:57:15.778448Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY → JOINER (TO: 555292267)** **2025-03-10T07:57:15.778539Z 16 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1** **2025-03-10T07:57:15.783888Z 16 [Note] [MY-000000] [Galera] Receiving IST: 12190 writesets, seqnos 555280068-555292257** **2025-03-10T07:57:15.792703Z 0 [Note] [MY-000000] [Galera] 1.0 (fra04v-xdb12): State transfer to 0.0 (fra05v-xdb11) complete.** **2025-03-10T07:57:15.793307Z 0 [Note] [MY-000000] [Galera] Member 1.0 (fra04v-xdb12) synced with group.** **2025-03-10T07:57:15.799061Z 0 [Note] [MY-000000] [Galera] ####### IST applying starts with 555280068** **2025-03-10T07:57:15.810217Z 0 [Note] [MY-000000] [Galera] ####### IST current seqno initialized to 555280068** **2025-03-10T07:57:15.842386Z 0 [Note] [MY-000000] [Galera] Receiving IST… 0.0% ( 0/12190 events) complete.** **2025-03-10T07:57:15.872284Z 0 [Note] [MY-000000] [Galera] ####### Passing IST CC 555280068, must\_apply: 1, preload: false** **2025-03-10T07:57:15.901558Z 0 [Note] [MY-000000] [Galera] First IST (CC) event for rebuilding cert index** **2025-03-10T07:57:15.909148Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)** **2025-03-10T07:57:15.920549Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.** **2025-03-10T07:57:15.926550Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280067, protocol version: 5** **2025-03-10T07:57:15.928458Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)** **2025-03-10T07:57:15.928480Z 0 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 555280067 → 555280068** **2025-03-10T07:57:15.928553Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.** **2025-03-10T07:57:15.941099Z 0 [Note] [MY-000000] [Galera] Recording CC from ist: 555280068** **2025-03-10T07:57:15.946944Z 0 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from ist: 555280068** **2025-03-10T07:57:15.946971Z 0 [Note] [MY-000000] [Galera] Min available from gcache for CC from ist: 555169280** **2025-03-10T07:57:16.033443Z 13 [Note] [MY-000000] [Galera] ================================================** **View:** \*\* id: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555280068\*\* \*\* status: primary\*\* \*\* protocol\_version: 4\*\* \*\* capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL\_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL\_READ, INCREMENTAL\_WS, UNORDERED, PREORDERED, STREAMING, NBO\*\* \*\* final: no\*\* \*\* own\_index: -1\*\* \*\* members(2):\*\* \*\* 0: 8d731cab-f9b6-11ef-ba76-7b134b929f3d, fra04v-xdb12\*\* \*\* 1: b8653e1e-f9a8-11ef-8dff-d6fe14f79c27, fra04v-xdb11\*\* **=================================================** **2025-03-10T07:57:16.104997Z 13 [Note] [MY-000000] [WSREP] Server status change joiner → joined** **2025-03-10T07:57:16.291064Z 13 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.** **2025-03-10T07:57:16.529878Z 0 [Note] [MY-000000] [Galera] (3c8fa886-b224, ‘tcp://0.0.0.0:4567’) turning message relay requesting off** **2025-03-10T07:57:16.493524Z 13 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.** **2025-03-10T07:57:16.860139Z 0 [Note] [MY-000000] [Galera] Receiving IST… 99.6% (12140/12190 events) complete.** **2025-03-10T07:57:16.864509Z 0 [Note] [MY-000000] [Galera] IST preload starting at 555292208** **2025-03-10T07:57:16.865107Z 0 [Note] [MY-000000] [Galera] ####### Passing IST CC 555292257, must\_apply: 1, preload: true** **2025-03-10T07:58:22.768653Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)** **2025-03-10T07:58:22.769383Z 0 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 555292256 → 555292257** **2025-03-10T07:58:22.769446Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.** **2025-03-10T07:58:22.769499Z 0 [Note] [MY-000000] [Galera] Recording CC from ist: 555292257** **2025-03-10T07:58:22.769508Z 0 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from ist: 555292208** **2025-03-10T07:58:22.769513Z 0 [Note] [MY-000000] [Galera] Min available from gcache for CC from ist: 555183035** **2025-03-10T07:58:22.769596Z 1 [Note] [MY-000000] [Galera] ================================================** **View:** \*\* id: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555292257\*\* \*\* status: primary\*\* \*\* protocol\_version: 4\*\* \*\* capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL\_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL\_READ, INCREMENTAL\_WS, UNORDERED, PREORDERED, STREAMING, NBO\*\* \*\* final: no\*\* \*\* own\_index: 0\*\* \*\* members(3):\*\* \*\* 0: 3c8fa886-f9a7-11ef-b224-4e7992fd2333, fra05v-xdb11\*\* \*\* 1: 8d731cab-f9b6-11ef-ba76-7b134b929f3d, fra04v-xdb12\*\* \*\* 2: b8653e1e-f9a8-11ef-8dff-d6fe14f79c27, fra04v-xdb11\*\* **=================================================** **2025-03-10T07:58:22.770431Z 0 [Note] [MY-000000] [Galera] Receiving IST…100.0% (12190/12190 events) complete.** **2025-03-10T07:58:22.771934Z 16 [Note] [MY-000000] [Galera] Draining apply monitors after IST up to 555292257** **2025-03-10T07:58:22.772355Z 1 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.** **2025-03-10T07:58:22.782674Z 16 [Note] [MY-000000] [Galera] IST received: 44c2c619-f26a-11ee-b9d2-cae0218742a0:555292257** **2025-03-10T07:58:22.784893Z 16 [Note] [MY-000000] [Galera] Recording CC from sst: 555292257** **2025-03-10T07:58:22.784931Z 16 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from sst: 555292208** **2025-03-10T07:58:22.784938Z 16 [Note] [MY-000000] [Galera] Min available from gcache for CC from sst: 555183035** **2025-03-10T07:58:22.786029Z 0 [Note] [MY-000000] [Galera] 0.0 (fra05v-xdb11): State transfer from 1.0 (fra04v-xdb12) complete.** **2025-03-10T07:58:22.786069Z 0 [Note] [MY-000000] [Galera] SST leaving flow control** **2025-03-10T07:58:22.786078Z 0 [Note] [MY-000000] [Galera] Shifting JOINER → JOINED (TO: 555293562)** **2025-03-10T07:58:29.242901Z 0 [Note] [MY-000000] [Galera] Member 0.0 (fra05v-xdb11) synced with group.** **2025-03-10T07:58:29.242947Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 555293645)** **2025-03-10T07:58:30.154148Z 15 [Note] [MY-000000] [Galera] Server fra05v-xdb11 synced with group** **2025-03-10T07:58:30.154195Z 15 [Note] [MY-000000] [WSREP] Server status change joined → synced** **2025-03-10T07:58:30.154202Z 15 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections** **2025-03-10T07:58:30.157263Z 15 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.** ---
**Author:** ![sachinjoshi](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@sachinjoshi](https://forums.percona.com/u/sachinjoshi)\ **Post date:** [March 10, 2025, 4:14pm UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/2 "2025-03-10T16:14:46Z")
Can someone help me here Please. ---
**Author:** ![Wayne\_Leutwyler](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/wayne_leutwyler/32/20829_2.png) [@Wayne\_Leutwyler](https://forums.percona.com/u/Wayne_Leutwyler)\ **Post date:** [March 10, 2025, 5:50pm UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/3 "2025-03-10T17:50:18Z")
You need to adjust the weight on the servers. Currently you have them all weighted at 1000. So try this: 112 weight 100 111 weight 90 113 weight 80 Try this out, see what happens. It should help. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [March 11, 2025, 1:39am UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/4 "2025-03-11T01:39:29Z")
> [@sachinjoshi](#): > > connection to peer b8653e1e-8dff with addr tcp://172.25.30.111:4567 timed out, no messages seen in PT3S > [@sachinjoshi](#): > > connection to peer 8d731cab-ba76 with addr tcp://172.25.30.112:4567 timed out, no messages seen in PT3S You do have some network issues, as both PMM and the other system see this. Have you looked at `dmesg` or any other journalctl message around the timeout times? As Wayne says, you need to change the weights to prefer another host. ---
**Author:** ![sachinjoshi](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@sachinjoshi](https://forums.percona.com/u/sachinjoshi)\ **Post date:** [March 11, 2025, 9:57am UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/5 "2025-03-11T09:57:14Z")
Hi Wayne, thanks For reply. As you suggest i will try to do following things. 1 Truncate mysql\_servers table. 2 Insert records like below with weight. ```auto +--------------+---------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | hostgroup_id | hostname | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | +--------------+---------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | 1 | 172.25.30.112 | 3306 | 0 | ONLINE | 100000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 172.25.30.111 | 3306 | 0 | ONLINE | 100000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 172.25.30.112 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 172.25.30.113 | 3306 | 0 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | | | 90 | 172.25.30.111 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | | | 90 | 172.25.30.113 | 3306 | 0 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | | +--------------+---------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ load mysql query rules to runtime; load mysql query rules to runtime; ``` 172.25.30.112 will be my primary with hostgroup id 1 If 112 Node down or disconnect , proxysql will select first 111 and then 113 for writer. I am right here ? Also i checked all logs i did not able to figure out whats happening with 113 server, dmsg log shows nothing. Thanks Sachin ---
**Author:** ![sachinjoshi](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@sachinjoshi](https://forums.percona.com/u/sachinjoshi)\ **Post date:** [March 11, 2025, 10:29am UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/6 "2025-03-11T10:29:09Z")
Hi @matthewb & @Wayne_Leutwyler should i insert records like above i mentioned , and before insert should i trucate mysql\_server table ? ---
**Author:** ![Wayne\_Leutwyler](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/wayne_leutwyler/32/20829_2.png) [@Wayne\_Leutwyler](https://forums.percona.com/u/Wayne_Leutwyler)\ **Post date:** [March 11, 2025, 11:32am UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/7 "2025-03-11T11:32:30Z")
Based on what I see in your setup, yes this should work the way you wish. ---
**Author:** ![sachinjoshi](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@sachinjoshi](https://forums.percona.com/u/sachinjoshi)\ **Post date:** [March 11, 2025, 12:57pm UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/8 "2025-03-11T12:57:05Z")
Hi @Wayne_Leutwyler I have similar setup in staging as well so before do it on production i tried in staging first, and now I see one problem, yes my writer node changed but Backup Writer node and reader node weight is different in mysql\_servers & runtime\_mysql\_servers ```auto mysql> SELECT * FROM mysql_servers; +--------------+------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | hostgroup_id | hostname | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | +--------------+------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | 1 | 10.1.25.53 | 3306 | 0 | ONLINE | 100000 | 0 | 1000 | 0 | 0 | 0 | | | 90 | 10.1.25.52 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | | | 90 | 10.1.25.54 | 3306 | 0 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 10.1.25.53 | 3306 | 0 | ONLINE | 100000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 10.1.25.52 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 10.1.25.54 | 3306 | 0 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | | +--------------+------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ mysql> SELECT * FROM runtime_mysql_servers; +--------------+------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | hostgroup_id | hostname | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | +--------------+------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | 1 | 10.1.25.52 | 3306 | 0 | SHUNNED | 1000 | 0 | 1000 | 0 | 0 | 0 | | | 1 | 10.1.25.53 | 3306 | 0 | ONLINE | 100000 | 0 | 1000 | 0 | 0 | 0 | | | 1 | 10.1.25.54 | 3306 | 0 | SHUNNED | 1000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 10.1.25.52 | 3306 | 0 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 10.1.25.53 | 3306 | 0 | ONLINE | 100000 | 0 | 1000 | 0 | 0 | 0 | | | 2 | 10.1.25.54 | 3306 | 0 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | | | 90 | 10.1.25.52 | 3306 | 0 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | | | 90 | 10.1.25.54 | 3306 | 0 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | | +--------------+------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ 8 rows in set (0.01 sec) ``` hostgroup\_id 90 is backup writer and its showing both host have same weight but i defined different value for both host. Any idea why its like that ? ---
**Author:** ![Wayne\_Leutwyler](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/wayne_leutwyler/32/20829_2.png) [@Wayne\_Leutwyler](https://forums.percona.com/u/Wayne_Leutwyler)\ **Post date:** [March 11, 2025, 1:25pm UTC](https://forums.percona.com/t/want-to-change-writter-node-in-percona-cluster/37113/9 "2025-03-11T13:25:37Z")
ProxySQL may be adjusting the weight on its own. I would suggest you checkout the official documentations at [www.proxysql.com](http://www.proxysql.com). I have not encountered this situation in my own experience.