# All PXC nodes entered non primary state at the same time **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [May 7, 2019, 5:26am UTC](https://forums.percona.com/t/all-pxc-nodes-entered-non-primary-state-at-the-same-time/7002 "2019-05-07T05:26:00Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![zarere](https://avatars.discourse-cdn.com/v4/letter/z/ac8455/32.png) [@zarere](https://forums.percona.com/u/zarere)\ **Post date:** [May 7, 2019, 5:26am UTC](https://forums.percona.com/t/all-pxc-nodes-entered-non-primary-state-at-the-same-time/7002/1 "2019-05-07T05:26:00Z")
Hi Support team, We have 3 DC-s . DC1 have 2 nodes. DC2 have 2 nodes DC3 have 1 nodes. Traffic is routed to one (VIP) node from DC1 and one (VIP) node from DC2.The DC3 node is used for backups analytics.We’ve made a switch update on DC1 and all nodes from all DC-s entered non primary state.Here are the logs from one of the nodes: ```auto 2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:48 11759 [Warning] WSREP: Quorum: No node with complete state: Version : 4 Flags : 0x3 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'haiku' Incoming addr: '10.42.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502826 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'demanet' Incoming addr: '10.100.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502826 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'erna' Incoming addr: '10.12.7.12:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'adler' Incoming addr: '10.12.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'burney' Incoming addr: '10.42.7.12:3306 ``` The DC1 nodes are located in Amsterdam. The DC2 nodes are located in Berlin. The DC3 nodes are located in Frankfurt. Here are the logs from DC1 nodes: node1: ```auto 2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S ``` node2: ```auto 2019-04-11 14:27:26 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S ``` DC2: node1: ```auto 2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3 ``` node2: ```auto 2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S ``` DC3: node1: ```auto 2019-04-11 14:27:26 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S ``` We are awared about the timeouts in WAN env as written here: [URL=“[http://galeracluster.com/documentation-webpages/configurationtips.html](http://galeracluster.com/documentation-webpages/configurationtips.html)”][http://galeracluster.com/documentati...ationtips.html[/URL]](http://galeracluster.com/documentati...ationtips.html%5B/URL%5D) But we are still using the default timeouts.Can you please advise why we have all nodes in non primary state since only 1 DC was down at a time?The OS is ```auto Distributor ID: Ubuntu Description: Ubuntu 16.04.6 LTS Release: 16.04 Codename: xenial ``` ```auto free -h total used free shared buff/cache available Mem: 125G 99G 642M 235M 25G 25G Swap: 9.3G 1.6G 7.7G ``` ```auto mysql> \s -------------- mysql Ver 14.14 Distrib 5.6.43-84.3, for debian-linux-gnu (x86_64) using 6.3 Connection id: 741302 Current database: Current user: root@localhost SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server version: 5.6.43-84.3-56-log Percona XtraDB Cluster (GPL), Release rel84.3, Revision e2908fe, WSREP version 28.32, wsrep_28.32 Protocol version: 10 Connection: Localhost via UNIX socket Server characterset: utf8 Db characterset: utf8 Client characterset: utf8 Conn. characterset: utf8 UNIX socket: /var/run/mysqld/mysqld.sock Uptime: 20 days 22 hours 49 min 42 sec Threads: 490 Questions: 496378241 Slow queries: 9980 Opens: 393 Flush tables: 1 Open tables: 386 Queries per second avg: 274.214 ```