Not the answer you need?
Register and ask your own question!

All PXC nodes entered non primary state at the same time

zarerezarere ContributorInactive User Role Supporter
Hi Support team,



We have 3 DC-s .

DC1 have 2 nodes.

DC2 have 2 nodes

DC3 have 1 nodes.



Traffic is routed to one (VIP) node from DC1 and one (VIP) node from DC2.The DC3 node is used for backups analytics.We've made a switch update on DC1 and all nodes from all DC-s entered non primary state.Here are the logs from one of the nodes:
2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:48 11759 [Warning] WSREP: Quorum: No node with complete state: Version : 4 Flags : 0x3 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'haiku' Incoming addr: '10.42.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502826 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'demanet' Incoming addr: '10.100.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502826 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'erna' Incoming addr: '10.12.7.12:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'adler' Incoming addr: '10.12.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'burney' Incoming addr: '10.42.7.12:3306
The DC1 nodes are located in Amsterdam. The DC2 nodes are located in Berlin. The DC3 nodes are located in Frankfurt. Here are the logs from DC1 nodes: node1:
2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S
node2:
2019-04-11 14:27:26 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S
DC2: node1:
 2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3

node2:
 2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S

DC3:

node1:
2019-04-11 14:27:26 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S

We are awared about the timeouts in WAN env as written here:

http://galeracluster.com/documentati...ationtips.html

But we are still using the default timeouts.Can you please advise why we have all nodes in non primary state since only 1 DC was down at a time?The OS is
 Distributor ID: Ubuntu Description: Ubuntu 16.04.6 LTS Release: 16.04 Codename: xenial
 free -h  total used free shared buff/cache available Mem: 125G 99G 642M 235M 25G 25G Swap: 9.3G 1.6G 7.7G
 mysql> \s -------------- mysql Ver 14.14 Distrib 5.6.43-84.3, for debian-linux-gnu (x86_64) using 6.3 Connection id: 741302 Current database:  Current user: root@localhost SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server version: 5.6.43-84.3-56-log Percona XtraDB Cluster (GPL), Release rel84.3, Revision e2908fe, WSREP version 28.32, wsrep_28.32 Protocol version: 10 Connection: Localhost via UNIX socket Server characterset: utf8 Db characterset: utf8 Client characterset: utf8 Conn. characterset: utf8 UNIX socket: /var/run/mysqld/mysqld.sock Uptime: 20 days 22 hours 49 min 42 sec Threads: 490 Questions: 496378241 Slow queries: 9980 Opens: 393 Flush tables: 1 Open tables: 386 Queries per second avg: 274.214
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.