Hi Support team,
We have 3 DC-s .
DC1 have 2 nodes.
DC2 have 2 nodes
DC3 have 1 nodes.
Traffic is routed to one (VIP) node from DC1 and one (VIP) node from DC2.The DC3 node is used for backups analytics.We’ve made a switch update on DC1 and all nodes from all DC-s entered non primary state.Here are the logs from one of the nodes:
2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:48 11759 [Warning] WSREP: Quorum: No node with complete state: Version : 4 Flags : 0x3 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'haiku' Incoming addr: '10.42.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502826 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'demanet' Incoming addr: '10.100.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502826 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'erna' Incoming addr: '10.12.7.12:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'adler' Incoming addr: '10.12.7.11:3306' Version : 4 Flags : 0x2 Protocols : 0 / 9 / 3 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 2c926a17-5487-11e9-8fc2-567a2d05135a Prim seqno : 1080 First seqno : 2015502825 Last seqno : 2019402781 Prim JOINED : 5 State UUID : 377b9a2a-5c55-11e9-9790-f36900a869d3 Group UUID : dcf19856-09f2-11e5-b556-b625f1f77d8f Name : 'burney' Incoming addr: '10.42.7.12:3306
The DC1 nodes are located in Amsterdam. The DC2 nodes are located in Berlin. The DC3 nodes are located in Frankfurt. Here are the logs from DC1 nodes: node1:
2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 11759 [Note] WSREP: (04626010, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S
node2:
2019-04-11 14:27:26 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 2833 [Note] WSREP: (6a590394, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S
DC2: node1:
2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 4627 [Note] WSREP: (5b800051, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3
node2:
2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:25 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 6a590394 with addr tcp://10.42.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 49557415 with addr tcp://10.100.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 24394 [Note] WSREP: (59f21cd2, 'tcp://0.0.0.0:4567') connection to peer 04626010 with addr tcp://10.42.7.11:4567 timed out, no messages seen in PT3S
DC3:
node1:
2019-04-11 14:27:26 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:26 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 5b800051 with addr tcp://10.12.7.11:4567 timed out, no messages seen in PT3S 2019-04-11 14:27:41 14089 [Note] WSREP: (49557415, 'tcp://0.0.0.0:4567') connection to peer 59f21cd2 with addr tcp://10.12.7.12:4567 timed out, no messages seen in PT3S
We are awared about the timeouts in WAN env as written here:
[URL=“http://galeracluster.com/documentation-webpages/configurationtips.html”]http://galeracluster.com/documentati...ationtips.html[/URL]
But we are still using the default timeouts.Can you please advise why we have all nodes in non primary state since only 1 DC was down at a time?The OS is
Distributor ID: Ubuntu Description: Ubuntu 16.04.6 LTS Release: 16.04 Codename: xenial
free -h total used free shared buff/cache available Mem: 125G 99G 642M 235M 25G 25G Swap: 9.3G 1.6G 7.7G
mysql> \s -------------- mysql Ver 14.14 Distrib 5.6.43-84.3, for debian-linux-gnu (x86_64) using 6.3 Connection id: 741302 Current database: Current user: root@localhost SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server version: 5.6.43-84.3-56-log Percona XtraDB Cluster (GPL), Release rel84.3, Revision e2908fe, WSREP version 28.32, wsrep_28.32 Protocol version: 10 Connection: Localhost via UNIX socket Server characterset: utf8 Db characterset: utf8 Client characterset: utf8 Conn. characterset: utf8 UNIX socket: /var/run/mysqld/mysqld.sock Uptime: 20 days 22 hours 49 min 42 sec Threads: 490 Questions: 496378241 Slow queries: 9980 Opens: 393 Flush tables: 1 Open tables: 386 Queries per second avg: 274.214