Hello!
My cluster contains from 3 nodes:
Amsterdam
New York
Frankfurt
OS: Ubuntu 16.04.6 LTS
Percona Cluster version - 5.7
my.cnf file:
[mysqld]
datadir=/var/lib/mysql
user=mysql
The Percona XtraDB Cluster 5.7 configuration file.
* IMPORTANT: Additional settings that can override those from this file!
The files must end with ‘.cnf’, otherwise they’ll be ignored.
Please make any edits and changes to the appropriate sectional files
included below.
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/percona-xtradb-cluster.conf.d/
wsrep_provider=/usr/lib/libgalera_smm.so
max_connections=400
wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://[nodes_ips]
wsrep_node_name=pxcny
wsrep_node_address=[current_node_ip]
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=“sstuser:sstpass”
pxc_strict_mode=ENFORCING
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
So! Sometimes 1-2 times a day NY node has those errors:
2019-04-03T23:46:44.853439Z 708036 [Warning] IP address ‘125.64.94.200’ could not be resolved: Name or service not known
2019-04-03T23:47:36.973584Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://amsterdam_ip:4567
2019-04-03T23:47:37.973708Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) reconnecting to 1f53f18d (tcp://amsterdam_ip:4567), attempt 0
2019-04-03T23:47:38.515111Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-03T23:47:39.467642Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-03T23:47:42.475200Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2019-04-03T23:47:45.975950Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://amsterdam_ip:4567
2019-04-03T23:47:46.973873Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-03T23:47:49.976568Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2019-04-04T03:20:42.184906Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection to peer 1f53f18d with addr tcp://amsterdam_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:42.185576Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://amsterdam_ip:4567
2019-04-04T03:20:43.185823Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) reconnecting to 1f53f18d (tcp://amsterdam_ip:4567), attempt 0
2019-04-04T03:20:46.195921Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection to peer 00000000 with addr tcp://amsterdam_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:47.611246Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-04T03:20:50.197061Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection to peer 00000000 with addr tcp://amsterdam_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:50.697396Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2019-04-04T03:20:52.197832Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://amsterdam_ip:4567
2019-04-04T03:20:53.198148Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) reconnecting to 1f53f18d (tcp://amsterdam_ip:4567), attempt 0
2019-04-04T03:20:55.607430Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-04T03:20:56.199301Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) connection to peer 00000000 with addr tcp://amsterdam_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:58.700023Z 0 [Note] WSREP: (252143b3, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
Meanwhile on Amsterdam node:
2019-04-03T23:47:36.572541Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection to peer 252143b3 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-03T23:47:36.573546Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://ny_ip:4567
2019-04-03T23:47:38.072733Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) reconnecting to 252143b3 (tcp://ny_ip:4567), attempt 0
2019-04-03T23:47:38.475317Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection established to 252143b3 tcp://ny_ip:4567
2019-04-03T23:47:39.607324Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection established to 252143b3 tcp://ny_ip:4567
2019-04-03T23:47:43.073597Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2019-04-03T23:47:45.574354Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection to peer 252143b3 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-03T23:47:45.574533Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://ny_ip:4567
2019-04-03T23:47:46.574474Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) reconnecting to 252143b3 (tcp://ny_ip:4567), attempt 0
2019-04-03T23:47:47.583490Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection established to 252143b3 tcp://ny_ip:4567
2019-04-03T23:47:51.075158Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2019-04-04T03:20:40.711399Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection to peer 252143b3 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:40.711601Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://ny_ip:4567
2019-04-04T03:20:42.211587Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) reconnecting to 252143b3 (tcp://ny_ip:4567), attempt 0
2019-04-04T03:20:45.712143Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection to peer 00000000 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:48.295260Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection established to 252143b3 tcp://ny_ip:4567
2019-04-04T03:20:51.712769Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection to peer 252143b3 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:53.212864Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) reconnecting to 252143b3 (tcp://ny_ip:4567), attempt 0
2019-04-04T03:20:55.746721Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) connection established to 252143b3 tcp://ny_ip:4567
2019-04-04T03:20:59.213535Z 0 [Note] WSREP: (1f53f18d, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
This problem causes some timeout disconnects on our production game server.
I couldn’t find any information about this issue.
Would be much appreciated for any help. Thanks!