Not the answer you need?
Register and ask your own question!

One of the nodes periodically reconnects to another one because of timeouts

seri52seri52 EntrantInactive User Role Novice
Hello!
My cluster contains from 3 nodes:
Amsterdam
New York
Frankfurt

OS: Ubuntu 16.04.6 LTS
Percona Cluster version - 5.7
my.cnf file:
[mysqld]
datadir=/var/lib/mysql
user=mysql

#
# The Percona XtraDB Cluster 5.7 configuration file.
#
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
# Please make any edits and changes to the appropriate sectional files
# included below.
#
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/percona-xtradb-cluster.conf.d/

wsrep_provider=/usr/lib/libgalera_smm.so
max_connections=400
wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://[nodes_ips]
wsrep_node_name=pxcny
wsrep_node_address=[current_node_ip]

wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth="sstuser:sstpass"

pxc_strict_mode=ENFORCING

binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2

So! Sometimes 1-2 times a day NY node has those errors:
2019-04-03T23:46:44.853439Z 708036 [Warning] IP address '125.64.94.200' could not be resolved: Name or service not known
2019-04-03T23:47:36.973584Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://amsterdam_ip:4567
2019-04-03T23:47:37.973708Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') reconnecting to 1f53f18d (tcp://amsterdam_ip:4567), attempt 0
2019-04-03T23:47:38.515111Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-03T23:47:39.467642Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-03T23:47:42.475200Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-04-03T23:47:45.975950Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://amsterdam_ip:4567
2019-04-03T23:47:46.973873Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-03T23:47:49.976568Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-04-04T03:20:42.184906Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection to peer 1f53f18d with addr tcp://amsterdam_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:42.185576Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://amsterdam_ip:4567
2019-04-04T03:20:43.185823Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') reconnecting to 1f53f18d (tcp://amsterdam_ip:4567), attempt 0
2019-04-04T03:20:46.195921Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://amsterdam_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:47.611246Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-04T03:20:50.197061Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://amsterdam_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:50.697396Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-04-04T03:20:52.197832Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://amsterdam_ip:4567
2019-04-04T03:20:53.198148Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') reconnecting to 1f53f18d (tcp://amsterdam_ip:4567), attempt 0
2019-04-04T03:20:55.607430Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection established to 1f53f18d tcp://amsterdam_ip:4567
2019-04-04T03:20:56.199301Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://amsterdam_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:58.700023Z 0 [Note] WSREP: (252143b3, 'tcp://0.0.0.0:4567') turning message relay requesting off

Meanwhile on Amsterdam node:

2019-04-03T23:47:36.572541Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection to peer 252143b3 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-03T23:47:36.573546Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://ny_ip:4567
2019-04-03T23:47:38.072733Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') reconnecting to 252143b3 (tcp://ny_ip:4567), attempt 0
2019-04-03T23:47:38.475317Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection established to 252143b3 tcp://ny_ip:4567
2019-04-03T23:47:39.607324Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection established to 252143b3 tcp://ny_ip:4567
2019-04-03T23:47:43.073597Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-04-03T23:47:45.574354Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection to peer 252143b3 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-03T23:47:45.574533Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://ny_ip:4567
2019-04-03T23:47:46.574474Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') reconnecting to 252143b3 (tcp://ny_ip:4567), attempt 0
2019-04-03T23:47:47.583490Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection established to 252143b3 tcp://ny_ip:4567
2019-04-03T23:47:51.075158Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-04-04T03:20:40.711399Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection to peer 252143b3 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:40.711601Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://ny_ip:4567
2019-04-04T03:20:42.211587Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') reconnecting to 252143b3 (tcp://ny_ip:4567), attempt 0
2019-04-04T03:20:45.712143Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:48.295260Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection established to 252143b3 tcp://ny_ip:4567
2019-04-04T03:20:51.712769Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection to peer 252143b3 with addr tcp://ny_ip:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2019-04-04T03:20:53.212864Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') reconnecting to 252143b3 (tcp://ny_ip:4567), attempt 0
2019-04-04T03:20:55.746721Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') connection established to 252143b3 tcp://ny_ip:4567
2019-04-04T03:20:59.213535Z 0 [Note] WSREP: (1f53f18d, 'tcp://0.0.0.0:4567') turning message relay requesting off

This problem causes some timeout disconnects on our production game server.

I couldn't find any information about this issue.
Would be much appreciated for any help. Thanks!
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.