Xtradb cluster at google cloud engine "Failed to open backend connection: -110"

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Hello,

I’v deployed cluster using deploy tool provided by google, that one works fine, but because it’s based on debian rather than ubuntu, i’v decided to setup new cluster. I’v got problem with cluster deployed by myself.
I have copied my.cnf and wsrep.cnf from working cluster and i’v changed nodes names, now it looks like this:

The MySQL database server configuration file.

!includedir /etc/mysql/conf.d/

Required mysqld options for Percona XtraDB Cluster

[mysqld]
binlog_format=ROW
innodb_autoinc_lock_mode=2
innodb_flush_method=O_DIRECT
innodb_flush_log_at_trx_commit=2
datadir=/var/lib/mysql
log_error = /var/log/mysql/error.log

Recommended mysqld options for Percona XtraDB Cluster

Set innodb-buffer-pool-size to 70% of available memory

innodb-buffer-pool-size=4G

wsrep.cnf:

[mysqld]
wsrep_provider=/usr/lib/libgalera_smm.so
wsrep_provider_options=“gcache.size=2G; gcs.fc_limit=128”
wsrep_sst_method=xtrabackup-v2
wsrep_slave_threads=4
wsrep_node_name=cluster-db-1a
wsrep_sst_auth=sstuser:Wathost0
wsrep_cluster_name=“cluster-db-1”
wsrep_node_address=10.240.0.7
wsrep_cluster_address=gcomm://cluster-db-1a,cluster-db-1b,cluster-db-1c

log:

2015-12-17 11:44:34 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see do$
2015-12-17 11:44:34 0 [Note] /usr/sbin/mysqld (mysqld 5.6.26-74.0-56) starting as process 29561 …
2015-12-17 11:44:34 29561 [Note] WSREP: Read nil XID from storage engines, skipping position init
2015-12-17 11:44:34 29561 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib/libgalera_smm.so’
2015-12-17 11:44:34 29561 [Note] WSREP: wsrep_load(): Galera 3.12.2(rf3e626d) by Codership Oy <info@codership.com> loaded successfully.
2015-12-17 11:44:34 29561 [Note] WSREP: CRC-32C: using hardware acceleration.
2015-12-17 11:44:34 29561 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2015-12-17 11:44:34 29561 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.240.0.7; base_port = 4567; cert.log_conflicts$
2015-12-17 11:44:34 29561 [Note] WSREP: Service thread queue flushed.
2015-12-17 11:44:34 29561 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2015-12-17 11:44:34 29561 [Note] WSREP: wsrep_sst_grab()
2015-12-17 11:44:34 29561 [Note] WSREP: Start replication
2015-12-17 11:44:34 29561 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2015-12-17 11:44:34 29561 [Note] WSREP: protonet asio version 0
2015-12-17 11:44:34 29561 [Note] WSREP: Using CRC-32C for message checksums.
2015-12-17 11:44:34 29561 [Note] WSREP: backend: asio
2015-12-17 11:44:34 29561 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2015-12-17 11:44:34 29561 [Note] WSREP: restore pc from disk failed
2015-12-17 11:44:34 29561 [Note] WSREP: GMCast version 0
2015-12-17 11:44:34 29561 [Note] WSREP: (8b28e8a6, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
2015-12-17 11:44:34 29561 [Note] WSREP: (8b28e8a6, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
2015-12-17 11:44:34 29561 [Note] WSREP: EVS version 0
2015-12-17 11:44:34 29561 [Note] WSREP: gcomm: connecting to group ‘cluster-db-1’, peer ‘cluster-db-1a:,cluster-db-1b:,cluster-db-1c:’
2015-12-17 11:44:34 29561 [Warning] WSREP: (8b28e8a6, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.240.0.7:4567’ points to own listening address, blacklisti$
2015-12-17 11:44:37 29561 [Warning] WSREP: no nodes coming from prim view, prim not possible
2015-12-17 11:44:37 29561 [Note] WSREP: view(view_id(NON_PRIM,8b28e8a6,1) memb {
8b28e8a6,0
} joined {
} left {
} partitioned {
})
2015-12-17 11:44:38 29561 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50439S), skipping check
2015-12-17 11:45:07 29561 [Note] WSREP: view((empty))
2015-12-17 11:45:07 29561 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():162
2015-12-17 11:45:07 29561 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():206: Failed to open backend connection: -110 (Connection timed out)
2015-12-17 11:45:07 29561 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1379: Failed to open channel ‘cluster-db-1’ at 'gcomm://cluster-db-1a,cluster-db-1b$
2015-12-17 11:45:07 29561 [ERROR] WSREP: gcs connect failed: Connection timed out
2015-12-17 11:45:07 29561 [ERROR] WSREP: wsrep::connect(gcomm://cluster-db-1a,cluster-db-1b,cluster-db-1c) failed: 7
2015-12-17 11:45:07 29561 [ERROR] Aborting

2015-12-17 11:45:07 29561 [Note] WSREP: Service disconnected.
2015-12-17 11:45:08 29561 [Note] WSREP: Some threads may fail to exit.
2015-12-17 11:45:08 29561 [Note] Binlog end
2015-12-17 11:45:08 29561 [Note] /usr/sbin/mysqld: Shutdown complete

151217 11:45:08 mysqld_safe mysqld from pid file /var/lib/mysql/cluster-db-1a.pid ended

Tags for this VM at GCE are copied from working cluster, apparmor is disabled for mysqld, i have tried to start node with service mysql start --wsrep-new-cluster", database starts fine without wsrep enabled (with default config)

2

Are you sure that the needed ports are open?
[url]http://galeracluster.com/documentation-webpages/firewallsettings.html[/url]

3

Yes, i’m sure. iptables are empty, to be sure i’v added accept rules, no changes. I’v checked firewall at google cloud engine, all trafiic internally is allowed. It makes me want to scream, because i have been fighting with that for over a week now without any visible results, it should be working by now, if it wont soon, propably i’ll be fired…

Problem was “wsrep_cluster_address=gcomm://cluster-db-1a,cluster-db-1b,cluster-db-1c”
When set “cluster-db-1a,” on donor node, it wouldn’t start up because it was blacklisting itself from joining… When it was removed, it started fine, but i’v got another problem, I cant get second node to join. Can’t make xtrabackup or rsync copy, again error logs were very “Helpfull”, it appeared ,that I havent installed meta package percona-xtradb-cluster-56 at first node, only other packages, that caused problems with innobackup…

OK “houston, we have (another problem)”, partial lack of replication among cluster. It will be in new topic…

4

Hi , Vein.
I have the same problem like you . I want to know what did you move and it will started fine ?
Thank you