# Node is not connecting. **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [December 26, 2013, 1:59am UTC](https://forums.percona.com/t/node-is-not-connecting/3163 "2013-12-26T01:59:11Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Periyannan](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@Periyannan](https://forums.percona.com/u/Periyannan)\ **Post date:** [December 26, 2013, 1:59am UTC](https://forums.percona.com/t/node-is-not-connecting/3163/1 "2013-12-26T01:59:11Z")
Hi , We are using three node Percona XtraDB Cluster. Right now only one node(Node No: 2) is running. All the (Node No: 1, Node No: 3) remaining nodes are down. here I attached the Down nodes error log. Please provide your suggestion to solve. Advance Thanks. Node No: 3 Error Log: 131224 4:16:06 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at gcomm/src/pc.cpp:connect():157 131224 4:16:06 [ERROR] WSREP: gcs/src/gcs\_core.c:gcs\_core\_open():195: Failed to open backend connection: -110 (Connection timed out) 131224 4:16:06 [ERROR] WSREP: gcs/src/gcs.c:gcs\_open():1290: Failed to open channel ‘ecomm’ at ‘gcomm://X.X.X.2’: -110 (Connection timed out) 131224 4:16:06 [ERROR] WSREP: gcs connect failed: Connection timed out 131224 4:16:06 [ERROR] WSREP: wsrep::connect() failed: 6 131224 4:16:06 [ERROR] Aborting 131224 4:16:06 [Note] WSREP: Service disconnected. 131224 4:16:07 [Note] WSREP: Some threads may fail to exit. 131224 4:16:07 [Note] /usr/sbin/mysqld: Shutdown complete 131224 04:16:07 mysqld\_safe mysqld from pid file /mnt/data//FD-R14-db3.pid ended 131224 04:19:56 mysqld\_safe Starting mysqld daemon with databases from /mnt/data/ 131224 4:19:56 [Note] Flashcache bypass: disabled 131224 4:19:56 [Note] Flashcache setup error is : ioctl failed 131224 4:19:56 [Note] WSREP: Read nil XID from storage engines, skipping position init 131224 4:19:56 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 131224 4:19:56 [Note] WSREP: wsrep\_load(): Galera 2.2(r114) by Codership Oy \ loaded succesfully. 131224 4:19:56 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 131224 4:19:56 [Note] WSREP: Reusing existing ‘/mnt/data//galera.cache’. 131224 4:19:56 [Note] WSREP: Passing config to GCS: base\_host = X.X.X.3; gcache.dir = /mnt/data/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /mnt/data//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 0.5; gcs.fc\_limit = 16; gcs.fc\_master\_slave = NO; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = NO; replicator.causal\_read\_timeout = PT30S; replicator.commit\_order = 3 131224 4:19:56 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 131224 4:19:56 [Note] WSREP: wsrep\_sst\_grab() 131224 4:19:56 [Note] WSREP: Start replication 131224 4:19:56 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 131224 4:19:56 [Note] WSREP: protonet asio version 0 131224 4:19:56 [Note] WSREP: backend: asio 131224 4:19:56 [Note] WSREP: GMCast version 0 131224 4:19:56 [Note] WSREP: (a5560fc4-6c52-11e3-0800-6de37b8550ef, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 131224 4:19:56 [Note] WSREP: (a5560fc4-6c52-11e3-0800-6de37b8550ef, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 131224 4:19:56 [Note] WSREP: EVS version 0 131224 4:19:56 [Note] WSREP: PC version 0 131224 4:19:56 [Note] WSREP: gcomm: connecting to group ‘ecomm’, peer ‘X.X.X.2:’ 131224 4:19:57 [Note] WSREP: declaring 771b0086-6c52-11e3-0800-6686b0fba2c7 stable 131224 4:19:57 [Note] WSREP: view(view\_id(PRIM,771b0086-6c52-11e3-0800-6686b0fba2c7,2) memb { 771b0086-6c52-11e3-0800-6686b0fba2c7, a5560fc4-6c52-11e3-0800-6de37b8550ef, } joined { } left { } partitioned { }) 131224 4:19:57 [Note] WSREP: gcomm: connected 131224 4:19:57 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 131224 4:19:57 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) 131224 4:19:57 [Note] WSREP: Opened channel ‘ecomm’ 131224 4:19:57 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 1, memb\_num = 2 131224 4:19:57 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 131224 4:19:57 [Note] WSREP: Waiting for SST to complete. 131224 4:19:57 [Note] WSREP: STATE EXCHANGE: sent state msg: a59fd8b1-6c52-11e3-0800-74c11aaf856b 131224 4:19:57 [Note] WSREP: STATE EXCHANGE: got state msg: a59fd8b1-6c52-11e3-0800-74c11aaf856b from 0 (FD-R14-db2) 131224 4:19:57 [Note] WSREP: STATE EXCHANGE: got state msg: a59fd8b1-6c52-11e3-0800-74c11aaf856b from 1 (FD-R14-db3) 131224 4:19:57 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf\_id = 1, members = 1/2 (joined/total), act\_id = 172, last\_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = 493d95e8-6c50-11e3-0800-b099dbf50e3d 131224 4:19:57 [Note] WSREP: Flow-control interval: [12, 23] 131224 4:19:57 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 172) 131224 4:19:57 [Note] WSREP: State transfer required: Group state: 493d95e8-6c50-11e3-0800-b099dbf50e3d:172 Local state: 00000000-0000-0000-0000-000000000000:-1 131224 4:19:57 [Note] WSREP: New cluster view: global state: 493d95e8-6c50-11e3-0800-b099dbf50e3d:172, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 2 131224 4:19:57 [Warning] WSREP: Gap in state sequence. Need state transfer. 131224 4:19:59 [Note] WSREP: Running: ‘wsrep\_sst\_rsync ‘joiner’ ‘X.X.X.3’ ‘’ ‘/mnt/data/’ ‘/etc/my.cnf’ ‘15049’ 2\>sst.err’ 131224 4:19:59 [Note] WSREP: Prepared SST request: rsync|X.X.X.3:4444/rsync\_sst 131224 4:19:59 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 131224 4:19:59 [Note] WSREP: Assign initial position for certification: 172, protocol version: 2 131224 4:19:59 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (493d95e8-6c50-11e3-0800-b099dbf50e3d): 1 (Operation not permitted) at galera/src/replicator\_str.cpp:prepare\_for\_IST():440. IST will be unavailable. 131224 4:19:59 [Note] WSREP: Node 1 (FD-R14-db3) requested state transfer from ‘_any_’. Selected 0 (FD-R14-db2)(SYNCED) as donor. 131224 4:19:59 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 172) 131224 4:19:59 [Note] WSREP: Requesting state transfer: success, donor: 0 Node No 1 : Error Log: at galera/src/replicator\_str.cpp:prepare\_for\_IST():440. IST will be unavailable. 131224 4:12:17 [Note] WSREP: Node 1 (FD-R14-db1) requested state transfer from ‘_any_’. Selected 0 (FD-R14-db2)(SYNCED) as donor. 131224 4:12:17 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 172) 131224 4:12:17 [Note] WSREP: Requesting state transfer: success, donor: 0 131224 04:17:09 mysqld\_safe Starting mysqld daemon with databases from /mnt/data/ 131224 4:17:09 [Note] Flashcache bypass: disabled 131224 4:17:09 [Note] Flashcache setup error is : ioctl failed 131224 4:17:09 [Note] WSREP: Read nil XID from storage engines, skipping position init 131224 4:17:09 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 131224 4:17:09 [Note] WSREP: wsrep\_load(): Galera 2.2(r114) by Codership Oy \ loaded succesfully. 131224 4:17:09 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 131224 4:17:09 [Note] WSREP: Reusing existing ‘/mnt/data//galera.cache’. 131224 4:17:09 [Note] WSREP: Passing config to GCS: base\_host = X.X.X.1; gcache.dir = /mnt/data/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /mnt/data//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 0.5; gcs.fc\_limit = 16; gcs.fc\_master\_slave = NO; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = NO; replicator.causal\_read\_timeout = PT30S; replicator.commit\_order = 3 131224 4:17:09 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 131224 4:17:09 [Note] WSREP: wsrep\_sst\_grab() 131224 4:17:09 [Note] WSREP: Start replication 131224 4:17:09 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 131224 4:17:09 [Note] WSREP: protonet asio version 0 131224 4:17:09 [Note] WSREP: backend: asio 131224 4:17:09 [Note] WSREP: GMCast version 0 131224 4:17:09 [Note] WSREP: (41e1954d-6c52-11e3-0800-a5888b35da99, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 131224 4:17:09 [Note] WSREP: (41e1954d-6c52-11e3-0800-a5888b35da99, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 131224 4:17:09 [Note] WSREP: EVS version 0 131224 4:17:09 [Note] WSREP: PC version 0 131224 4:17:09 [Note] WSREP: gcomm: connecting to group ‘ecomm’, peer ‘X.X.X.2:’ 131224 4:17:10 [Note] WSREP: declaring 493c5523-6c50-11e3-0800-25c9bc4e1fff stable 131224 4:17:10 [Note] WSREP: view(view\_id(NON\_PRIM,41e1954d-6c52-11e3-0800-a5888b35da99,8) memb { 41e1954d-6c52-11e3-0800-a5888b35da99, 493c5523-6c50-11e3-0800-25c9bc4e1fff, } joined { } left { } partitioned { 0975f0b3-6c52-11e3-0800-2d07df149f30, 9183e102-6c51-11e3-0800-62536ac728a9, }) 131224 4:17:40 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at gcomm/src/pc.cpp:connect():157 131224 4:17:40 [ERROR] WSREP: gcs/src/gcs\_core.c:gcs\_core\_open():195: Failed to open backend connection: -110 (Connection timed out) 131224 4:17:40 [ERROR] WSREP: gcs/src/gcs.c:gcs\_open():1290: Failed to open channel ‘ecomm’ at ‘gcomm://X.X.X.2’: -110 (Connection timed out) 131224 4:17:40 [ERROR] WSREP: gcs connect failed: Connection timed out 131224 4:17:40 [ERROR] WSREP: wsrep::connect() failed: 6 131224 4:17:40 [ERROR] Aborting 131224 4:17:40 [Note] WSREP: Service disconnected. 131224 4:17:41 [Note] WSREP: Some threads may fail to exit. 131224 4:17:41 [Note] /usr/sbin/mysqld: Shutdown complete 131224 04:17:41 mysqld\_safe mysqld from pid file /mnt/data//FD-R14-db1.pid ended Advance Thanks, Periyannan N. ---
**Author:** ![madhusudan](https://avatars.discourse-cdn.com/v4/letter/m/76d3ee/32.png) [@madhusudan](https://forums.percona.com/u/madhusudan)\ **Post date:** [December 26, 2013, 5:33am UTC](https://forums.percona.com/t/node-is-not-connecting/3163/2 "2013-12-26T05:33:27Z")
looks like both node1 & 3 needs SST. make node 2 as primary in mysql prompt give SET GLOBAL wsrep\_provider\_options=“pc.bootstrap=1”; (if u have done already then leave it) then put =node2 address in both node 1& node3 my.cnf file and then try to start again. If you provide some more information about you setup it will be helpful, like OS,percona version,status of innobackup.backup.log,innobackup.prepare.log (if any). ---
**Author:** ![Periyannan](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@Periyannan](https://forums.percona.com/u/Periyannan)\ **Post date:** [December 26, 2013, 5:55am UTC](https://forums.percona.com/t/node-is-not-connecting/3163/3 "2013-12-26T05:55:50Z")
Hi Madhusudan, First Thanks for your reply. Please find the below are my server setup: OS Version: CentOS 6.3 (Final) Percona Version: Percona-XtraDB-Cluster-server-5.5.27-23.6.356.rhel6.x86\_64 And then we are using rsync for SST. I think we dont have innobackup.backup.log,innobackup.prepare.log. Please provide any more suggestion, i am new to this Percona XtraDB Cluster. Advance Thanks, Regards, Periyannan N. ---
**Author:** ![madhusudan](https://avatars.discourse-cdn.com/v4/letter/m/76d3ee/32.png) [@madhusudan](https://forums.percona.com/u/madhusudan)\ **Post date:** [December 26, 2013, 6:03am UTC](https://forums.percona.com/t/node-is-not-connecting/3163/4 "2013-12-26T06:03:49Z")
did u try, what I suggested.? and your cluster is a production or in testing…? more info about recovering cluster… [URL][Percona XtraDB Cluster](http://www.percona.com/doc/percona-xtradb-cluster/5.5/manual/failover.html%5B/URL%5D) ---
**Author:** ![Periyannan](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@Periyannan](https://forums.percona.com/u/Periyannan)\ **Post date:** [December 26, 2013, 6:13am UTC](https://forums.percona.com/t/node-is-not-connecting/3163/5 "2013-12-26T06:13:01Z")
Hi, This production database, we took down time tomorrow only, we could not able to check right now. And then for information, we are using galera-2.0 Thanks and Regards, Periyannan N. ---
**Author:** ![Periyannan](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@Periyannan](https://forums.percona.com/u/Periyannan)\ **Post date:** [December 26, 2013, 6:24am UTC](https://forums.percona.com/t/node-is-not-connecting/3163/6 "2013-12-26T06:24:00Z")
Hi, For your Information, And then here, three nodes are act as a master only(multi master). Now the node no: 2 is running, we didnt execute the “SET GLOBAL wsrep\_provider\_options=“pc.bootstrap=1”;” from begin onwards. we didnt put “wsrep\_sst\_donor=node2” in my.cnf file from begin onwards. Thanks and Regards, Periyannan N. ---
**Author:** ![Periyannan](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@Periyannan](https://forums.percona.com/u/Periyannan)\ **Post date:** [December 26, 2013, 6:29am UTC](https://forums.percona.com/t/node-is-not-connecting/3163/7 "2013-12-26T06:29:34Z")
Hi, whether i need to delete the below log before execute the rsync command. GRA\_1\_151865761.log rsync\_sst.conf rsync\_sst.pid Thanks and Regards, Periyannan N. ---
**Author:** ![Chintan](https://avatars.discourse-cdn.com/v4/letter/c/a3d4f5/32.png) [@Chintan](https://forums.percona.com/u/Chintan)\ **Post date:** [September 9, 2020, 4:34am UTC](https://forums.percona.com/t/node-is-not-connecting/3163/8 "2020-09-09T04:34:26Z")
While adding node to the Cluster server I m not able to start mysql service. And I have read the logs and I found below error on logs  Failed to open Gcomm backend connections:  -110 (connection timeout)  Failed to open channel ‘mysql1-cluster’ at ‘Gcomm:/(cluster IP)’ : -110 (connection time out) Please help to resolve this issue…!