# Third Node cannot connect / 2 Nodes in AWS / 1 node inhouse with NAT **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [June 13, 2015, 12:23am UTC](https://forums.percona.com/t/third-node-cannot-connect-2-nodes-in-aws-1-node-inhouse-with-nat/4263 "2015-06-13T00:23:21Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![NeonNexxus](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@NeonNexxus](https://forums.percona.com/u/NeonNexxus)\ **Post date:** [June 13, 2015, 12:23am UTC](https://forums.percona.com/t/third-node-cannot-connect-2-nodes-in-aws-1-node-inhouse-with-nat/4263/1 "2015-06-13T00:23:21Z")
Hi guys … I always read all posts and normally they solve my issues. I would like to take the time to explain my situation. Please bear with me … All node using Ubuntu 12.04 AWS nodes using percona Server version: 5.5.34-31.1 Percona XtraDB Cluster (GPL), Release 31.1, wsrep\_25.9.r3928 Both AWS have elastic public IP In house node using Percona 5.5.41-25.11-853 AWS configured of course with the internal IPs 10.xxx.xxx.xxx and In house its a 172.21.12.11 through NAT on a public IP 199.xxx.xxx.xxx Both AWS nodes have in the gcomm the public IP of the in house server and the in house server has the 2 elastic ips and the internal 172.21.12.11 And here is the dump of the tail -f /var/lib/mysql/novo.err 150612 23:25:52 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql 150612 23:25:52 mysqld\_safe WSREP: Running position recovery with --log\_error=‘/var/lib/mysql/wsrep\_recovery.UoqEvJ’ --pid-file=‘/var/lib/mysql/noomedici-recover.pid’ 150612 23:25:55 mysqld\_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1 150612 23:25:55 [Note] WSREP: wsrep\_start\_position var submitted: ‘00000000-0000-0000-0000-000000000000:-1’ 150612 23:25:55 [Note] WSREP: Read nil XID from storage engines, skipping position init 150612 23:25:55 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib/libgalera\_smm.so’ 150612 23:25:55 [Note] WSREP: wsrep\_load(): Galera 2.12(r318911d) by Codership Oy \ loaded successfully. 150612 23:25:55 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 150612 23:25:55 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’. 150612 23:25:55 [Note] WSREP: Passing config to GCS: base\_host = 172.21.12.11; base\_port = 4567; cert.log\_conflicts = no; debug = no; evs.inactivecheck\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 4; evs.stats\_eport\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 2; evs.view\_forget\_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.kep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0;gcs.fc\_factor = 1.0; gcs.fc\_limit = 16; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 923372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignor\_quorum = false; pc.ignore\_sb = false; pc.npvo = false; pc.version = 0; pc.wait\_prim = true; pc.wait\_prim\_timeout = P30S; pc.weight = 1; protonet.ackend = asio; pr 150612 23:25:55 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 150612 23:25:55 [Note] WSREP: wsrep\_sst\_grab() 150612 23:25:55 [Note] WSREP: Start replication 150612 23:25:55 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 150612 23:25:55 [Note] WSREP: protonet asio version 0 150612 23:25:55 [Note] WSREP: backend: asio 150612 23:25:55 [Note] WSREP: GMCast version 0 150612 23:25:55 [Note] WSREP: (aa293f06, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 150612 23:25:55 [Note] WSREP: (aa293f06, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 150612 23:25:55 [Note] WSREP: EVS version 0 150612 23:25:55 [Note] WSREP: PC version 0 150612 23:25:55 [Note] WSREP: gcomm: connecting to group ‘Percona-XtraDB-Cluster’, peer ‘107.xx.xxx.xx:,54.xxx.xx.xxx:,172.21.12.11:’ 150612 23:25:55 [Warning] WSREP: (aa293f06, ‘tcp://0.0.0.0:4567’) address ‘tcp://172.21.12.11:4567’ points to own listening address, blacklisting 150612 23:25:55 [Note] WSREP: (aa293f06, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://10.147.174.188:4567 150612 23:25:55 [Note] WSREP: declaring 919c045d at tcp://10.147.174.188:4567 stable 150612 23:25:55 [Note] WSREP: declaring af980804 at tcp://10.154.176.220:4567 stable 150612 23:25:56 [Note] WSREP: Node 919c045d state prim 150612 23:25:57 [Note] WSREP: gcomm: connected 150612 23:25:57 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 150612 23:25:57 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) 150612 23:25:57 [Note] WSREP: Opened channel ‘Percona-XtraDB-Cluster’ 150612 23:25:57 [Note] WSREP: Waiting for SST to complete. 150612 23:25:57 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 1, memb\_num = 3 150612 23:25:57 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 150612 23:25:57 [Note] WSREP: STATE EXCHANGE: sent state msg: ab12c5d2-118c-11e5-a9f3-cf551b1e5734 150612 23:25:57 [Note] WSREP: STATE EXCHANGE: got state msg: ab12c5d2-118c-11e5-a9f3-cf551b1e5734 from 0 (db2) 150612 23:25:57 [Note] WSREP: STATE EXCHANGE: got state msg: ab12c5d2-118c-11e5-a9f3-cf551b1e5734 from 2 (db1) 150612 23:25:57 [Note] WSREP: STATE EXCHANGE: got state msg: ab12c5d2-118c-11e5-a9f3-cf551b1e5734 from 1 (db3) 150612 23:25:57 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf\_id = 2, members = 2/3 (joined/total), act\_id = 35909744, last\_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = d3538475-b069-11e3-92aa-c323b5f792a4 150612 23:25:57 [Note] WSREP: Flow-control interval: [28, 28] 150612 23:25:57 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 35909744) 150612 23:25:57 [Note] WSREP: State transfer required: Group state: d3538475-b069-11e3-92aa-c323b5f792a4:35909744 Local state: 00000000-0000-0000-0000-000000000000:-1 150612 23:25:57 [Note] WSREP: New cluster view: global state: d3538475-b069-11e3-92aa-c323b5f792a4:35909744, view# 3: Primary, number of nodes: 3,my index: 1, protocol version 2 150612 23:25:57 [Note] WSREP: closing client connections for protocol change 3 → 2 150612 23:25:58 [Note] WSREP: (aa293f06, ‘tcp://0.0.0.0:4567’) turning message relay requesting off 150612 23:25:59 [Warning] WSREP: Gap in state sequence. Need state transfer. 150612 23:25:59 [Note] WSREP: Running: ‘wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘172.21.12.11’ --auth ‘sstuser:zASDF@7890@yhn’ --datadir ‘/ar/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘25936’’ WSREP\_SST: [INFO] Streaming with tar (20150612 23:25:59.618) WSREP\_SST: [INFO] Using socat as streamer (20150612 23:25:59.624) WSREP\_SST: [INFO] Stale sst\_in\_progress file: /var/lib/mysql//sst\_in\_progress (20150612 23:25:59.636) WSREP\_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20150612 23:25:5.664) 150612 23:25:59 [Note] WSREP: Prepared SST request: xtrabackup|172.21.12.11:4444/xtrabackup\_sst 150612 23:25:59 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 150612 23:25:59 [Note] WSREP: Assign initial position for certification: 35909744, protocol version: 2 150612 23:25:59 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does no match group state UUID (d3538475-b069-11e3-92aa-c323b5f792a4): 1 (Operation not permitted) at galera/src/replicator\_str.cpp:prepare\_for\_IST():447. IST will be unavailable. 150612 23:25:59 [Note] WSREP: Node 1 (db3) requested state transfer from ‘_any_’. Selected 0 (db2)(SYNCED) as donor. 150612 23:25:59 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 35909744) 150612 23:25:59 [Note] WSREP: Requesting state transfer: success, donor: 0 150612 23:27:04 [Warning] WSREP: 0 (db2): State transfer to 1 (db3) failed: -1 (Operation not permitted) 150612 23:27:04 [ERROR] WSREP: gcs/src/gcs\_group.cpp:long int gcs\_group\_handle\_join\_msg(gcs\_group\_t\*, const gcs\_recv\_msg\_t\*)():717: Will never recive state. Need to abort. 150612 23:27:04 [Note] WSREP: gcomm: terminating thread 150612 23:27:04 [Note] WSREP: gcomm: joining thread 150612 23:27:04 [Note] WSREP: gcomm: closing backend 150612 23:27:04 [Note] WSREP: gcomm: closed 150612 23:27:04 [Note] WSREP: /usr/sbin/mysqld: Terminated. Aborted (core dumped) 150612 23:27:04 mysqld\_safe mysqld from pid file /var/lib/mysql/novomedici.pid ended FYI Galera used on the AWS servers is … | wsrep\_provider\_version | 2.8(r165) | and on the in house server as you can see above Galera 2.12(r318911d) could this be the problem ?? Thank you everyone and sorry if I posted on the wrong place hope not … Ed