# Issue in adding nodes with primary Percona XtraDB Cluster node **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [November 12, 2014, 6:27am UTC](https://forums.percona.com/t/issue-in-adding-nodes-with-primary-percona-xtradb-cluster-node/3874 "2014-11-12T06:27:06Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![Sahabuddin](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@Sahabuddin](https://forums.percona.com/u/Sahabuddin)\ **Post date:** [November 12, 2014, 6:28am UTC](https://forums.percona.com/t/issue-in-adding-nodes-with-primary-percona-xtradb-cluster-node/3874/2 "2014-11-12T06:28:56Z")
CONTENT OF /var/lib/mysql/server2.ERR " 2014-11-11 19:38:27 7329 [Note] WSREP: gcomm: closed 2014-11-11 19:38:27 7329 [Note] WSREP: /usr/sbin/mysqld: Terminated. 141111 19:38:27 mysqld\_safe mysqld from pid file /var/lib/mysql/server2.pid ended 141111 19:48:29 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql 141111 19:48:29 mysqld\_safe WSREP: Running position recovery with --log\_error=‘/var/lib/mysql/wsrep\_recovery.5XhKi7’ --pid-file=‘/var/lib/mysql/server2-recover.pid’ 2014-11-11 19:48:30 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit\_defaults\_for\_timestamp server option (see documentation for more details). 141111 19:48:33 mysqld\_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1 2014-11-11 19:48:33 0 [Note] WSREP: wsrep\_start\_position var submitted: ‘00000000-0000-0000-0000-000000000000:-1’ 2014-11-11 19:48:33 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit\_defaults\_for\_timestamp server option (see documentation for more details). 2014-11-11 19:48:33 1970 [Note] WSREP: Read nil XID from storage engines, skipping position init 2014-11-11 19:48:33 1970 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 2014-11-11 19:48:33 1970 [Note] WSREP: wsrep\_load(): Galera 3.7(r7f44a18) by Codership Oy \ loaded successfully. 2014-11-11 19:48:33 1970 [Note] WSREP: CRC-32C: using hardware acceleration. 2014-11-11 19:48:33 1970 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 2014-11-11 19:48:33 1970 [Note] WSREP: Passing config to GCS: base\_host = 10.12.252.62; base\_port = 4567; cert.log\_conflicts = no; debug = no; evs.inactive\_check\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 4; evs.stats\_report\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 2; evs.view\_forget\_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1.0; gcs.fc\_limit = 16; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignore\_quorum = false; pc.ignore\_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait\_prim = true; pc.wait\_prim\_timeout = P30S; pc.we 2014-11-11 19:48:33 1970 [Note] WSREP: Service thread queue flushed. 2014-11-11 19:48:33 1970 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 2014-11-11 19:48:33 1970 [Note] WSREP: wsrep\_sst\_grab() 2014-11-11 19:48:33 1970 [Note] WSREP: Start replication 2014-11-11 19:48:33 1970 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 2014-11-11 19:48:33 1970 [Note] WSREP: protonet asio version 0 2014-11-11 19:48:33 1970 [Note] WSREP: Using CRC-32C for message checksums. 2014-11-11 19:48:33 1970 [Note] WSREP: backend: asio 2014-11-11 19:48:33 1970 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory) 2014-11-11 19:48:33 1970 [Note] WSREP: restore pc from disk failed 2014-11-11 19:48:33 1970 [Note] WSREP: GMCast version 0 2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 2014-11-11 19:48:33 1970 [Note] WSREP: EVS version 0 2014-11-11 19:48:33 1970 [Note] WSREP: PC version 0 2014-11-11 19:48:33 1970 [Note] WSREP: gcomm: connecting to group ‘mothallah’, peer ‘10.12.252.61:,10.12.252.62:’ 2014-11-11 19:48:33 1970 [Warning] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.12.252.62:4567’ points to own listening address, blacklisting 2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.12.252.62:4567’ pointing to uuid c6435b12 is blacklisted, skipping 2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: 2014-11-11 19:48:33 1970 [Note] WSREP: declaring e71df5fe at tcp://10.12.252.61:4567 stable 2014-11-11 19:48:33 1970 [Note] WSREP: Node e71df5fe state prim 2014-11-11 19:48:33 1970 [Note] WSREP: save pc into disk 2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: connected 2014-11-11 19:48:34 1970 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2014-11-11 19:48:34 1970 [Note] WSREP: Shifting CLOSED -\> OPEN (TO: 0) 2014-11-11 19:48:34 1970 [Note] WSREP: Opened channel ‘mothallah’ 2014-11-11 19:48:34 1970 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 0, memb\_num = 2 2014-11-11 19:48:34 1970 [Note] WSREP: Waiting for SST to complete. 2014-11-11 19:48:34 1970 [Note] WSREP: STATE\_EXCHANGE: sent state UUID: c6901f42-6a1e-11e4-9d4c-ef9725a3033d 2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: sent state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d 2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: got state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d from 0 (server2) 2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: got state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d from 1 (server1) 2014-11-11 19:48:34 1970 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf\_id = 5, members = 1/2 (joined/total), act\_id = 0, last\_appl. = -1, protocols = 0/6/3 (gcs/repl/appl), group UUID = 51a09534-697c-11e4-99c0-1e9de9467ddb 2014-11-11 19:48:34 1970 [Note] WSREP: Flow-control interval: [23, 23] 2014-11-11 19:48:34 1970 [Note] WSREP: Shifting OPEN -\> PRIMARY (TO: 0) 2014-11-11 19:48:34 1970 [Note] WSREP: State transfer required: Group state: 51a09534-697c-11e4-99c0-1e9de9467ddb:0 Local state: 00000000-0000-0000-0000-000000000000:-1 2014-11-11 19:48:34 1970 [Note] WSREP: New cluster view: global state: 51a09534-697c-11e4-99c0-1e9de9467ddb:0, view# 6: Primary, number of nodes: 2, my index: 0, protocol version 3 2014-11-11 19:48:34 1970 [Warning] WSREP: Gap in state sequence. Need state transfer. 2014-11-11 19:48:34 1970 [Note] WSREP: Running: 'wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘10.12.252.62’ --auth ‘sstuser:SST\_PWD’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘1970’ ‘’ ’ WSREP\_SST: [INFO] Streaming with tar (20141111 19:48:34.176) WSREP\_SST: [INFO] Using socat as streamer (20141111 19:48:34.177) WSREP\_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20141111 19:48:34.185) 2014-11-11 19:48:34 1970 [Note] WSREP: Prepared SST request: xtrabackup|10.12.252.62:4444/xtrabackup\_sst 2014-11-11 19:48:34 1970 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2014-11-11 19:48:34 1970 [Note] WSREP: REPL Protocols: 6 (3, 2) 2014-11-11 19:48:34 1970 [Note] WSREP: Service thread queue flushed. 2014-11-11 19:48:34 1970 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3 2014-11-11 19:48:34 1970 [Note] WSREP: Service thread queue flushed. 2014-11-11 19:48:34 1970 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (51a09534-697c-11e4-99c0-1e9de9467ddb): 1 (Operation not permitted) at galera/src/replicator\_str.cpprepare\_for\_IST():455. IST will be unavailable. 2014-11-11 19:48:34 1970 [Note] WSREP: Member 0.0 (server2) requested state transfer from ‘_any_’. Selected 1.0 (server1)(SYNCED) as donor. 2014-11-11 19:48:34 1970 [Note] WSREP: Shifting PRIMARY -\> JOINER (TO: 0) 2014-11-11 19:48:34 1970 [Note] WSREP: Requesting state transfer: success, donor: 1 tar: This does not look like a tar archive tar: Exiting with failure status due to previous errors WSREP\_SST: [ERROR] Error while getting data from donor node: exit codes: 0 2 (20141111 19:48:34.646) WSREP\_SST: [ERROR] Cleanup after exit with status:32 (20141111 19:48:34.647) 2014-11-11 19:48:34 1970 [Warning] WSREP: 1.0 (server1): State transfer to 0.0 (server2) failed: -22 (Invalid argument) 2014-11-11 19:48:34 1970 [ERROR] WSREP: gcs/src/gcs\_group.cpp:int gcs\_group\_handle\_join\_msg(gcs\_group\_t\*, const gcs\_recv\_msg\_t\*)():722: Will never receive state. Need to abort. 2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: terminating thread 2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: joining thread WSREP\_SST: [INFO] Removing the sst\_in\_progress file (20141111 19:48:34.649) 2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: closing backend 2014-11-11 19:48:34 1970 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘10.12.252.62’ --auth ‘sstuser:SST\_PWD’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘1970’ ‘’ : 32 (Broken pipe) 2014-11-11 19:48:34 1970 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 2014-11-11 19:48:34 1970 [ERROR] WSREP: SST failed: 32 (Broken pipe) 2014-11-11 19:48:34 1970 [ERROR] Aborting 2014-11-11 19:48:35 1970 [Note] WSREP: gcomm: closed 2014-11-11 19:48:35 1970 [Note] WSREP: /usr/sbin/mysqld: Terminated. 141111 19:48:35 mysqld\_safe mysqld from pid file /var/lib/mysql/server2.pid ended " --- _[View the full topic](https://forums.percona.com/t/issue-in-adding-nodes-with-primary-percona-xtradb-cluster-node/3874)._