While trying to join/start new node according to all relevant documentation i encountered a problem, attached is the log output.
Both machine are running on AWS environment with ubuntu 14.04 in VPC with the correct security group rules.
Percona version installed is: percona-xtradb-cluster-56
Log:
016-02-24 10:18:57 27567 [Note] WSREP: Flow-control interval: [23, 23]
2016-02-24 10:18:57 27567 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 2)
2016-02-24 10:18:57 27567 [Note] WSREP: State transfer required:
Group state: “remove id”
Local state: 00000000-0000-0000-0000-000000000000:-1
2016-02-24 10:18:57 27567 [Note] WSREP: New cluster view: global state: :2, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
2016-02-24 10:18:57 27567 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-02-24 10:18:57 27567 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘27567’ ‘’ ’
WSREP_SST: [INFO] Streaming with xbstream (20160224 )
WSREP_SST: [INFO] Using socat as streamer (20160224)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160224 10:18:57.395)
2016-02-24 10:18:57 27567 [Note] WSREP: Prepared SST request: xtrabackup-v2|$IP:4444/xtrabackup_sst//1
2016-02-24 10:18:57 27567 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-02-24 10:18:57 27567 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-02-24 10:18:57 27567 [Note] WSREP: Service thread queue flushed.
2016-02-24 10:18:57 27567 [Note] WSREP: Assign initial position for certification: 2, protocol version: 3
2016-02-24 10:18:57 27567 [Note] WSREP: Service thread queue flushed.
2016-02-24 10:18:57 27567 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID ($UUID): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():489. IST will be unavailable.
2016-02-24 10:18:57 27567 [Note] WSREP: Member 0.0 () requested state transfer from ‘any’. Selected 1.0 ()(SYNCED) as donor.
2016-02-24 10:18:57 27567 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 2)
2016-02-24 10:18:57 27567 [Note] WSREP: Requesting state transfer: success, donor: 1
2016-02-24 10:18:57 27567 [Warning] WSREP: 1.0 (): State transfer to 0.0 () failed: -12 (Cannot allocate memory)
2016-02-24 10:18:57 27567 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():733: Will never receive state. Need to abort.
2016-02-24 10:18:57 27567 [Note] WSREP: gcomm: terminating thread
2016-02-24 10:18:57 27567 [Note] WSREP: gcomm: joining thread
2016-02-24 10:18:57 27567 [Note] WSREP: gcomm: closing backend
2016-02-24 10:18:58 27567 [Note] WSREP: view(view_id(NON_PRIM,0300a1c4,2) memb {
0300a1c4,0
} joined {
} left {
} partitioned {
e31df3b0,0
})
2016-02-24 10:18:58 27567 [Note] WSREP: view((empty))
2016-02-24 10:18:58 27567 [Note] WSREP: gcomm: closed
2016-02-24 10:18:58 27567 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Aborted (core dumped)
160224 10:19:21 mysqld_safe mysqld from pid file /var/lib/mysql/mysqldb-test-08.pid ended
WSREP_SST: [ERROR] Possible timeout in receving first data from donor in gtid stage (20160224)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160224 1)
Any suggestions ?