Issue in adding nodes with primary Percona XtraDB Cluster node

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Issue in adding nodes with Primary Percona XtraDB Cluster node
We have the same test environment setup on VMWare and everything is working perfectly.
But, when we tried to implement on the Cisco(UCS C220 M3L) servers having raid hhd of 1TB size, are not working.
Any help on this issue, would be greatly appreciated.

Here are my environment info :

3 nodes Centos6.5 servers with Percona XtraDB Cluster 5.6 installed
Server1 (donar server)- when started mysql, bootstrapped and works fine

Server2 (joiner server) - when started mysql, showing errors as " Starting MySQL (Percona XtraDB Cluster)…State transfer in progress, setting sleep higher
. ERROR! The server quit without updating PID file (/var/lib/mysql/server2.pid).
ERROR! MySQL (Percona XtraDB Cluster) server startup failed!
ERROR! Failed to restart server."
But when I comment the line “wsrep_cluster_address=gcomm://10.11.254.61, 10.11.254.62” of /etc/my.cnf file, mysql starts but cluster status showing “Disconnected”

Server3 (joiner server) - same error as with Server2

2

CONTENT OF /var/lib/mysql/server2.ERR
"
2014-11-11 19:38:27 7329 [Note] WSREP: gcomm: closed
2014-11-11 19:38:27 7329 [Note] WSREP: /usr/sbin/mysqld: Terminated.
141111 19:38:27 mysqld_safe mysqld from pid file /var/lib/mysql/server2.pid ended
141111 19:48:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141111 19:48:29 mysqld_safe WSREP: Running position recovery with --log_error=‘/var/lib/mysql/wsrep_recovery.5XhKi7’ --pid-file=‘/var/lib/mysql/server2-recover.pid’
2014-11-11 19:48:30 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
141111 19:48:33 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
2014-11-11 19:48:33 0 [Note] WSREP: wsrep_start_position var submitted: ‘00000000-0000-0000-0000-000000000000:-1’
2014-11-11 19:48:33 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-11-11 19:48:33 1970 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-11-11 19:48:33 1970 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’
2014-11-11 19:48:33 1970 [Note] WSREP: wsrep_load(): Galera 3.7(r7f44a18) by Codership Oy <info@codership.com> loaded successfully.
2014-11-11 19:48:33 1970 [Note] WSREP: CRC-32C: using hardware acceleration.
2014-11-11 19:48:33 1970 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2014-11-11 19:48:33 1970 [Note] WSREP: Passing config to GCS: base_host = 10.12.252.62; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.we
2014-11-11 19:48:33 1970 [Note] WSREP: Service thread queue flushed.
2014-11-11 19:48:33 1970 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2014-11-11 19:48:33 1970 [Note] WSREP: wsrep_sst_grab()
2014-11-11 19:48:33 1970 [Note] WSREP: Start replication
2014-11-11 19:48:33 1970 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2014-11-11 19:48:33 1970 [Note] WSREP: protonet asio version 0
2014-11-11 19:48:33 1970 [Note] WSREP: Using CRC-32C for message checksums.
2014-11-11 19:48:33 1970 [Note] WSREP: backend: asio
2014-11-11 19:48:33 1970 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory)
2014-11-11 19:48:33 1970 [Note] WSREP: restore pc from disk failed
2014-11-11 19:48:33 1970 [Note] WSREP: GMCast version 0
2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
2014-11-11 19:48:33 1970 [Note] WSREP: EVS version 0
2014-11-11 19:48:33 1970 [Note] WSREP: PC version 0
2014-11-11 19:48:33 1970 [Note] WSREP: gcomm: connecting to group ‘mothallah’, peer ‘10.12.252.61:,10.12.252.62:’
2014-11-11 19:48:33 1970 [Warning] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.12.252.62:4567’ points to own listening address, blacklisting
2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.12.252.62:4567’ pointing to uuid c6435b12 is blacklisted, skipping
2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers:
2014-11-11 19:48:33 1970 [Note] WSREP: declaring e71df5fe at tcp://10.12.252.61:4567 stable
2014-11-11 19:48:33 1970 [Note] WSREP: Node e71df5fe state prim
2014-11-11 19:48:33 1970 [Note] WSREP: save pc into disk
2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: connected
2014-11-11 19:48:34 1970 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2014-11-11 19:48:34 1970 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2014-11-11 19:48:34 1970 [Note] WSREP: Opened channel ‘mothallah’
2014-11-11 19:48:34 1970 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2014-11-11 19:48:34 1970 [Note] WSREP: Waiting for SST to complete.
2014-11-11 19:48:34 1970 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c6901f42-6a1e-11e4-9d4c-ef9725a3033d
2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: sent state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d
2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: got state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d from 0 (server2)
2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: got state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d from 1 (server1)
2014-11-11 19:48:34 1970 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 5,
members = 1/2 (joined/total),
act_id = 0,
last_appl. = -1,
protocols = 0/6/3 (gcs/repl/appl),
group UUID = 51a09534-697c-11e4-99c0-1e9de9467ddb
2014-11-11 19:48:34 1970 [Note] WSREP: Flow-control interval: [23, 23]
2014-11-11 19:48:34 1970 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
2014-11-11 19:48:34 1970 [Note] WSREP: State transfer required:
Group state: 51a09534-697c-11e4-99c0-1e9de9467ddb:0
Local state: 00000000-0000-0000-0000-000000000000:-1
2014-11-11 19:48:34 1970 [Note] WSREP: New cluster view: global state: 51a09534-697c-11e4-99c0-1e9de9467ddb:0, view# 6: Primary, number of nodes: 2, my index: 0, protocol version 3
2014-11-11 19:48:34 1970 [Warning] WSREP: Gap in state sequence. Need state transfer.
2014-11-11 19:48:34 1970 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role ‘joiner’ --address ‘10.12.252.62’ --auth ‘sstuser:SST_PWD’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘1970’ ‘’ ’
WSREP_SST: [INFO] Streaming with tar (20141111 19:48:34.176)
WSREP_SST: [INFO] Using socat as streamer (20141111 19:48:34.177)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20141111 19:48:34.185)
2014-11-11 19:48:34 1970 [Note] WSREP: Prepared SST request: xtrabackup|10.12.252.62:4444/xtrabackup_sst
2014-11-11 19:48:34 1970 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-11-11 19:48:34 1970 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-11-11 19:48:34 1970 [Note] WSREP: Service thread queue flushed.
2014-11-11 19:48:34 1970 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2014-11-11 19:48:34 1970 [Note] WSREP: Service thread queue flushed.
2014-11-11 19:48:34 1970 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (51a09534-697c-11e4-99c0-1e9de9467ddb): 1 (Operation not permitted)
at galera/src/replicator_str.cpprepare_for_IST():455. IST will be unavailable.
2014-11-11 19:48:34 1970 [Note] WSREP: Member 0.0 (server2) requested state transfer from ‘any’. Selected 1.0 (server1)(SYNCED) as donor.
2014-11-11 19:48:34 1970 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
2014-11-11 19:48:34 1970 [Note] WSREP: Requesting state transfer: success, donor: 1
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 0 2 (20141111 19:48:34.646)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20141111 19:48:34.647)
2014-11-11 19:48:34 1970 [Warning] WSREP: 1.0 (server1): State transfer to 0.0 (server2) failed: -22 (Invalid argument)
2014-11-11 19:48:34 1970 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():722: Will never receive state. Need to abort.
2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: terminating thread
2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: joining thread
WSREP_SST: [INFO] Removing the sst_in_progress file (20141111 19:48:34.649)
2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: closing backend
2014-11-11 19:48:34 1970 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role ‘joiner’ --address ‘10.12.252.62’ --auth ‘sstuser:SST_PWD’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘1970’ ‘’ : 32 (Broken pipe)
2014-11-11 19:48:34 1970 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-11-11 19:48:34 1970 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2014-11-11 19:48:34 1970 [ERROR] Aborting

2014-11-11 19:48:35 1970 [Note] WSREP: gcomm: closed
2014-11-11 19:48:35 1970 [Note] WSREP: /usr/sbin/mysqld: Terminated.
141111 19:48:35 mysqld_safe mysqld from pid file /var/lib/mysql/server2.pid ended

"

3

CONTENT OF /var/log/mysqld.lod
"
141111 03:38:29 mysqld_safe mysqld from pid file /var/lib/mysql/server2.pid ended
141111 03:38:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141111 03:38:37 mysqld_safe WSREP: Running position recovery with --log_error=‘/var/lib/mysql/wsrep_recovery.3Q5bPp’ --pid-file=‘/var/lib/mysql/server2-recover.pid’
2014-11-11 03:38:37 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
141111 03:38:40 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
2014-11-11 03:38:40 0 [Note] WSREP: wsrep_start_position var submitted: ‘00000000-0000-0000-0000-000000000000:-1’
2014-11-11 03:38:40 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-11-11 03:38:40 3914 [Note] Plugin ‘FEDERATED’ is disabled.
2014-11-11 03:38:40 3914 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-11-11 03:38:40 3914 [Note] InnoDB: The InnoDB memory heap is disabled
2014-11-11 03:38:40 3914 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-11-11 03:38:40 3914 [Note] InnoDB: Memory barrier is not used
2014-11-11 03:38:40 3914 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-11-11 03:38:40 3914 [Note] InnoDB: Using Linux native AIO
2014-11-11 03:38:40 3914 [Note] InnoDB: Using CPU crc32 instructions
2014-11-11 03:38:40 3914 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-11-11 03:38:40 3914 [Note] InnoDB: Completed initialization of buffer pool
2014-11-11 03:38:40 3914 [Note] InnoDB: Highest supported file format is Barracuda.
2014-11-11 03:38:40 3914 [Note] InnoDB: 128 rollback segment(s) are active.
2014-11-11 03:38:40 3914 [Note] InnoDB: Waiting for purge to start
2014-11-11 03:38:40 3914 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 1626241
2014-11-11 03:38:40 3914 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2014-11-11 03:38:40 3914 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2014-11-11 03:38:40 3914 [Note] Server hostname (bind-address): ‘*’; port: 3306
2014-11-11 03:38:40 3914 [Note] IPv6 is available.
2014-11-11 03:38:40 3914 [Note] - ‘::’ resolves to ‘::’;
2014-11-11 03:38:40 3914 [Note] Server socket created on IP: ‘::’.
2014-11-11 03:38:40 3914 [Note] Event Scheduler: Loaded 0 events
2014-11-11 03:38:40 3914 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-11-11 03:38:40 3914 [Note] WSREP: wsrep_load(): loading provider library ‘none’
2014-11-11 03:38:40 3914 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘5.6.20-68.0-56’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel68.0, Revision 888, WSREP version 25.7, wsrep_25.7.r4126
2014-11-11 03:38:45 3914 [Note] /usr/sbin/mysqld: Normal shutdown

141111 03:38:46 mysqld_safe mysqld from pid file /var/lib/mysql/server2.pid ended

"

4

Are you sure you don’t have a firewall running that is blocking connections to 10.11.254.61 port 4567 you need the following ports open

5

Firewall on all servers are stop and Selinux=disabled
As mentioned above, 3 nodes with same configuration on the test environment is already running on the same network perfectly.