Not the answer you need?
Register and ask your own question!

Issue in adding nodes with primary Percona XtraDB Cluster node

SahabuddinSahabuddin EntrantCurrent User Role Beginner
Issue in adding nodes with Primary Percona XtraDB Cluster node
We have the same test environment setup on VMWare and everything is working perfectly.
But, when we tried to implement on the Cisco(UCS C220 M3L) servers having raid hhd of 1TB size, are not working.
Any help on this issue, would be greatly appreciated.

Here are my environment info :

3 nodes Centos6.5 servers with Percona XtraDB Cluster 5.6 installed
Server1 (donar server)- when started mysql, bootstrapped and works fine

Server2 (joiner server) - when started mysql, showing errors as " Starting MySQL (Percona XtraDB Cluster)....State transfer in progress, setting sleep higher
. ERROR! The server quit without updating PID file (/var/lib/mysql/server2.pid).
ERROR! MySQL (Percona XtraDB Cluster) server startup failed!
ERROR! Failed to restart server."
But when I comment the line "wsrep_cluster_address=gcomm://10.11.254.61, 10.11.254.62" of /etc/my.cnf file, mysql starts but cluster status showing "Disconnected"

Server3 (joiner server) - same error as with Server2

Comments

  • SahabuddinSahabuddin Entrant Current User Role Beginner
    CONTENT OF /var/lib/mysql/server2.ERR
    "
    2014-11-11 19:38:27 7329 [Note] WSREP: gcomm: closed
    2014-11-11 19:38:27 7329 [Note] WSREP: /usr/sbin/mysqld: Terminated.
    141111 19:38:27 mysqld_safe mysqld from pid file /var/lib/mysql/server2.pid ended
    141111 19:48:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    141111 19:48:29 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.5XhKi7' --pid-file='/var/lib/mysql/server2-recover.pid'
    2014-11-11 19:48:30 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    141111 19:48:33 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
    2014-11-11 19:48:33 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
    2014-11-11 19:48:33 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    2014-11-11 19:48:33 1970 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2014-11-11 19:48:33 1970 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
    2014-11-11 19:48:33 1970 [Note] WSREP: wsrep_load(): Galera 3.7(r7f44a18) by Codership Oy <[email protected]> loaded successfully.
    2014-11-11 19:48:33 1970 [Note] WSREP: CRC-32C: using hardware acceleration.
    2014-11-11 19:48:33 1970 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
    2014-11-11 19:48:33 1970 [Note] WSREP: Passing config to GCS: base_host = 10.12.252.62; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.we
    2014-11-11 19:48:33 1970 [Note] WSREP: Service thread queue flushed.
    2014-11-11 19:48:33 1970 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
    2014-11-11 19:48:33 1970 [Note] WSREP: wsrep_sst_grab()
    2014-11-11 19:48:33 1970 [Note] WSREP: Start replication
    2014-11-11 19:48:33 1970 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
    2014-11-11 19:48:33 1970 [Note] WSREP: protonet asio version 0
    2014-11-11 19:48:33 1970 [Note] WSREP: Using CRC-32C for message checksums.
    2014-11-11 19:48:33 1970 [Note] WSREP: backend: asio
    2014-11-11 19:48:33 1970 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory)
    2014-11-11 19:48:33 1970 [Note] WSREP: restore pc from disk failed
    2014-11-11 19:48:33 1970 [Note] WSREP: GMCast version 0
    2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2014-11-11 19:48:33 1970 [Note] WSREP: EVS version 0
    2014-11-11 19:48:33 1970 [Note] WSREP: PC version 0
    2014-11-11 19:48:33 1970 [Note] WSREP: gcomm: connecting to group 'mothallah', peer '10.12.252.61:,10.12.252.62:'
    2014-11-11 19:48:33 1970 [Warning] WSREP: (c6435b12, 'tcp://0.0.0.0:4567') address 'tcp://10.12.252.62:4567' points to own listening address, blacklisting
    2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, 'tcp://0.0.0.0:4567') address 'tcp://10.12.252.62:4567' pointing to uuid c6435b12 is blacklisted, skipping
    2014-11-11 19:48:33 1970 [Note] WSREP: (c6435b12, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
    2014-11-11 19:48:33 1970 [Note] WSREP: declaring e71df5fe at tcp://10.12.252.61:4567 stable
    2014-11-11 19:48:33 1970 [Note] WSREP: Node e71df5fe state prim
    2014-11-11 19:48:33 1970 [Note] WSREP: save pc into disk
    2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: connected
    2014-11-11 19:48:34 1970 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
    2014-11-11 19:48:34 1970 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
    2014-11-11 19:48:34 1970 [Note] WSREP: Opened channel 'mothallah'
    2014-11-11 19:48:34 1970 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
    2014-11-11 19:48:34 1970 [Note] WSREP: Waiting for SST to complete.
    2014-11-11 19:48:34 1970 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c6901f42-6a1e-11e4-9d4c-ef9725a3033d
    2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: sent state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d
    2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: got state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d from 0 (server2)
    2014-11-11 19:48:34 1970 [Note] WSREP: STATE EXCHANGE: got state msg: c6901f42-6a1e-11e4-9d4c-ef9725a3033d from 1 (server1)
    2014-11-11 19:48:34 1970 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 5,
    members = 1/2 (joined/total),
    act_id = 0,
    last_appl. = -1,
    protocols = 0/6/3 (gcs/repl/appl),
    group UUID = 51a09534-697c-11e4-99c0-1e9de9467ddb
    2014-11-11 19:48:34 1970 [Note] WSREP: Flow-control interval: [23, 23]
    2014-11-11 19:48:34 1970 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
    2014-11-11 19:48:34 1970 [Note] WSREP: State transfer required:
    Group state: 51a09534-697c-11e4-99c0-1e9de9467ddb:0
    Local state: 00000000-0000-0000-0000-000000000000:-1
    2014-11-11 19:48:34 1970 [Note] WSREP: New cluster view: global state: 51a09534-697c-11e4-99c0-1e9de9467ddb:0, view# 6: Primary, number of nodes: 2, my index: 0, protocol version 3
    2014-11-11 19:48:34 1970 [Warning] WSREP: Gap in state sequence. Need state transfer.
    2014-11-11 19:48:34 1970 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '10.12.252.62' --auth 'sstuser:SST_PWD' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '1970' '' '
    WSREP_SST: [INFO] Streaming with tar (20141111 19:48:34.176)
    WSREP_SST: [INFO] Using socat as streamer (20141111 19:48:34.177)
    WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20141111 19:48:34.185)
    2014-11-11 19:48:34 1970 [Note] WSREP: Prepared SST request: xtrabackup|10.12.252.62:4444/xtrabackup_sst
    2014-11-11 19:48:34 1970 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2014-11-11 19:48:34 1970 [Note] WSREP: REPL Protocols: 6 (3, 2)
    2014-11-11 19:48:34 1970 [Note] WSREP: Service thread queue flushed.
    2014-11-11 19:48:34 1970 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
    2014-11-11 19:48:34 1970 [Note] WSREP: Service thread queue flushed.
    2014-11-11 19:48:34 1970 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (51a09534-697c-11e4-99c0-1e9de9467ddb): 1 (Operation not permitted)
    at galera/src/replicator_str.cpprepare_for_IST():455. IST will be unavailable.
    2014-11-11 19:48:34 1970 [Note] WSREP: Member 0.0 (server2) requested state transfer from '*any*'. Selected 1.0 (server1)(SYNCED) as donor.
    2014-11-11 19:48:34 1970 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
    2014-11-11 19:48:34 1970 [Note] WSREP: Requesting state transfer: success, donor: 1
    tar: This does not look like a tar archive
    tar: Exiting with failure status due to previous errors
    WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 0 2 (20141111 19:48:34.646)
    WSREP_SST: [ERROR] Cleanup after exit with status:32 (20141111 19:48:34.647)
    2014-11-11 19:48:34 1970 [Warning] WSREP: 1.0 (server1): State transfer to 0.0 (server2) failed: -22 (Invalid argument)
    2014-11-11 19:48:34 1970 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():722: Will never receive state. Need to abort.
    2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: terminating thread
    2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: joining thread
    WSREP_SST: [INFO] Removing the sst_in_progress file (20141111 19:48:34.649)
    2014-11-11 19:48:34 1970 [Note] WSREP: gcomm: closing backend
    2014-11-11 19:48:34 1970 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '10.12.252.62' --auth 'sstuser:SST_PWD' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '1970' '' : 32 (Broken pipe)
    2014-11-11 19:48:34 1970 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
    2014-11-11 19:48:34 1970 [ERROR] WSREP: SST failed: 32 (Broken pipe)
    2014-11-11 19:48:34 1970 [ERROR] Aborting

    2014-11-11 19:48:35 1970 [Note] WSREP: gcomm: closed
    2014-11-11 19:48:35 1970 [Note] WSREP: /usr/sbin/mysqld: Terminated.
    141111 19:48:35 mysqld_safe mysqld from pid file /var/lib/mysql/server2.pid ended

    "
  • SahabuddinSahabuddin Entrant Current User Role Beginner
    CONTENT OF /var/log/mysqld.lod
    "
    141111 03:38:29 mysqld_safe mysqld from pid file /var/lib/mysql/server2.pid ended
    141111 03:38:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    141111 03:38:37 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.3Q5bPp' --pid-file='/var/lib/mysql/server2-recover.pid'
    2014-11-11 03:38:37 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    141111 03:38:40 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
    2014-11-11 03:38:40 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
    2014-11-11 03:38:40 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    2014-11-11 03:38:40 3914 [Note] Plugin 'FEDERATED' is disabled.
    2014-11-11 03:38:40 3914 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2014-11-11 03:38:40 3914 [Note] InnoDB: The InnoDB memory heap is disabled
    2014-11-11 03:38:40 3914 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2014-11-11 03:38:40 3914 [Note] InnoDB: Memory barrier is not used
    2014-11-11 03:38:40 3914 [Note] InnoDB: Compressed tables use zlib 1.2.3
    2014-11-11 03:38:40 3914 [Note] InnoDB: Using Linux native AIO
    2014-11-11 03:38:40 3914 [Note] InnoDB: Using CPU crc32 instructions
    2014-11-11 03:38:40 3914 [Note] InnoDB: Initializing buffer pool, size = 128.0M
    2014-11-11 03:38:40 3914 [Note] InnoDB: Completed initialization of buffer pool
    2014-11-11 03:38:40 3914 [Note] InnoDB: Highest supported file format is Barracuda.
    2014-11-11 03:38:40 3914 [Note] InnoDB: 128 rollback segment(s) are active.
    2014-11-11 03:38:40 3914 [Note] InnoDB: Waiting for purge to start
    2014-11-11 03:38:40 3914 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 1626241
    2014-11-11 03:38:40 3914 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
    2014-11-11 03:38:40 3914 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
    2014-11-11 03:38:40 3914 [Note] Server hostname (bind-address): '*'; port: 3306
    2014-11-11 03:38:40 3914 [Note] IPv6 is available.
    2014-11-11 03:38:40 3914 [Note] - '::' resolves to '::';
    2014-11-11 03:38:40 3914 [Note] Server socket created on IP: '::'.
    2014-11-11 03:38:40 3914 [Note] Event Scheduler: Loaded 0 events
    2014-11-11 03:38:40 3914 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2014-11-11 03:38:40 3914 [Note] WSREP: wsrep_load(): loading provider library 'none'
    2014-11-11 03:38:40 3914 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.6.20-68.0-56' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel68.0, Revision 888, WSREP version 25.7, wsrep_25.7.r4126
    2014-11-11 03:38:45 3914 [Note] /usr/sbin/mysqld: Normal shutdown

    141111 03:38:46 mysqld_safe mysqld from pid file /var/lib/mysql/server2.pid ended


    "
  • wat0075wat0075 Entrant Current User Role Beginner
    Are you sure you don't have a firewall running that is blocking connections to 10.11.254.61 port 4567 you need the following ports open
  • SahabuddinSahabuddin Entrant Current User Role Beginner
    Firewall on all servers are stop and Selinux=disabled
    As mentioned above, 3 nodes with same configuration on the test environment is already running on the same network perfectly.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.