Percona cluster node joining issue

Please Help!!!
when i start my second node like this systemctl start mysql its going to hang and doing nothing and not joining the cluster. but on the the first node its running ok and status is sync or ready for cluster…

theses are the logs of joiner node.

2021-04-22T06:53:15.554158Z 0 [Note] WSREP: (699fb144, 'tcp://') multicast: , ttl: 1
2021-04-22T06:53:15.554692Z 0 [Note] WSREP: EVS version 0
2021-04-22T06:53:15.554914Z 0 [Note] WSREP: gcomm: connecting to group 'nbbcluster', peer ','
2021-04-22T06:53:15.557414Z 0 [Note] WSREP: (699fb144, 'tcp://') Found matching local endpoint for a connection, blacklisting address tcp://
2021-04-22T06:53:15.569229Z 0 [Note] WSREP: (699fb144, 'tcp://') connection established to 21aa9db1 tcp://
2021-04-22T06:53:15.569416Z 0 [Note] WSREP: (699fb144, 'tcp://') turning message relay requesting on, nonlive peers:
2021-04-22T06:53:16.060215Z 0 [Note] WSREP: declaring 21aa9db1 at tcp:// stable
2021-04-22T06:53:16.061414Z 0 [Note] WSREP: Node 21aa9db1 state primary
2021-04-22T06:53:16.063729Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,21aa9db1,6)
memb {
joined {
left {
partitioned {
2021-04-22T06:53:16.063790Z 0 [Note] WSREP: Save the discovered primary-component to disk
2021-04-22T06:53:16.557824Z 0 [Note] WSREP: gcomm: connected
2021-04-22T06:53:16.557998Z 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2021-04-22T06:53:16.558173Z 0 [Note] WSREP: Waiting for SST/IST to complete.
2021-04-22T06:53:16.558972Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2021-04-22T06:53:16.559012Z 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2021-04-22T06:53:16.559060Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 69fe3d65-a337-11eb-9220-da773877a821
2021-04-22T06:53:16.559070Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 69fe3d65-a337-11eb-9220-da773877a821 from 0 (centos-01)
2021-04-22T06:53:16.560312Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 69fe3d65-a337-11eb-9220-da773877a821 from 1 (centos-02)
2021-04-22T06:53:16.560356Z 0 [Note] WSREP: Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 0,
        members    = 1/2 (primary/total),
        act_id     = 68,
        last_appl. = -1,
        protocols  = 0/9/3 (gcs/repl/appl),
        group UUID = 6b6a3e9e-a19e-11eb-abba-1a700e6ae9ac
2021-04-22T06:53:16.560370Z 0 [Note] WSREP: Flow-control interval: [141, 141]
2021-04-22T06:53:16.560386Z 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 68)
2021-04-22T06:53:16.560485Z 2 [Note] WSREP: State transfer required:
        Group state: 6b6a3e9e-a19e-11eb-abba-1a700e6ae9ac:68
        Local state: 00000000-0000-0000-0000-000000000000:-1
2021-04-22T06:53:16.560517Z 2 [Note] WSREP: REPL Protocols: 9 (4, 2)
2021-04-22T06:53:16.560528Z 2 [Note] WSREP: REPL Protocols: 9 (4, 2)
2021-04-22T06:53:16.560554Z 2 [Note] WSREP: New cluster view: global state: 6b6a3e9e-a19e-11eb-abba-1a700e6ae9ac:68, view# 1: Primary, number of nodes: 2, my index: 1, protocol version 3
2021-04-22T06:53:16.560562Z 2 [Note] WSREP: Setting wsrep_ready to true
2021-04-22T06:53:16.560570Z 2 [Warning] WSREP: Gap in state sequence. Need state transfer.
2021-04-22T06:53:16.560577Z 2 [Note] WSREP: Setting wsrep_ready to false
2021-04-22T06:53:16.560821Z 0 [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '2159' --mysqld-version '5.7.33-36-57'   '' )
2021-04-22T06:53:16.562536Z 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '2159' --mysqld-version '5.7.33-36-57'   ''
        Read: '(null)'
2021-04-22T06:53:16.562598Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '2159' --mysqld-version '5.7.33-36-57'   '' : 2 (No such file or directory)
2021-04-22T06:53:16.562811Z 2 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
2021-04-22T06:53:16.562828Z 2 [ERROR] Aborting

2021-04-22T06:53:16.562834Z 2 [Note] Giving 2 client threads a chance to die gracefully
2021-04-22T06:53:18.563409Z 2 [Note] WSREP: Closing send monitor...
2021-04-22T06:53:18.563480Z 2 [Note] WSREP: Closed send monitor.
2021-04-22T06:53:18.563510Z 2 [Note] WSREP: gcomm: terminating thread
2021-04-22T06:53:18.563526Z 2 [Note] WSREP: gcomm: joining thread
2021-04-22T06:53:18.563554Z 1 [Note] WSREP: rollbacker thread exiting
2021-04-22T06:53:18.568729Z 2 [Note] WSREP: gcomm: closing backend
2021-04-22T06:53:19.059277Z 2 [Note] WSREP: (699fb144, 'tcp://') turning message relay requesting off
2021-04-22T06:53:19.573283Z 2 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,21aa9db1,6)
memb {
joined {
left {
partitioned {
2021-04-22T06:53:19.573424Z 2 [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
2021-04-22T06:53:19.574008Z 2 [Note] WSREP: gcomm: closed
2021-04-22T06:53:19.574140Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2021-04-22T06:53:19.574205Z 0 [Note] WSREP: Flow-control interval: [100, 100]
2021-04-22T06:53:19.574213Z 0 [Note] WSREP: Received NON-PRIMARY.
2021-04-22T06:53:19.574217Z 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 68)
2021-04-22T06:53:19.574229Z 0 [Note] WSREP: Received self-leave message.
2021-04-22T06:53:19.574235Z 0 [Note] WSREP: Flow-control interval: [0, 0]
2021-04-22T06:53:19.574239Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2021-04-22T06:53:19.574243Z 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 68)
2021-04-22T06:53:19.574252Z 0 [Note] WSREP: RECV thread exiting 0: Success
2021-04-22T06:53:19.574306Z 2 [Note] WSREP: recv_thread() joined.
2021-04-22T06:53:19.574312Z 2 [Note] WSREP: Closing replication queue.
2021-04-22T06:53:19.574316Z 2 [Note] WSREP: Closing slave action queue.
2021-04-22T06:53:19.574488Z 0 [Note] WSREP: Waiting for active wsrep applier to exit
2021-04-22T06:53:19.574502Z 0 [Note] WSREP: Service disconnected.
2021-04-22T06:53:19.574506Z 0 [Note] WSREP: Waiting to close threads......
2021-04-22T06:53:24.575260Z 0 [Note] WSREP: Some threads may fail to exit.
2021-04-22T06:53:24.589227Z 0 [Note] WSREP: Service thread queue flushed.
2021-04-22T06:53:24.589516Z 0 [Note] WSREP: MemPool(SlaveTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2021-04-22T06:53:24.589597Z 0 [Warning] WSREP: Waiting for 2 items to be fetched.
1 Like

Is this PXC 5.7? You posted into the 8.0 forum but the logs indicate 5.7. If it is 5.7, you need to install percona xtrabackup. Then make sure ports 4444 and 4567 are open between the servers.

1 Like

yes it is pxc 5.7 … but i have already installed percona xtrabackup-24 . also its not working…
the problem is that when i start my first node as a bootstrap its working fine but when i start the second node its going to hang and after that status failed…

1 Like

You need to share the logs of the joining node.