Will never receive state. Need to abort

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

use case
Cluster : PXC 8.0.27
No.Of.Nodes :3
OS : CentOS

Hi there,
I had bootstrapped the node1 and it is online,when iam trying to add the node 2 to the cluster below is the error reported

[Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receiv e state. Need to abort.

LogFile of node2



[testuser@testserveruat mysql]$ sudo tail -n 100 /var/log/mysqld.log
2022-06-29T17:56:23.065780+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') Found matching local endpoint for a                           connection, blacklisting address tcp://172.31.2.59:4567
2022-06-29T17:56:23.066651+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') connection established to ded2c73f-a                          972 tcp://172.31.15.228:4567
2022-06-29T17:56:23.066719+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') turning message relay requesting on,                           nonlive peers:
2022-06-29T17:56:23.520837+05:30 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2022-06-29T17:56:23.520898+05:30 0 [Note] [MY-000000] [Galera] declaring ded2c73f-a972 at tcp://172.31.15.228:4567 stable
2022-06-29T17:56:23.520952+05:30 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2022-06-29T17:56:23.521157+05:30 0 [Note] [MY-000000] [Galera] Node ded2c73f-a972 state primary
2022-06-29T17:56:23.521334+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,aff98fa6-a32e,70)
memb {
        aff98fa6-a32e,0
        ded2c73f-a972,0
        }
joined {
        }
left {
        }
partitioned {
        }
)
2022-06-29T17:56:23.521365+05:30 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-29T17:56:23.523251+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://172.31.2.202:4567
2022-06-29T17:56:23.523278+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x6787640
2022-06-29T17:56:23.565683+05:30 0 [Note] [MY-000000] [Galera] gcomm: connected
2022-06-29T17:56:23.565725+05:30 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2022-06-29T17:56:23.565794+05:30 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2022-06-29T17:56:23.565815+05:30 0 [Note] [MY-000000] [Galera] Opened channel 'pxcragabh'
2022-06-29T17:56:23.565928+05:30 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-06-29T17:56:23.566226+05:30 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: b04623c7-f7a6-11ec-bc10-1bfa88f0178b
2022-06-29T17:56:23.566248+05:30 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2022-06-29T17:56:23.566333+05:30 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2022-06-29T17:56:23.566498+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b
2022-06-29T17:56:23.566702+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b from 0                           (pxc1)
2022-06-29T17:56:23.566745+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b from 1                           (pxc2)
2022-06-29T17:56:23.566781+05:30 0 [Note] [MY-000000] [Galera] Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 69,
        members    = 1/2 (primary/total),
        act_id     = 76,
        last_appl. = 7,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 680c03b4-f3b4-11ec-ae26-a71ace89b9e1
2022-06-29T17:56:23.566850+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2022-06-29T17:56:23.566880+05:30 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 77)
2022-06-29T17:56:23.566936+05:30 2 [Note] [MY-000000] [Galera] ####### processing CC 77, local, ordered
2022-06-29T17:56:23.566976+05:30 2 [Note] [MY-000000] [Galera] Maybe drain monitors from -1 upto current CC event 77 upto:-1
2022-06-29T17:56:23.567003+05:30 2 [Note] [MY-000000] [Galera] Drain monitors from -1 up to -1
2022-06-29T17:56:23.567044+05:30 2 [Note] [MY-000000] [Galera] Process first view: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1 my uuid: aff98fa6                          -f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567084+05:30 2 [Note] [MY-000000] [Galera] Server pxc1 connected to cluster at position 680c03b4-f3b4-11ec-ae26-a71ac                          e89b9e1:77 with ID aff98fa6-f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567112+05:30 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2022-06-29T17:56:23.567136+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-29T17:56:23.567183+05:30 2 [Note] [MY-000000] [Galera] ####### My UUID: aff98fa6-f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567219+05:30 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), s                          tate transfer needed: yes
2022-06-29T17:56:23.567289+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-29T17:56:23.567336+05:30 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000                          -000000000000:-1, protocol version: -1
2022-06-29T17:56:23.567365+05:30 2 [Note] [MY-000000] [Galera] State transfer required:
        Group state: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1:77
        Local state: 00000000-0000-0000-0000-000000000000:-1
2022-06-29T17:56:23.567388+05:30 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-06-29T17:56:23.567405+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-29T17:56:23.567511+05:30 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role                           'joiner' --address '172.31.2.59' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/e                          tc/my.cnf' --defaults-group-suffix '' --parent '4364' --mysqld-version '8.0.27-18.1'   '' )
2022-06-29T17:56:23.743985+05:30 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2022-06-29T17:56:24.013527+05:30 2 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|172.31.2.59:4444/xtrabackup_sst//1
2022-06-29T17:56:24.013586+05:30 2 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2022-06-29T17:56:24.013627+05:30 2 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 680c03b4-f                          3b4-11ec-ae26-a71ace89b9e1
2022-06-29T17:56:24.013653+05:30 2 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 77, STRv: 3
2022-06-29T17:56:24.013754+05:30 2 [Note] [MY-000000] [Galera] IST receiver addr using tcp://172.31.2.59:4568
2022-06-29T17:56:24.013959+05:30 2 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-77, listening at: tcp://172.31.2.59:4568
2022-06-29T17:56:24.014467+05:30 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc1) requested state transfer from '*any*'. Selected 1.0 (pxc                          2)(SYNCED) as donor.
2022-06-29T17:56:24.014499+05:30 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 77)
2022-06-29T17:56:24.014533+05:30 2 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1
2022-06-29T17:56:24.014555+05:30 2 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2022-06-29T17:56:24.014579+05:30 2 [Note] [MY-000000] [Galera] GCache history reset: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1:0 -> 680c03b4-f                          3b4-11ec-ae26-a71ace89b9e1:77
2022-06-29T17:56:24.197374+05:30 0 [Warning] [MY-000000] [Galera] 1.0 (pxc2): State transfer to 0.0 (pxc1) failed: -78 (Remote address ch                          anged)
2022-06-29T17:56:24.197864+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receiv                          e state. Need to abort.
2022-06-29T17:56:24.197996+05:30 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-06-29T17:56:24.198024+05:30 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-06-29T17:56:24.198129+05:30 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-06-29T17:56:25.200428+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,aff98fa6-a32e,70)
memb {
        aff98fa6-a32e,0
        }
joined {
        }
left {
        }
partitioned {
        ded2c73f-a972,0
        }
)
2022-06-29T17:56:25.200497+05:30 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-06-29T17:56:25.200511+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-06-29T17:56:25.200667+05:30 0 [Note] [MY-000000] [Galera] gcomm: closed
2022-06-29T17:56:25.200701+05:30 0 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2022-06-29T17:56:25.200724+05:30 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2022-06-29T17:56:25.200752+05:30 0 [Note] [MY-000000] [WSREP] Terminating SST process
[testuser@testserveruat mysql]$
1 Like
2

Hi @AneeshBabu

Is this the same case as Not able to add the node 2 to the cluster - PXC 8.0.27 - #12 by vadimtk ? @vadimtk noted that your gcomm_address line uses IPs in the 192.X range while your error log show you are using IPs in the 172.X range.

If your database nodes have multiple IP addresses then you should set this using wsrep_node_address so that PXC knows on which address to listen/send PXC cluster traffic.

1 Like
3

Hi there

I had bootstrapped the node1 and iam attempting to add the node2 and node3.,below is the issue i reported…

Following are the IP’s

172.31.2.59
172.31.15.228
172.31.2.202

LogFile of node2



[testuser@testserveruat mysql]$ sudo tail -n 100 /var/log/mysqld.log
2022-06-29T17:56:23.065780+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') Found matching local endpoint for a                           connection, blacklisting address tcp://172.31.2.59:4567
2022-06-29T17:56:23.066651+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') connection established to ded2c73f-a                          972 tcp://172.31.15.228:4567
2022-06-29T17:56:23.066719+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') turning message relay requesting on,                           nonlive peers:
2022-06-29T17:56:23.520837+05:30 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2022-06-29T17:56:23.520898+05:30 0 [Note] [MY-000000] [Galera] declaring ded2c73f-a972 at tcp://172.31.15.228:4567 stable
2022-06-29T17:56:23.520952+05:30 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2022-06-29T17:56:23.521157+05:30 0 [Note] [MY-000000] [Galera] Node ded2c73f-a972 state primary
2022-06-29T17:56:23.521334+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,aff98fa6-a32e,70)
memb {
        aff98fa6-a32e,0
        ded2c73f-a972,0
        }
joined {
        }
left {
        }
partitioned {
        }
)
2022-06-29T17:56:23.521365+05:30 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-29T17:56:23.523251+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://172.31.2.202:4567
2022-06-29T17:56:23.523278+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x6787640
2022-06-29T17:56:23.565683+05:30 0 [Note] [MY-000000] [Galera] gcomm: connected
2022-06-29T17:56:23.565725+05:30 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2022-06-29T17:56:23.565794+05:30 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2022-06-29T17:56:23.565815+05:30 0 [Note] [MY-000000] [Galera] Opened channel 'pxcragabh'
2022-06-29T17:56:23.565928+05:30 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-06-29T17:56:23.566226+05:30 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: b04623c7-f7a6-11ec-bc10-1bfa88f0178b
2022-06-29T17:56:23.566248+05:30 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2022-06-29T17:56:23.566333+05:30 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2022-06-29T17:56:23.566498+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b
2022-06-29T17:56:23.566702+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b from 0                           (pxc1)
2022-06-29T17:56:23.566745+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b from 1                           (pxc2)
2022-06-29T17:56:23.566781+05:30 0 [Note] [MY-000000] [Galera] Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 69,
        members    = 1/2 (primary/total),
        act_id     = 76,
        last_appl. = 7,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 680c03b4-f3b4-11ec-ae26-a71ace89b9e1
2022-06-29T17:56:23.566850+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2022-06-29T17:56:23.566880+05:30 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 77)
2022-06-29T17:56:23.566936+05:30 2 [Note] [MY-000000] [Galera] ####### processing CC 77, local, ordered
2022-06-29T17:56:23.566976+05:30 2 [Note] [MY-000000] [Galera] Maybe drain monitors from -1 upto current CC event 77 upto:-1
2022-06-29T17:56:23.567003+05:30 2 [Note] [MY-000000] [Galera] Drain monitors from -1 up to -1
2022-06-29T17:56:23.567044+05:30 2 [Note] [MY-000000] [Galera] Process first view: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1 my uuid: aff98fa6                          -f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567084+05:30 2 [Note] [MY-000000] [Galera] Server pxc1 connected to cluster at position 680c03b4-f3b4-11ec-ae26-a71ac                          e89b9e1:77 with ID aff98fa6-f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567112+05:30 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2022-06-29T17:56:23.567136+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-29T17:56:23.567183+05:30 2 [Note] [MY-000000] [Galera] ####### My UUID: aff98fa6-f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567219+05:30 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), s                          tate transfer needed: yes
2022-06-29T17:56:23.567289+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-29T17:56:23.567336+05:30 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000                          -000000000000:-1, protocol version: -1
2022-06-29T17:56:23.567365+05:30 2 [Note] [MY-000000] [Galera] State transfer required:
        Group state: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1:77
        Local state: 00000000-0000-0000-0000-000000000000:-1
2022-06-29T17:56:23.567388+05:30 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-06-29T17:56:23.567405+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-29T17:56:23.567511+05:30 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role                           'joiner' --address '172.31.2.59' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/e                          tc/my.cnf' --defaults-group-suffix '' --parent '4364' --mysqld-version '8.0.27-18.1'   '' )
2022-06-29T17:56:23.743985+05:30 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2022-06-29T17:56:24.013527+05:30 2 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|172.31.2.59:4444/xtrabackup_sst//1
2022-06-29T17:56:24.013586+05:30 2 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2022-06-29T17:56:24.013627+05:30 2 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 680c03b4-f                          3b4-11ec-ae26-a71ace89b9e1
2022-06-29T17:56:24.013653+05:30 2 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 77, STRv: 3
2022-06-29T17:56:24.013754+05:30 2 [Note] [MY-000000] [Galera] IST receiver addr using tcp://172.31.2.59:4568
2022-06-29T17:56:24.013959+05:30 2 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-77, listening at: tcp://172.31.2.59:4568
2022-06-29T17:56:24.014467+05:30 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc1) requested state transfer from '*any*'. Selected 1.0 (pxc                          2)(SYNCED) as donor.
2022-06-29T17:56:24.014499+05:30 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 77)
2022-06-29T17:56:24.014533+05:30 2 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1
2022-06-29T17:56:24.014555+05:30 2 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2022-06-29T17:56:24.014579+05:30 2 [Note] [MY-000000] [Galera] GCache history reset: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1:0 -> 680c03b4-f                          3b4-11ec-ae26-a71ace89b9e1:77
2022-06-29T17:56:24.197374+05:30 0 [Warning] [MY-000000] [Galera] 1.0 (pxc2): State transfer to 0.0 (pxc1) failed: -78 (Remote address ch                          anged)
2022-06-29T17:56:24.197864+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receiv                          e state. Need to abort.
2022-06-29T17:56:24.197996+05:30 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-06-29T17:56:24.198024+05:30 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-06-29T17:56:24.198129+05:30 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-06-29T17:56:25.200428+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,aff98fa6-a32e,70)
memb {
        aff98fa6-a32e,0
        }
joined {
        }
left {
        }
partitioned {
        ded2c73f-a972,0
        }
)
2022-06-29T17:56:25.200497+05:30 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-06-29T17:56:25.200511+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-06-29T17:56:25.200667+05:30 0 [Note] [MY-000000] [Galera] gcomm: closed
2022-06-29T17:56:25.200701+05:30 0 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2022-06-29T17:56:25.200724+05:30 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2022-06-29T17:56:25.200752+05:30 0 [Note] [MY-000000] [WSREP] Terminating SST process

Log File of Node3 :


[testuser@testserveruat ~]$ sudo tail -n 100 /var/log/mysqld.log
2022-06-30T18:26:22.405745+05:30 0 [Note] [MY-000000] [Galera] Node eec9281c-80b4 state primary
2022-06-30T18:26:22.406315+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,0aa4f7b3-933d,4)
memb {
        0aa4f7b3-933d,0
        eec9281c-80b4,0
        }
joined {
        }
left {
        }
partitioned {
        }
)
2022-06-30T18:26:22.406334+05:30 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-30T18:26:22.408093+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://172.31.2.59:4567
2022-06-30T18:26:22.408112+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7206480
2022-06-30T18:26:22.513389+05:30 0 [Note] [MY-000000] [Galera] gcomm: connected
2022-06-30T18:26:22.513434+05:30 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2022-06-30T18:26:22.513518+05:30 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2022-06-30T18:26:22.513543+05:30 0 [Note] [MY-000000] [Galera] Opened channel 'pxcragabh'
2022-06-30T18:26:22.513702+05:30 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-06-30T18:26:22.513911+05:30 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 0af185d9-f874-11ec-88f2-2fa06bb125a7
2022-06-30T18:26:22.513990+05:30 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2
2022-06-30T18:26:22.514050+05:30 1 [Note] [MY-000000] [WSREP] Starting applier thread 1
2022-06-30T18:26:22.514863+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 0af185d9-f874-11ec-88f2-2fa06bb125a7
2022-06-30T18:26:22.515713+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 0af185d9-f874-11ec-88f2-2fa06bb125a7 from 0 (pxc2)
2022-06-30T18:26:22.516490+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 0af185d9-f874-11ec-88f2-2fa06bb125a7 from 1 (pxc3)
2022-06-30T18:26:22.516506+05:30 0 [Note] [MY-000000] [Galera] Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 3,
        members    = 1/2 (primary/total),
        act_id     = 4,
        last_appl. = 1,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = f22a223b-f86e-11ec-a380-d7db6c7591e4
2022-06-30T18:26:22.516543+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2022-06-30T18:26:22.516556+05:30 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 5)
2022-06-30T18:26:22.516612+05:30 1 [Note] [MY-000000] [Galera] ####### processing CC 5, local, ordered
2022-06-30T18:26:22.516638+05:30 1 [Note] [MY-000000] [Galera] Maybe drain monitors from -1 upto current CC event 5 upto:-1
2022-06-30T18:26:22.516652+05:30 1 [Note] [MY-000000] [Galera] Drain monitors from -1 up to -1
2022-06-30T18:26:22.516684+05:30 1 [Note] [MY-000000] [Galera] Process first view: f22a223b-f86e-11ec-a380-d7db6c7591e4 my uuid: 0aa4f7b3-f874-11ec-933d-6ba8c6077ae3
2022-06-30T18:26:22.516710+05:30 1 [Note] [MY-000000] [Galera] Server pxc2 connected to cluster at position f22a223b-f86e-11ec-a380-d7db6c7591e4:5 with ID 0aa4f7b3-f874-11ec-933d-6ba8c6077ae3
2022-06-30T18:26:22.516726+05:30 1 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2022-06-30T18:26:22.516742+05:30 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-30T18:26:22.516769+05:30 1 [Note] [MY-000000] [Galera] ####### My UUID: 0aa4f7b3-f874-11ec-933d-6ba8c6077ae3
2022-06-30T18:26:22.516788+05:30 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2022-06-30T18:26:22.516838+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-30T18:26:22.516989+05:30 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2022-06-30T18:26:22.517008+05:30 1 [Note] [MY-000000] [Galera] State transfer required:
        Group state: f22a223b-f86e-11ec-a380-d7db6c7591e4:5
        Local state: 00000000-0000-0000-0000-000000000000:-1
2022-06-30T18:26:22.517022+05:30 1 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-06-30T18:26:22.517033+05:30 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-30T18:26:22.517109+05:30 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.15.228' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3685' --mysqld-version '8.0.27-18.1'   '' )
2022-06-30T18:26:22.728758+05:30 0 [Note] [MY-000000] [WSREP-SST] which: no openssl in (/usr/sbin:/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
2022-06-30T18:26:22.729773+05:30 0 [ERROR] [MY-000000] [WSREP] Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.15.228' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3685' --mysqld-version '8.0.27-18.1'   ''
        Read: ''openssl' not found in PATH'
2022-06-30T18:26:22.730097+05:30 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.15.228' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3685' --mysqld-version '8.0.27-18.1'   '' : 2 (No such file or directory)
2022-06-30T18:26:22.730180+05:30 1 [ERROR] [MY-000000] [WSREP] Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
2022-06-30T18:26:22.730295+05:30 1 [ERROR] [MY-000000] [Galera] SST request callback failed. This is unrecoverable, restart required.
2022-06-30T18:26:22.730321+05:30 1 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()
2022-06-30T18:26:22.730340+05:30 1 [Note] [MY-000000] [Galera] Closing send monitor...
2022-06-30T18:26:22.730357+05:30 1 [Note] [MY-000000] [Galera] Closed send monitor.
2022-06-30T18:26:22.730377+05:30 1 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-06-30T18:26:22.730409+05:30 1 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-06-30T18:26:22.730471+05:30 1 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-06-30T18:26:23.737405+05:30 1 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,0aa4f7b3-933d,4)
memb {
        0aa4f7b3-933d,0
        }
joined {
        }
left {
        }
partitioned {
        eec9281c-80b4,0
        }
)
2022-06-30T18:26:23.737503+05:30 1 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-06-30T18:26:23.737533+05:30 1 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-06-30T18:26:23.737741+05:30 1 [Note] [MY-000000] [Galera] gcomm: closed
2022-06-30T18:26:23.737816+05:30 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2022-06-30T18:26:23.737935+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2022-06-30T18:26:23.737975+05:30 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2022-06-30T18:26:23.738012+05:30 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> OPEN (TO: 5)
2022-06-30T18:26:23.738049+05:30 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2022-06-30T18:26:23.738100+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2022-06-30T18:26:23.738135+05:30 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2022-06-30T18:26:23.738165+05:30 0 [Note] [MY-000000] [Galera] Shifting OPEN -> CLOSED (TO: 5)
2022-06-30T18:26:23.738193+05:30 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2022-06-30T18:26:23.738274+05:30 1 [Note] [MY-000000] [Galera] recv_thread() joined.
2022-06-30T18:26:23.738297+05:30 1 [Note] [MY-000000] [Galera] Closing replication queue.
2022-06-30T18:26:23.738315+05:30 1 [Note] [MY-000000] [Galera] Closing slave action queue.
2022-06-30T18:26:23.738336+05:30 1 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2022-06-30T18:26:23.738352+05:30 1 [Note] [MY-000000] [WSREP] Initiating SST cancellation
1 Like
4

Hi there
Can anyone please look into this.

1 Like
5

Both Node 2 and Node 3 have different errors.

Let’s start with Node 3. Did you investigate the error: ‘‘openssl’ not found in path’?
Is openssl installed on Node 3?

By default PXC will encrypt cluster traffic. You can read more about that at this link:
“Redirecting”

Now on to Node 2. In the Log it shows a warning that the remote address changed, then it Errors out on the next line. Have you dug into the warning? Did something happen that the IP of node 1 changed while Node 2 was requesting a state transfer?

What does the wsrep_cluster_address value look like on all 3 nodes?

1 Like
6

1.cert files have been copied from node1 to node2 and node3.

2.below is the wsrep_cluster_address in all the 3nodes.

wsrep_cluster_address=gcomm://172.31.2.59,172.31.15.228,172.31.2.202

1 Like
7

Did you look into the error on Node 2 where it reports the address changed?

Did you verify openssl is on Node 3?

Have you tried disabling encrypted traffic to see if problem continues?

What type of troubleshooting have you done so far?

2 Likes