Hi there
-
It is not same as = > Not able to add the node 2 to the cluster - PXC 8.0.27 - #12 by vadimtk
-
I started the setup of PXC 8.0.27 right from the beginning (a fresh one).
I had bootstrapped the node1 and iam attempting to add the node2 and node3.,below is the issue i reported…
Following are the IP’s
172.31.2.59
172.31.15.228
172.31.2.202
LogFile of node2
[testuser@testserveruat mysql]$ sudo tail -n 100 /var/log/mysqld.log
2022-06-29T17:56:23.065780+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://172.31.2.59:4567
2022-06-29T17:56:23.066651+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') connection established to ded2c73f-a 972 tcp://172.31.15.228:4567
2022-06-29T17:56:23.066719+05:30 0 [Note] [MY-000000] [Galera] (aff98fa6-a32e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2022-06-29T17:56:23.520837+05:30 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2022-06-29T17:56:23.520898+05:30 0 [Note] [MY-000000] [Galera] declaring ded2c73f-a972 at tcp://172.31.15.228:4567 stable
2022-06-29T17:56:23.520952+05:30 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2022-06-29T17:56:23.521157+05:30 0 [Note] [MY-000000] [Galera] Node ded2c73f-a972 state primary
2022-06-29T17:56:23.521334+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,aff98fa6-a32e,70)
memb {
aff98fa6-a32e,0
ded2c73f-a972,0
}
joined {
}
left {
}
partitioned {
}
)
2022-06-29T17:56:23.521365+05:30 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-29T17:56:23.523251+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://172.31.2.202:4567
2022-06-29T17:56:23.523278+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x6787640
2022-06-29T17:56:23.565683+05:30 0 [Note] [MY-000000] [Galera] gcomm: connected
2022-06-29T17:56:23.565725+05:30 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2022-06-29T17:56:23.565794+05:30 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2022-06-29T17:56:23.565815+05:30 0 [Note] [MY-000000] [Galera] Opened channel 'pxcragabh'
2022-06-29T17:56:23.565928+05:30 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-06-29T17:56:23.566226+05:30 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: b04623c7-f7a6-11ec-bc10-1bfa88f0178b
2022-06-29T17:56:23.566248+05:30 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2022-06-29T17:56:23.566333+05:30 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2022-06-29T17:56:23.566498+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b
2022-06-29T17:56:23.566702+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b from 0 (pxc1)
2022-06-29T17:56:23.566745+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: b04623c7-f7a6-11ec-bc10-1bfa88f0178b from 1 (pxc2)
2022-06-29T17:56:23.566781+05:30 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 69,
members = 1/2 (primary/total),
act_id = 76,
last_appl. = 7,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 680c03b4-f3b4-11ec-ae26-a71ace89b9e1
2022-06-29T17:56:23.566850+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2022-06-29T17:56:23.566880+05:30 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 77)
2022-06-29T17:56:23.566936+05:30 2 [Note] [MY-000000] [Galera] ####### processing CC 77, local, ordered
2022-06-29T17:56:23.566976+05:30 2 [Note] [MY-000000] [Galera] Maybe drain monitors from -1 upto current CC event 77 upto:-1
2022-06-29T17:56:23.567003+05:30 2 [Note] [MY-000000] [Galera] Drain monitors from -1 up to -1
2022-06-29T17:56:23.567044+05:30 2 [Note] [MY-000000] [Galera] Process first view: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1 my uuid: aff98fa6 -f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567084+05:30 2 [Note] [MY-000000] [Galera] Server pxc1 connected to cluster at position 680c03b4-f3b4-11ec-ae26-a71ac e89b9e1:77 with ID aff98fa6-f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567112+05:30 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2022-06-29T17:56:23.567136+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-29T17:56:23.567183+05:30 2 [Note] [MY-000000] [Galera] ####### My UUID: aff98fa6-f7a6-11ec-a32e-ae5404ca2152
2022-06-29T17:56:23.567219+05:30 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), s tate transfer needed: yes
2022-06-29T17:56:23.567289+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-29T17:56:23.567336+05:30 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000 -000000000000:-1, protocol version: -1
2022-06-29T17:56:23.567365+05:30 2 [Note] [MY-000000] [Galera] State transfer required:
Group state: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1:77
Local state: 00000000-0000-0000-0000-000000000000:-1
2022-06-29T17:56:23.567388+05:30 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-06-29T17:56:23.567405+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-29T17:56:23.567511+05:30 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.2.59' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/e tc/my.cnf' --defaults-group-suffix '' --parent '4364' --mysqld-version '8.0.27-18.1' '' )
2022-06-29T17:56:23.743985+05:30 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2022-06-29T17:56:24.013527+05:30 2 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|172.31.2.59:4444/xtrabackup_sst//1
2022-06-29T17:56:24.013586+05:30 2 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2022-06-29T17:56:24.013627+05:30 2 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 680c03b4-f 3b4-11ec-ae26-a71ace89b9e1
2022-06-29T17:56:24.013653+05:30 2 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 77, STRv: 3
2022-06-29T17:56:24.013754+05:30 2 [Note] [MY-000000] [Galera] IST receiver addr using tcp://172.31.2.59:4568
2022-06-29T17:56:24.013959+05:30 2 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-77, listening at: tcp://172.31.2.59:4568
2022-06-29T17:56:24.014467+05:30 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc1) requested state transfer from '*any*'. Selected 1.0 (pxc 2)(SYNCED) as donor.
2022-06-29T17:56:24.014499+05:30 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 77)
2022-06-29T17:56:24.014533+05:30 2 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1
2022-06-29T17:56:24.014555+05:30 2 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2022-06-29T17:56:24.014579+05:30 2 [Note] [MY-000000] [Galera] GCache history reset: 680c03b4-f3b4-11ec-ae26-a71ace89b9e1:0 -> 680c03b4-f 3b4-11ec-ae26-a71ace89b9e1:77
2022-06-29T17:56:24.197374+05:30 0 [Warning] [MY-000000] [Galera] 1.0 (pxc2): State transfer to 0.0 (pxc1) failed: -78 (Remote address ch anged)
2022-06-29T17:56:24.197864+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receiv e state. Need to abort.
2022-06-29T17:56:24.197996+05:30 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-06-29T17:56:24.198024+05:30 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-06-29T17:56:24.198129+05:30 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-06-29T17:56:25.200428+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,aff98fa6-a32e,70)
memb {
aff98fa6-a32e,0
}
joined {
}
left {
}
partitioned {
ded2c73f-a972,0
}
)
2022-06-29T17:56:25.200497+05:30 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-06-29T17:56:25.200511+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-06-29T17:56:25.200667+05:30 0 [Note] [MY-000000] [Galera] gcomm: closed
2022-06-29T17:56:25.200701+05:30 0 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2022-06-29T17:56:25.200724+05:30 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2022-06-29T17:56:25.200752+05:30 0 [Note] [MY-000000] [WSREP] Terminating SST process
Log File of Node3 :
[testuser@testserveruat ~]$ sudo tail -n 100 /var/log/mysqld.log
2022-06-30T18:26:22.405745+05:30 0 [Note] [MY-000000] [Galera] Node eec9281c-80b4 state primary
2022-06-30T18:26:22.406315+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,0aa4f7b3-933d,4)
memb {
0aa4f7b3-933d,0
eec9281c-80b4,0
}
joined {
}
left {
}
partitioned {
}
)
2022-06-30T18:26:22.406334+05:30 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-30T18:26:22.408093+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://172.31.2.59:4567
2022-06-30T18:26:22.408112+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7206480
2022-06-30T18:26:22.513389+05:30 0 [Note] [MY-000000] [Galera] gcomm: connected
2022-06-30T18:26:22.513434+05:30 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2022-06-30T18:26:22.513518+05:30 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2022-06-30T18:26:22.513543+05:30 0 [Note] [MY-000000] [Galera] Opened channel 'pxcragabh'
2022-06-30T18:26:22.513702+05:30 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-06-30T18:26:22.513911+05:30 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 0af185d9-f874-11ec-88f2-2fa06bb125a7
2022-06-30T18:26:22.513990+05:30 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2
2022-06-30T18:26:22.514050+05:30 1 [Note] [MY-000000] [WSREP] Starting applier thread 1
2022-06-30T18:26:22.514863+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 0af185d9-f874-11ec-88f2-2fa06bb125a7
2022-06-30T18:26:22.515713+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 0af185d9-f874-11ec-88f2-2fa06bb125a7 from 0 (pxc2)
2022-06-30T18:26:22.516490+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 0af185d9-f874-11ec-88f2-2fa06bb125a7 from 1 (pxc3)
2022-06-30T18:26:22.516506+05:30 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 3,
members = 1/2 (primary/total),
act_id = 4,
last_appl. = 1,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = f22a223b-f86e-11ec-a380-d7db6c7591e4
2022-06-30T18:26:22.516543+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2022-06-30T18:26:22.516556+05:30 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 5)
2022-06-30T18:26:22.516612+05:30 1 [Note] [MY-000000] [Galera] ####### processing CC 5, local, ordered
2022-06-30T18:26:22.516638+05:30 1 [Note] [MY-000000] [Galera] Maybe drain monitors from -1 upto current CC event 5 upto:-1
2022-06-30T18:26:22.516652+05:30 1 [Note] [MY-000000] [Galera] Drain monitors from -1 up to -1
2022-06-30T18:26:22.516684+05:30 1 [Note] [MY-000000] [Galera] Process first view: f22a223b-f86e-11ec-a380-d7db6c7591e4 my uuid: 0aa4f7b3-f874-11ec-933d-6ba8c6077ae3
2022-06-30T18:26:22.516710+05:30 1 [Note] [MY-000000] [Galera] Server pxc2 connected to cluster at position f22a223b-f86e-11ec-a380-d7db6c7591e4:5 with ID 0aa4f7b3-f874-11ec-933d-6ba8c6077ae3
2022-06-30T18:26:22.516726+05:30 1 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2022-06-30T18:26:22.516742+05:30 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-30T18:26:22.516769+05:30 1 [Note] [MY-000000] [Galera] ####### My UUID: 0aa4f7b3-f874-11ec-933d-6ba8c6077ae3
2022-06-30T18:26:22.516788+05:30 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2022-06-30T18:26:22.516838+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-30T18:26:22.516989+05:30 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2022-06-30T18:26:22.517008+05:30 1 [Note] [MY-000000] [Galera] State transfer required:
Group state: f22a223b-f86e-11ec-a380-d7db6c7591e4:5
Local state: 00000000-0000-0000-0000-000000000000:-1
2022-06-30T18:26:22.517022+05:30 1 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-06-30T18:26:22.517033+05:30 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-30T18:26:22.517109+05:30 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.15.228' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3685' --mysqld-version '8.0.27-18.1' '' )
2022-06-30T18:26:22.728758+05:30 0 [Note] [MY-000000] [WSREP-SST] which: no openssl in (/usr/sbin:/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
2022-06-30T18:26:22.729773+05:30 0 [ERROR] [MY-000000] [WSREP] Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.15.228' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3685' --mysqld-version '8.0.27-18.1' ''
Read: ''openssl' not found in PATH'
2022-06-30T18:26:22.730097+05:30 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.15.228' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3685' --mysqld-version '8.0.27-18.1' '' : 2 (No such file or directory)
2022-06-30T18:26:22.730180+05:30 1 [ERROR] [MY-000000] [WSREP] Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
2022-06-30T18:26:22.730295+05:30 1 [ERROR] [MY-000000] [Galera] SST request callback failed. This is unrecoverable, restart required.
2022-06-30T18:26:22.730321+05:30 1 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()
2022-06-30T18:26:22.730340+05:30 1 [Note] [MY-000000] [Galera] Closing send monitor...
2022-06-30T18:26:22.730357+05:30 1 [Note] [MY-000000] [Galera] Closed send monitor.
2022-06-30T18:26:22.730377+05:30 1 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-06-30T18:26:22.730409+05:30 1 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-06-30T18:26:22.730471+05:30 1 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-06-30T18:26:23.737405+05:30 1 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,0aa4f7b3-933d,4)
memb {
0aa4f7b3-933d,0
}
joined {
}
left {
}
partitioned {
eec9281c-80b4,0
}
)
2022-06-30T18:26:23.737503+05:30 1 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-06-30T18:26:23.737533+05:30 1 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-06-30T18:26:23.737741+05:30 1 [Note] [MY-000000] [Galera] gcomm: closed
2022-06-30T18:26:23.737816+05:30 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2022-06-30T18:26:23.737935+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2022-06-30T18:26:23.737975+05:30 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2022-06-30T18:26:23.738012+05:30 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> OPEN (TO: 5)
2022-06-30T18:26:23.738049+05:30 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2022-06-30T18:26:23.738100+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2022-06-30T18:26:23.738135+05:30 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2022-06-30T18:26:23.738165+05:30 0 [Note] [MY-000000] [Galera] Shifting OPEN -> CLOSED (TO: 5)
2022-06-30T18:26:23.738193+05:30 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2022-06-30T18:26:23.738274+05:30 1 [Note] [MY-000000] [Galera] recv_thread() joined.
2022-06-30T18:26:23.738297+05:30 1 [Note] [MY-000000] [Galera] Closing replication queue.
2022-06-30T18:26:23.738315+05:30 1 [Note] [MY-000000] [Galera] Closing slave action queue.
2022-06-30T18:26:23.738336+05:30 1 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2022-06-30T18:26:23.738352+05:30 1 [Note] [MY-000000] [WSREP] Initiating SST cancellation