I’ve successfully bootstrapped the first node and it is active. The firewall allows all traffic on all ports between the two servers. I’ve copied the ca.pem, server-cert.pem, server-key.pem files from the /var/lib/mysql directory on node1 into node 2 at /etc/mysql/certs and have made the mysql service the owner of the copied certs for node 2.
Here’s the log output I receive:
2021-12-04T23:12:01.413230Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-12-04T23:12:01.413394Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: c160989b-5552-11ec-9f38-dad52a103d84:1, protocol version: -1
2021-12-04T23:12:01.413509Z 0 [Note] [MY-000000] [WSREP] Starting replication
2021-12-04T23:12:01.413601Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0
2021-12-04T23:12:01.413697Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to c160989b-5552-11ec-9f38-dad52a103d84:1
2021-12-04T23:12:01.413848Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2021-12-04T23:12:01.414058Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2021-12-04T23:12:01.414175Z 0 [Note] [MY-000000] [Galera] initializing ssl context
2021-12-04T23:12:01.414473Z 0 [Note] [MY-000000] [Galera] backend: asio
2021-12-04T23:12:01.414648Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2021-12-04T23:12:01.414823Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2021-12-04T23:12:01.414946Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2021-12-04T23:12:01.415219Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2021-12-04T23:12:01.415492Z 0 [Note] [MY-000000] [Galera] (96528c37-9f8d, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567
2021-12-04T23:12:01.415615Z 0 [Note] [MY-000000] [Galera] (96528c37-9f8d, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1
2021-12-04T23:12:01.416008Z 0 [Note] [MY-000000] [Galera] EVS version 1
2021-12-04T23:12:01.416208Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group ‘pxc-cluster’, peer ‘172.31.45.49:,172.31.33.27:,172.31.39.125:’
2021-12-04T23:12:01.425048Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://172.31.33.27:4567 local endpoint ssl://172.31.33.27:47908 cipher: TLS_AES_256_GCM_SHA384 compression: none
2021-12-04T23:12:01.425629Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://172.31.33.27:47908 local endpoint ssl://172.31.33.27:4567 cipher: TLS_AES_256_GCM_SHA384 compression: none
2021-12-04T23:12:01.427306Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://172.31.45.49:4567 local endpoint ssl://172.31.33.27:52554 cipher: TLS_AES_256_GCM_SHA384 compression: none
2021-12-04T23:12:01.427491Z 0 [Note] [MY-000000] [Galera] (96528c37-9f8d, ‘ssl://0.0.0.0:4567’) Found matching local endpoint for a connection, blacklisting address ssl://172.31.33.27:4567
2021-12-04T23:12:01.428895Z 0 [Note] [MY-000000] [Galera] (96528c37-9f8d, ‘ssl://0.0.0.0:4567’) connection established to 7929fac9-9d41 ssl://172.31.45.49:4567
2021-12-04T23:12:01.429143Z 0 [Note] [MY-000000] [Galera] (96528c37-9f8d, ‘ssl://0.0.0.0:4567’) turning message relay requesting on, nonlive peers:
2021-12-04T23:12:01.920869Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 → 1
2021-12-04T23:12:01.921122Z 0 [Note] [MY-000000] [Galera] declaring 7929fac9-9d41 at ssl://172.31.45.49:4567 stable
2021-12-04T23:12:01.921284Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 → 1
2021-12-04T23:12:01.921985Z 0 [Note] [MY-000000] [Galera] Node 7929fac9-9d41 state primary
2021-12-04T23:12:01.922945Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,7929fac9-9d41,2)
memb {
7929fac9-9d41,0
96528c37-9f8d,0
}
joined {
}
left {
}
partitioned {
}
)
2021-12-04T23:12:01.923114Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2021-12-04T23:12:01.925445Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://172.31.39.125:4567
2021-12-04T23:12:02.419012Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2021-12-04T23:12:02.419326Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2021-12-04T23:12:02.419552Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)
2021-12-04T23:12:02.419755Z 0 [Note] [MY-000000] [Galera] Opened channel ‘pxc-cluster’
2021-12-04T23:12:02.420248Z 1 [Note] [MY-000000] [WSREP] Starting applier thread 1
2021-12-04T23:12:02.420543Z 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2
2021-12-04T23:12:02.420816Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2021-12-04T23:12:02.421027Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: Waiting for state UUID.
2021-12-04T23:12:02.421367Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 969b9aeb-5557-11ec-87fd-ff907c91e781
2021-12-04T23:12:02.421617Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 969b9aeb-5557-11ec-87fd-ff907c91e781 from 0 (pxc-1)
2021-12-04T23:12:02.421989Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 969b9aeb-5557-11ec-87fd-ff907c91e781 from 1 (pxc-2)
2021-12-04T23:12:02.422205Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 1,
members = 1/2 (primary/total),
act_id = 2,
last_appl. = 1,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 0aad7cc0-5550-11ec-8127-e663ae9bb018
2021-12-04T23:12:02.422459Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2021-12-04T23:12:02.422662Z 0 [Note] [MY-000000] [Galera] Shifting OPEN → PRIMARY (TO: 3)
2021-12-04T23:12:02.422916Z 1 [Note] [MY-000000] [Galera] ####### processing CC 3, local, ordered
2021-12-04T23:12:02.423141Z 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 1 upto current CC event 3 upto:1
2021-12-04T23:12:02.423355Z 1 [Note] [MY-000000] [Galera] Drain monitors from 1 up to 1
2021-12-04T23:12:02.423583Z 1 [Note] [MY-000000] [Galera] Process first view: 0aad7cc0-5550-11ec-8127-e663ae9bb018 my uuid: 96528c37-5557-11ec-9f8d-16f32bd0f4eb
2021-12-04T23:12:02.423827Z 1 [Note] [MY-000000] [Galera] Server pxc-2 connected to cluster at position 0aad7cc0-5550-11ec-8127-e663ae9bb018:3 with ID 96528c37-5557-11ec-9f8d-16f32bd0f4eb
2021-12-04T23:12:02.424055Z 1 [Note] [MY-000000] [WSREP] Server status change disconnected → connected
2021-12-04T23:12:02.424290Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-12-04T23:12:02.424557Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 96528c37-5557-11ec-9f8d-16f32bd0f4eb
2021-12-04T23:12:02.424791Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2021-12-04T23:12:02.425049Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-12-04T23:12:02.425304Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2021-12-04T23:12:02.425543Z 1 [Note] [MY-000000] [Galera] State transfer required:
Group state: 0aad7cc0-5550-11ec-8127-e663ae9bb018:3
Local state: c160989b-5552-11ec-9f38-dad52a103d84:1
2021-12-04T23:12:02.425789Z 1 [Note] [MY-000000] [WSREP] Server status change connected → joiner
2021-12-04T23:12:02.426043Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-12-04T23:12:02.426415Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘172.31.33.27’ --datadir ‘/var/lib/mysql/’ --basedir ‘/usr/’ --plugindir ‘/usr/lib/mysql/plugin/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16923’ --mysqld-version ‘8.0.25-15.1’ ‘’ )
2021-12-04T23:12:03.510608Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|172.31.33.27:4444/xtrabackup_sst//1
2021-12-04T23:12:03.510958Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2021-12-04T23:12:03.511229Z 1 [Note] [MY-000000] [Galera] Local UUID: c160989b-5552-11ec-9f38-dad52a103d84 != Group UUID: 0aad7cc0-5550-11ec-8127-e663ae9bb018
2021-12-04T23:12:03.511504Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:c160989b-5552-11ec-9f38-dad52a103d84 f: 0, l: 3, STRv: 3
2021-12-04T23:12:03.511885Z 1 [Note] [MY-000000] [Galera] IST receiver addr using ssl://172.31.33.27:4568
2021-12-04T23:12:03.512219Z 1 [Note] [MY-000000] [Galera] IST receiver using ssl
2021-12-04T23:12:03.512758Z 1 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-3, listening at: ssl://172.31.33.27:4568
2021-12-04T23:12:03.516120Z 0 [Note] [MY-000000] [Galera] Member 1.0 (pxc-2) requested state transfer from ‘any’. Selected 0.0 (pxc-1)(SYNCED) as donor.
2021-12-04T23:12:03.516424Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY → JOINER (TO: 3)
2021-12-04T23:12:03.516751Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
2021-12-04T23:12:03.517045Z 1 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2021-12-04T23:12:03.517347Z 1 [Note] [MY-000000] [Galera] GCache history reset: c160989b-5552-11ec-9f38-dad52a103d84:1 → 0aad7cc0-5550-11ec-8127-e663ae9bb018:3
2021-12-04T23:12:03.518386Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 18446744073709551608 bytes
2021-12-04T23:12:03.518692Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 1/2 locked buffers
2021-12-04T23:12:04.919772Z 0 [Note] [MY-000000] [Galera] (96528c37-9f8d, ‘ssl://0.0.0.0:4567’) turning message relay requesting off
2021-12-04T23:13:43.378716Z 0 [Note] [MY-000000] [WSREP-SST] Trying to terminate (17382) socat -u openssl-listen:4444,reuseaddr,cert=/etc/mysql/certs/server-cert.pem,key=/etc/mysql/certs/server-key.pem,cafile=/etc/mysql/certs/ca.pem,verify=1,retry=30 stdio | /usr/bin/pxc_extra/pxb-8.0/bin/xbstream -x with SIGTERM
2021-12-04T23:13:44.397460Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2021-12-04T23:13:44.398198Z 0 [ERROR] [MY-000000] [WSREP-SST] Possible timeout in receving first data from donor in gtid/keyring stage
2021-12-04T23:13:44.398216Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1286
2021-12-04T23:13:44.398230Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************************************************
2021-12-04T23:13:44.398244Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32
2021-12-04T23:13:44.405295Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘172.31.33.27’ --datadir ‘/var/lib/mysql/’ --basedir ‘/usr/’ --plugindir ‘/usr/lib/mysql/plugin/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16923’ --mysqld-version ‘8.0.25-15.1’ ‘’ : 32 (Broken pipe)
2021-12-04T23:13:44.405397Z 0 [ERROR] [MY-000000] [WSREP] Failed to read uuid:seqno from joiner script.
2021-12-04T23:13:44.405423Z 0 [ERROR] [MY-000000] [WSREP] SST script aborted with error 32 (Broken pipe)
2021-12-04T23:13:44.405508Z 3 [Note] [MY-000000] [Galera] Processing SST received
2021-12-04T23:13:44.405546Z 3 [Note] [MY-000000] [Galera] SST request was cancelled
2021-12-04T23:13:44.405575Z 3 [ERROR] [MY-000000] [Galera] State transfer request failed unrecoverably: 32 (Broken pipe). Most likely it is due to inability to communicate with the cluster primary component. Restart required.
2021-12-04T23:13:44.405593Z 3 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()
2021-12-04T23:13:44.405613Z 3 [Note] [MY-000000] [Galera] Closing send monitor…
2021-12-04T23:13:44.405633Z 3 [Note] [MY-000000] [Galera] Closed send monitor.
2021-12-04T23:13:44.405651Z 3 [Note] [MY-000000] [Galera] gcomm: terminating thread
2021-12-04T23:13:44.405673Z 3 [Note] [MY-000000] [Galera] gcomm: joining thread
2021-12-04T23:13:44.405747Z 3 [Note] [MY-000000] [Galera] gcomm: closing backend
2021-12-04T23:13:45.407397Z 3 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,7929fac9-9d41,2)
memb {
96528c37-9f8d,0
}
joined {
}
left {
}
partitioned {
7929fac9-9d41,0
}
)
2021-12-04T23:13:45.407507Z 3 [Note] [MY-000000] [Galera] PC protocol downgrade 1 → 0
2021-12-04T23:13:45.407532Z 3 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2021-12-04T23:13:45.407881Z 3 [Note] [MY-000000] [Galera] gcomm: closed
2021-12-04T23:13:45.407925Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2021-12-04T23:13:45.407990Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2021-12-04T23:13:45.408011Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2021-12-04T23:13:45.408028Z 0 [Note] [MY-000000] [Galera] Shifting JOINER → OPEN (TO: 3)
2021-12-04T23:13:45.408046Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2021-12-04T23:13:45.408101Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2021-12-04T23:13:45.408118Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2021-12-04T23:13:45.408133Z 0 [Note] [MY-000000] [Galera] Shifting OPEN → CLOSED (TO: 3)
2021-12-04T23:13:45.408148Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2021-12-04T23:13:45.408256Z 3 [Note] [MY-000000] [Galera] recv_thread() joined.
2021-12-04T23:13:45.408275Z 3 [Note] [MY-000000] [Galera] Closing replication queue.
2021-12-04T23:13:45.408289Z 3 [Note] [MY-000000] [Galera] Closing slave action queue.
2021-12-04T23:13:45.408310Z 3 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2021-12-04T23:13:45.408325Z 3 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2021-12-04T23:13:45.408339Z 3 [Note] [MY-000000] [WSREP] Terminating SST process
Here my config for node #2:
I’ve been following the instructions from one of your technical writeups here:
I know I’m close(hopefully), thank you in advance!