Hi there,
I have a PCX with 3 nodes, each one in a separated AWS EC2 instance, I could install everything without any issue, I bootstrapped the first node and when I try to start the second one it cannot start.
Here is my error.log from the first node
2022-06-01T19:08:34.824261Z 0 [Note] [MY-000000] [Galera] (eb715e2b-b844, 'ssl://0.0.0.0:4567') connection established to 3c0cac31-a842 ssl://54.83.86.171:4567
2022-06-01T19:08:34.824961Z 0 [Note] [MY-000000] [Galera] (eb715e2b-b844, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2022-06-01T19:08:35.320282Z 0 [Note] [MY-000000] [Galera] declaring 3c0cac31-a842 at ssl://54.83.86.171:4567 stable
2022-06-01T19:08:35.320957Z 0 [Note] [MY-000000] [Galera] Node eb715e2b-b844 state primary
2022-06-01T19:08:35.324939Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,3c0cac31-a842,2)
memb {
3c0cac31-a842,0
eb715e2b-b844,0
}
joined {
}
left {
}
partitioned {
}
)
2022-06-01T19:08:35.324985Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-01T19:08:35.329645Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2022-06-01T19:08:35.329697Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: Waiting for state UUID.
2022-06-01T19:08:35.820491Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 3ca59654-e1de-11ec-b875-3b1c6a7c8838
2022-06-01T19:08:35.821137Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 3ca59654-e1de-11ec-b875-3b1c6a7c8838 from 0 (pxc2)
2022-06-01T19:08:35.821178Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 3ca59654-e1de-11ec-b875-3b1c6a7c8838 from 1 (pxc1)
2022-06-01T19:08:35.821194Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 1,
members = 1/2 (primary/total),
act_id = 11,
last_appl. = 10,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 9c36ebea-e1d4-11ec-8283-4a1456563429
2022-06-01T19:08:35.821256Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2022-06-01T19:08:35.821403Z 2 [Note] [MY-000000] [Galera] ####### processing CC 12, local, ordered
2022-06-01T19:08:35.821449Z 2 [Note] [MY-000000] [Galera] Maybe drain monitors from 11 upto current CC event 12 upto:11
2022-06-01T19:08:35.821465Z 2 [Note] [MY-000000] [Galera] Drain monitors from 11 up to 11
2022-06-01T19:08:35.821481Z 2 [Note] [MY-000000] [Galera] ####### My UUID: eb715e2b-e1dd-11ec-b844-d61c3a3d1d8b
2022-06-01T19:08:35.821493Z 2 [Note] [MY-000000] [Galera] Skipping cert index reset
2022-06-01T19:08:35.821506Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2022-06-01T19:08:35.821517Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 11 -> 12
2022-06-01T19:08:35.821558Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-01T19:08:35.832607Z 2 [Note] [MY-000000] [Galera] ================================================
View:
id: 9c36ebea-e1d4-11ec-8283-4a1456563429:12
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 1
members(2):
0: 3c0cac31-e1de-11ec-a842-3a88e9c0c0ca, pxc2
1: eb715e2b-e1dd-11ec-b844-d61c3a3d1d8b, pxc1
=================================================
2022-06-01T19:08:35.832662Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-01T19:08:35.845637Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 12
2022-06-01T19:08:35.845678Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 12
2022-06-01T19:08:35.845692Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2022-06-01T19:08:36.506986Z 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc2) requested state transfer from '*any*'. Selected 1.0 (pxc1)(SYNCED) as donor.
2022-06-01T19:08:36.507057Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> DONOR/DESYNCED (TO: 12)
2022-06-01T19:08:36.507150Z 2 [Note] [MY-000000] [Galera] Detected STR version: 1, req_len: 64, req: STRv1
2022-06-01T19:08:36.507202Z 2 [Warning] [MY-000000] [Galera] Joiner didn't provide IST connection info - cert. index preload impossible, bailing out.
2022-06-01T19:08:36.507907Z 0 [Warning] [MY-000000] [Galera] 1.0 (pxc1): State transfer to 0.0 (pxc2) failed: -42 (No message of desired type)
2022-06-01T19:08:36.507940Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 12)
2022-06-01T19:08:36.508556Z 0 [Note] [MY-000000] [Galera] Member 1.0 (pxc1) synced with group.
2022-06-01T19:08:36.508586Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 12)
2022-06-01T19:08:36.508628Z 15 [Note] [MY-000000] [Galera] Server pxc1 synced with group
2022-06-01T19:08:37.509447Z 0 [Note] [MY-000000] [Galera] Deferred close timer started for socket with remote endpoint: ssl://54.83.86.171:47080
2022-06-01T19:08:37.509539Z 0 [Note] [MY-000000] [Galera] forgetting 3c0cac31-a842 (ssl://54.83.86.171:4567)
2022-06-01T19:08:37.509583Z 0 [Note] [MY-000000] [Galera] Node eb715e2b-b844 state primary
2022-06-01T19:08:37.509623Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,eb715e2b-b844,3)
memb {
eb715e2b-b844,0
}
joined {
}
left {
}
partitioned {
3c0cac31-a842,0
}
)
2022-06-01T19:08:37.509669Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-01T19:08:37.512916Z 0 [Note] [MY-000000] [Galera] forgetting 3c0cac31-a842 (ssl://54.83.86.171:4567)
2022-06-01T19:08:37.512957Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2022-06-01T19:08:37.513072Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 3da92f28-e1de-11ec-a2d0-2be0cda9b3dd
2022-06-01T19:08:37.513127Z 0 [Note] [MY-000000] [Galera] Deferred close timer handle_wait Operation aborted. for 0x556e1e8c96d0
2022-06-01T19:08:37.513167Z 0 [Note] [MY-000000] [Galera] Deferred close timer destruct
2022-06-01T19:08:37.513133Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 3da92f28-e1de-11ec-a2d0-2be0cda9b3dd
2022-06-01T19:08:37.513234Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 3da92f28-e1de-11ec-a2d0-2be0cda9b3dd from 0 (pxc1)
2022-06-01T19:08:37.513262Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 2,
members = 1/1 (primary/total),
act_id = 12,
last_appl. = 10,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 9c36ebea-e1d4-11ec-8283-4a1456563429
2022-06-01T19:08:37.513316Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2022-06-01T19:08:37.513509Z 12 [Note] [MY-000000] [Galera] ####### processing CC 13, local, ordered
2022-06-01T19:08:37.513557Z 12 [Note] [MY-000000] [Galera] Maybe drain monitors from 12 upto current CC event 13 upto:12
2022-06-01T19:08:37.513572Z 12 [Note] [MY-000000] [Galera] Drain monitors from 12 up to 12
2022-06-01T19:08:37.513601Z 12 [Note] [MY-000000] [Galera] ####### My UUID: eb715e2b-e1dd-11ec-b844-d61c3a3d1d8b
2022-06-01T19:08:37.513623Z 12 [Note] [MY-000000] [Galera] Skipping cert index reset
2022-06-01T19:08:37.513634Z 12 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2022-06-01T19:08:37.513645Z 12 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 12 -> 13
2022-06-01T19:08:37.513688Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-01T19:08:37.520298Z 12 [Note] [MY-000000] [Galera] ================================================
View:
id: 9c36ebea-e1d4-11ec-8283-4a1456563429:13
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: eb715e2b-e1dd-11ec-b844-d61c3a3d1d8b, pxc1
=================================================
2022-06-01T19:08:37.520371Z 12 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-01T19:08:37.528269Z 12 [Note] [MY-000000] [Galera] Recording CC from group: 13
2022-06-01T19:08:37.528312Z 12 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 13
2022-06-01T19:08:37.528327Z 12 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2022-06-01T19:08:38.100366Z 0 [Note] [MY-000000] [Galera] (eb715e2b-b844, 'ssl://0.0.0.0:4567') turning message relay requesting off
2022-06-01T19:08:42.600792Z 0 [Note] [MY-000000] [Galera] cleaning up 3c0cac31-a842 (ssl://54.83.86.171:4567)
And here is the log from my second node
Log of wsrep recovery (--wsrep-recover):
INFO: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery_verbose.jZ5Beg' --pid-file='/var/lib/mysql/ip-172-31-26-122-recover.pid'
INFO: WSREP: Recovered position a0111776-e1d4-11ec-ac9d-625f66f7f006:1
2022-06-01T19:08:34.390587Z 0 [Warning] [MY-011068] [Server] The syntax 'wsrep_slave_threads' is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.
2022-06-01T19:08:34.393300Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-18.1) starting as process 20311
2022-06-01T19:08:34.395903Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2022-06-01T19:08:34.395933Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2022-06-01T19:08:34.397906Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-06-01T19:08:34.397967Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-06-01T19:08:34.397993Z 0 [Note] [MY-000000] [WSREP] New joining cluster node configured to use specified SSL artifacts
2022-06-01T19:08:34.398030Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib/galera4/libgalera_smm.so initial position: a0111776-e1d4-11ec-ac9d-625f66f7f006:1
2022-06-01T19:08:34.398068Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib/galera4/libgalera_smm.so'
2022-06-01T19:08:34.398692Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.10(9728532) by Codership Oy <info@codership.com> (modified by Percona <https://percona.com/>) loaded successfully.
2022-06-01T19:08:34.398756Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
2022-06-01T19:08:34.399332Z 0 [Note] [MY-000000] [Galera] Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2022-06-01T19:08:34.399456Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: 9c36ebea-e1d4-11ec-8283-4a1456563429
Seqno: -1 - -1
Offset: -1
Synced: 0
2022-06-01T19:08:34.399513Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 9c36ebea-e1d4-11ec-8283-4a1456563429, offset: -1
2022-06-01T19:08:34.399613Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.
2022-06-01T19:08:34.452296Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2022-06-01T19:08:34.452369Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: didn't recover any events.
2022-06-01T19:08:34.452995Z 0 [Note] [MY-000000] [Galera] Complete reset of the galera cache
2022-06-01T19:08:34.549150Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...
2022-06-01T19:08:34.791782Z 0 [Warning] [MY-000000] [Galera] Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead.
2022-06-01T19:08:34.792633Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 54.83.86.171; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem; socket.ssl_reload = 1;
2022-06-01T19:08:34.808763Z 0 [Note] [MY-000000] [WSREP] Starting replication
2022-06-01T19:08:34.808822Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0
2022-06-01T19:08:34.808856Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2022-06-01T19:08:34.808918Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2022-06-01T19:08:34.809242Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2022-06-01T19:08:34.809292Z 0 [Note] [MY-000000] [Galera] backend: asio
2022-06-01T19:08:34.809376Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2022-06-01T19:08:34.809485Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2022-06-01T19:08:34.809524Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2022-06-01T19:08:34.809712Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2022-06-01T19:08:34.809895Z 0 [Note] [MY-000000] [Galera] (3c0cac31-a842, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2022-06-01T19:08:34.809944Z 0 [Note] [MY-000000] [Galera] (3c0cac31-a842, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2022-06-01T19:08:34.810218Z 0 [Note] [MY-000000] [Galera] EVS version 1
2022-06-01T19:08:34.810327Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-cluster', peer '34.224.223.50:,54.83.86.171:,18.232.149.232:'
2022-06-01T19:08:34.817101Z 0 [Note] [MY-000000] [Galera] (3c0cac31-a842, 'ssl://0.0.0.0:4567') connection established to eb715e2b-b844 ssl://34.224.223.50:4567
2022-06-01T19:08:34.817275Z 0 [Note] [MY-000000] [Galera] (3c0cac31-a842, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2022-06-01T19:08:34.817347Z 0 [Note] [MY-000000] [Galera] (3c0cac31-a842, 'ssl://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address ssl://54.83.86.171:4567
2022-06-01T19:08:35.312501Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2022-06-01T19:08:35.312591Z 0 [Note] [MY-000000] [Galera] declaring eb715e2b-b844 at ssl://34.224.223.50:4567 stable
2022-06-01T19:08:35.312629Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2022-06-01T19:08:35.313381Z 0 [Note] [MY-000000] [Galera] Node eb715e2b-b844 state primary
2022-06-01T19:08:35.314056Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,3c0cac31-a842,2)
memb {
3c0cac31-a842,0
eb715e2b-b844,0
}
joined {
}
left {
}
partitioned {
}
)
2022-06-01T19:08:35.314112Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-01T19:08:35.317063Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://18.232.149.232:4567
2022-06-01T19:08:35.811005Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2022-06-01T19:08:35.811107Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2022-06-01T19:08:35.811358Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2022-06-01T19:08:35.811399Z 0 [Note] [MY-000000] [Galera] Opened channel 'pxc-cluster'
2022-06-01T19:08:35.811581Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-06-01T19:08:35.811810Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 3ca59654-e1de-11ec-b875-3b1c6a7c8838
2022-06-01T19:08:35.811960Z 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2
2022-06-01T19:08:35.812028Z 1 [Note] [MY-000000] [WSREP] Starting applier thread 1
2022-06-01T19:08:35.812625Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 3ca59654-e1de-11ec-b875-3b1c6a7c8838
2022-06-01T19:08:35.813317Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 3ca59654-e1de-11ec-b875-3b1c6a7c8838 from 0 (pxc2)
2022-06-01T19:08:35.813909Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 3ca59654-e1de-11ec-b875-3b1c6a7c8838 from 1 (pxc1)
2022-06-01T19:08:35.813952Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 1,
members = 1/2 (primary/total),
act_id = 11,
last_appl. = 10,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 9c36ebea-e1d4-11ec-8283-4a1456563429
2022-06-01T19:08:35.814001Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2022-06-01T19:08:35.814036Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 12)
2022-06-01T19:08:35.814129Z 1 [Note] [MY-000000] [Galera] ####### processing CC 12, local, ordered
2022-06-01T19:08:35.814183Z 1 [Note] [MY-000000] [Galera] Maybe drain monitors from -1 upto current CC event 12 upto:-1
2022-06-01T19:08:35.814209Z 1 [Note] [MY-000000] [Galera] Drain monitors from -1 up to -1
2022-06-01T19:08:35.814247Z 1 [Note] [MY-000000] [Galera] Process first view: 9c36ebea-e1d4-11ec-8283-4a1456563429 my uuid: 3c0cac31-e1de-11ec-a842-3a88e9c0c0ca
2022-06-01T19:08:35.814284Z 1 [Note] [MY-000000] [Galera] Server pxc2 connected to cluster at position 9c36ebea-e1d4-11ec-8283-4a1456563429:12 with ID 3c0cac31-e1de-11ec-a842-3a88e9c0c0ca
2022-06-01T19:08:35.814313Z 1 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2022-06-01T19:08:35.814370Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-01T19:08:35.814435Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 3c0cac31-e1de-11ec-a842-3a88e9c0c0ca
2022-06-01T19:08:35.814477Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2022-06-01T19:08:35.814594Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-01T19:08:35.814682Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2022-06-01T19:08:35.814721Z 1 [Note] [MY-000000] [Galera] State transfer required:
Group state: 9c36ebea-e1d4-11ec-8283-4a1456563429:12
Local state: 00000000-0000-0000-0000-000000000000:-1
2022-06-01T19:08:35.814744Z 1 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-06-01T19:08:35.814773Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-01T19:08:35.814990Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '54.83.86.171' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib/mysql/plugin/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '20311' --mysqld-version '8.0.27-18.1' '' )
2022-06-01T19:08:36.138648Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2022-06-01T19:08:36.497640Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|54.83.86.171:4444/xtrabackup_sst//1
2022-06-01T19:08:36.497716Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2022-06-01T19:08:36.497758Z 1 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 9c36ebea-e1d4-11ec-8283-4a1456563429
2022-06-01T19:08:36.497805Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 12, STRv: 3
2022-06-01T19:08:36.497943Z 1 [Note] [MY-000000] [Galera] IST receiver addr using ssl://54.83.86.171:4568
2022-06-01T19:08:36.498048Z 1 [Note] [MY-000000] [Galera] IST receiver using ssl
2022-06-01T19:08:36.498220Z 1 [Note] [MY-000000] [Galera] State gap can't be serviced using IST. Switching to SST
2022-06-01T19:08:36.498268Z 1 [Warning] [MY-000000] [Galera] Failed to prepare for incremental state transfer: Failed to open IST listener at ssl://54.83.86.171:4568', asio error 'Failed to listen: bind: Cannot assign requested address: 99 (Cannot assign requested address)
at galerautils/src/gu_asio_stream_react.cpp:listen():746': 99 (Cannot assign requested address)
at galera/src/ist.cpp:prepare():344. IST will be unavailable.
2022-06-01T19:08:36.499154Z 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc2) requested state transfer from '*any*'. Selected 1.0 (pxc1)(SYNCED) as donor.
2022-06-01T19:08:36.499209Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 12)
2022-06-01T19:08:36.499280Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1
2022-06-01T19:08:36.499335Z 1 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2022-06-01T19:08:36.499380Z 1 [Note] [MY-000000] [Galera] GCache history reset: 9c36ebea-e1d4-11ec-8283-4a1456563429:0 -> 9c36ebea-e1d4-11ec-8283-4a1456563429:12
2022-06-01T19:08:36.500669Z 0 [Warning] [MY-000000] [Galera] 1.0 (pxc1): State transfer to 0.0 (pxc2) failed: -42 (No message of desired type)
2022-06-01T19:08:36.501097Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receive state. Need to abort.
2022-06-01T19:08:36.501134Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-06-01T19:08:36.501161Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-06-01T19:08:36.501267Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-06-01T19:08:37.505813Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,3c0cac31-a842,2)
memb {
3c0cac31-a842,0
}
joined {
}
left {
}
partitioned {
eb715e2b-b844,0
}
)
2022-06-01T19:08:37.505924Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-06-01T19:08:37.505952Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-06-01T19:08:37.506116Z 0 [Note] [MY-000000] [Galera] gcomm: closed
2022-06-01T19:08:37.506164Z 0 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2022-06-01T19:08:37.506185Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2022-06-01T19:08:37.506202Z 0 [Note] [MY-000000] [WSREP] Terminating SST process
mysqld.cnf
# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800
######## wsrep ###############
# Path to Galera library
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://34.224.223.50,54.83.86.171,18.232.149.232
# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
# Slave thread to use
wsrep_slave_threads=8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2
# Node IP address
wsrep_node_address=54.83.86.171
# Cluster name
wsrep_cluster_name=pxc-cluster
#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=pxc2
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING
# SST method
wsrep_sst_method=xtrabackup-v2
wsrep_provider_options="socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"
[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem
If someone could check if I missing anything I’ll apreciate.
Thanks