Status of the cluster = > VAR=bash /usr/bin/mysql-systemd galera-recovery

PXC version : 8.0.27
OS : Centos 7
No .Of nodes : 3

Hi there ,

  • when iam checking the status of cluster … it is showing as VAR=bash /usr/bin/mysql-systemd galera-recovery
  • And the other nodes 2 & 3 are also not able to join the cluster …


[testuser@cluster1 mysql]$ sudo systemctl status mysql@bootstrap.service
● mysql@bootstrap.service - Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap
   Loaded: loaded (/usr/lib/systemd/system/mysql@.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-06-27 18:53:21 IST; 4min 35s ago
  Process: 29863 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=0/SUCCESS)
  Process: 29861 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 29675 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 29673 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 29630 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
 Main PID: 29727 (mysqld)
   Status: "Server is operational"
   CGroup: /system.slice/system-mysql.slice/mysql@bootstrap.service
           └─29727 /usr/sbin/mysqld --wsrep-new-cluster --wsrep_start_position=aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:17

Jun 27 18:53:19 cluster1 systemd[1]: Starting Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap...
Jun 27 18:53:21 cluster1 mysql-systemd[29863]: SUCCESS!
Jun 27 18:53:21 cluster1 systemd[1]: Started Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap.
[testuser@cluster1 mysql]$

Hi @AneeshBabu
Can you please share to us the error.log entries from node1 and one of the nodes attempting to join

Hi there
below is the log file of bootstrapped node (node1)…

LogFile of bootstrapped node (node1)

2022-06-28T00:42:06.278122+05:30 0 [Note] [MY-000000] [WSREP] Shutdown replication
2022-06-28T00:42:06.278175+05:30 0 [Note] [MY-000000] [WSREP] Server status change synced -> disconnecting
2022-06-28T00:42:06.278189+05:30 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:06.278216+05:30 0 [Note] [MY-000000] [Galera] Closing send monitor...
2022-06-28T00:42:06.278226+05:30 0 [Note] [MY-000000] [Galera] Closed send monitor.
2022-06-28T00:42:06.278238+05:30 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-06-28T00:42:06.278255+05:30 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-06-28T00:42:06.278364+05:30 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-06-28T00:42:06.278404+05:30 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-06-28T00:42:06.278416+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-06-28T00:42:06.278514+05:30 0 [Note] [MY-000000] [Galera] gcomm: closed
2022-06-28T00:42:06.278550+05:30 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2022-06-28T00:42:06.278597+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2022-06-28T00:42:06.278609+05:30 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2022-06-28T00:42:06.278618+05:30 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> CLOSED (TO: 18)
2022-06-28T00:42:06.278638+05:30 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2022-06-28T00:42:06.278685+05:30 2 [Note] [MY-000000] [Galera] Maybe drain monitors from 18 upto current CC event 18 upto:18
2022-06-28T00:42:06.278693+05:30 0 [Note] [MY-000000] [Galera] recv_thread() joined.
2022-06-28T00:42:06.278707+05:30 2 [Note] [MY-000000] [Galera] Drain monitors from 18 up to 18
2022-06-28T00:42:06.278713+05:30 0 [Note] [MY-000000] [Galera] Closing replication queue.
2022-06-28T00:42:06.278725+05:30 0 [Note] [MY-000000] [Galera] Closing slave action queue.
2022-06-28T00:42:06.278729+05:30 2 [Note] [MY-000000] [Galera] ================================================
View:
  id: aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:18
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: yes
  own_index: -1
  members(0):
=================================================
2022-06-28T00:42:06.278749+05:30 2 [Note] [MY-000000] [Galera] Non-primary view
2022-06-28T00:42:06.278759+05:30 2 [Note] [MY-000000] [WSREP] Server status change disconnecting -> disconnected
2022-06-28T00:42:06.278769+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:06.278783+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:06.278805+05:30 43 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 43
2022-06-28T00:42:06.278813+05:30 44 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 44
2022-06-28T00:42:06.278817+05:30 39 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 39
2022-06-28T00:42:06.278865+05:30 47 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 47
2022-06-28T00:42:06.278927+05:30 45 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 45
2022-06-28T00:42:06.278971+05:30 48 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 48
2022-06-28T00:42:06.278978+05:30 49 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 49
2022-06-28T00:42:06.279053+05:30 46 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 46
2022-06-28T00:42:06.279063+05:30 52 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 52
2022-06-28T00:42:06.279054+05:30 50 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 50
2022-06-28T00:42:06.279144+05:30 51 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 51
2022-06-28T00:42:06.279155+05:30 55 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 55
2022-06-28T00:42:06.279202+05:30 53 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 53
2022-06-28T00:42:06.279234+05:30 54 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 54
2022-06-28T00:42:06.279255+05:30 57 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 57
2022-06-28T00:42:06.279279+05:30 58 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 58
2022-06-28T00:42:06.279380+05:30 56 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 56
2022-06-28T00:42:06.279395+05:30 62 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 62
2022-06-28T00:42:06.279381+05:30 60 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 60
2022-06-28T00:42:06.279408+05:30 64 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 64
2022-06-28T00:42:06.279390+05:30 61 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 61
2022-06-28T00:42:06.279530+05:30 63 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 63
2022-06-28T00:42:06.279539+05:30 69 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 69
2022-06-28T00:42:06.279553+05:30 67 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 67
2022-06-28T00:42:06.279564+05:30 68 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 68
2022-06-28T00:42:06.279386+05:30 59 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 59
2022-06-28T00:42:06.279620+05:30 65 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 65
2022-06-28T00:42:06.279666+05:30 71 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 71
2022-06-28T00:42:06.279728+05:30 66 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 66
2022-06-28T00:42:06.279735+05:30 70 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 70
2022-06-28T00:42:06.279742+05:30 72 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 72
2022-06-28T00:42:06.280213+05:30 2 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 0 thd: 2
2022-06-28T00:42:06.280214+05:30 0 [Note] [MY-000000] [WSREP] Waiting for active wsrep applier to exit
2022-06-28T00:42:06.280264+05:30 0 [Note] [MY-000000] [WSREP] Waiting for (33) applier thread(s) to terminate
2022-06-28T00:42:06.288760+05:30 12 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 12
2022-06-28T00:42:06.288777+05:30 28 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 28
2022-06-28T00:42:06.288813+05:30 29 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 29
2022-06-28T00:42:06.288828+05:30 33 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 33
2022-06-28T00:42:06.288768+05:30 24 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 24
2022-06-28T00:42:06.288943+05:30 21 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 21
2022-06-28T00:42:06.288970+05:30 42 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 42
2022-06-28T00:42:06.288980+05:30 40 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 40
2022-06-28T00:42:06.288811+05:30 31 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 31
2022-06-28T00:42:06.289067+05:30 13 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 13
2022-06-28T00:42:06.289082+05:30 36 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 36
2022-06-28T00:42:06.288760+05:30 23 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 23
2022-06-28T00:42:06.289138+05:30 14 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 14
2022-06-28T00:42:06.288770+05:30 16 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 16
2022-06-28T00:42:06.289159+05:30 19 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 19
2022-06-28T00:42:06.289135+05:30 38 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 38
2022-06-28T00:42:06.289231+05:30 17 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 17
2022-06-28T00:42:06.289242+05:30 15 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 15
2022-06-28T00:42:06.289229+05:30 18 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 18
2022-06-28T00:42:06.289262+05:30 27 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 27
2022-06-28T00:42:06.289265+05:30 35 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 35
2022-06-28T00:42:06.288798+05:30 32 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 32
2022-06-28T00:42:06.289271+05:30 11 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 11
2022-06-28T00:42:06.289298+05:30 34 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 34
2022-06-28T00:42:06.289301+05:30 22 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 22
2022-06-28T00:42:06.289308+05:30 37 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 37
2022-06-28T00:42:06.289332+05:30 30 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 30
2022-06-28T00:42:06.289251+05:30 41 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 41
2022-06-28T00:42:06.288769+05:30 10 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 10
2022-06-28T00:42:06.289477+05:30 26 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 26
2022-06-28T00:42:06.289302+05:30 20 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 20
2022-06-28T00:42:06.289517+05:30 25 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 25
2022-06-28T00:42:07.280366+05:30 0 [Note] [MY-000000] [WSREP] All applier thread terminated. Will now terminate rollback thread
2022-06-28T00:42:07.280425+05:30 0 [Note] [MY-000000] [WSREP] Waiting for rollback thread to terminate
2022-06-28T00:42:07.280473+05:30 1 [Note] [MY-000000] [WSREP] rollbacker thread exiting 1
2022-06-28T00:42:08.280514+05:30 0 [Note] [MY-000000] [WSREP] Rollback thread terminated
2022-06-28T00:42:08.281352+05:30 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
2022-06-28T00:42:08.281378+05:30 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2022-06-28T00:42:08.282107+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-06-28T00:42:08.282801+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-06-28T00:42:08.283487+05:30 0 [Note] [MY-000000] [Galera] mon: entered 3 oooe fraction 0 oool fraction 0
2022-06-28T00:42:08.283501+05:30 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
2022-06-28T00:42:08.283507+05:30 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
2022-06-28T00:42:08.283516+05:30 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
2022-06-28T00:42:08.283522+05:30 0 [Note] [MY-000000] [Galera] avg deps dist 0
2022-06-28T00:42:08.283527+05:30 0 [Note] [MY-000000] [Galera] avg cert interval 0
2022-06-28T00:42:08.283532+05:30 0 [Note] [MY-000000] [Galera] cert index size 0
2022-06-28T00:42:08.283567+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-28T00:42:08.283610+05:30 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
2022-06-28T00:42:08.283621+05:30 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2022-06-28T00:42:08.284688+05:30 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...
2022-06-28T00:42:09.507138+05:30 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.27-18.1)  Percona XtraDB Cluster (GPL), Release rel18, Revision ac35177, WSREP version 26.4.3.
 INFO: Skipping wsrep-recover for aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:18 pair
 INFO: Assigning aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:18 to wsrep_start_position
2022-06-28T00:42:09.918898+05:30 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-06-28T00:42:09.918918+05:30 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-18.1) starting as process 30614
2022-06-28T00:42:09.921501+05:30 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2022-06-28T00:42:09.921511+05:30 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2022-06-28T00:42:09.921621+05:30 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_main. See below for the description of exact issue.
2022-06-28T00:42:09.921633+05:30 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL_CTX_new failed
2022-06-28T00:42:09.921678+05:30 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/galera4/libgalera_smm.so initial position: aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:18
2022-06-28T00:42:09.921697+05:30 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/galera4/libgalera_smm.so'
2022-06-28T00:42:09.923069+05:30 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.10(9728532) by Codership Oy <info@codership.com> (modified by Percona <https://percona.com/>) loaded successfully.
2022-06-28T00:42:09.923102+05:30 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
2022-06-28T00:42:09.923913+05:30 0 [Note] [MY-000000] [Galera] Found saved state: aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:18, safe_to_bootstrap: 1
2022-06-28T00:42:09.923984+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9
Seqno: 1 - 18
Offset: 1280
Synced: 1
2022-06-28T00:42:09.924000+05:30 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9, offset: 1280
2022-06-28T00:42:09.924060+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
2022-06-28T00:42:09.924084+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2022-06-28T00:42:09.924097+05:30 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-18
2022-06-28T00:42:09.924125+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...  0.0% (   0/3064 bytes) complete.
2022-06-28T00:42:09.924138+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (3064/3064 bytes) complete.
2022-06-28T00:42:09.924148+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/18 locked buffers
2022-06-28T00:42:09.924159+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): free space: 134214664/134217728
2022-06-28T00:42:09.924881+05:30 0 [Warning] [MY-000000] [Galera] Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead.
2022-06-28T00:42:09.925394+05:30 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.1.1; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem; socket.ssl_reload = 1;
2022-06-28T00:42:09.929181+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-28T00:42:09.929247+05:30 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:18, protocol version: -1
2022-06-28T00:42:09.929319+05:30 0 [Note] [MY-000000] [WSREP] Starting replication
2022-06-28T00:42:09.929337+05:30 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1
2022-06-28T00:42:09.929354+05:30 0 [Note] [MY-000000] [Galera] Setting GCS initial position to aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:18
2022-06-28T00:42:09.929404+05:30 0 [Note] [MY-000000] [Galera] protonet asio version 0
2022-06-28T00:42:09.929600+05:30 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2022-06-28T00:42:09.929621+05:30 0 [Note] [MY-000000] [Galera] backend: asio
2022-06-28T00:42:09.929665+05:30 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2022-06-28T00:42:09.929739+05:30 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2022-06-28T00:42:09.929752+05:30 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2022-06-28T00:42:09.929922+05:30 0 [Note] [MY-000000] [Galera] GMCast version 0
2022-06-28T00:42:09.930041+05:30 0 [Note] [MY-000000] [Galera] (0b02e9b1-ada4, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2022-06-28T00:42:09.930053+05:30 0 [Note] [MY-000000] [Galera] (0b02e9b1-ada4, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2022-06-28T00:42:09.930242+05:30 0 [Note] [MY-000000] [Galera] EVS version 1
2022-06-28T00:42:09.930308+05:30 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group 'pxcuat'
2022-06-28T00:42:09.930338+05:30 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery
2022-06-28T00:42:09.930846+05:30 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2022-06-28T00:42:09.930894+05:30 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2022-06-28T00:42:09.930927+05:30 0 [Note] [MY-000000] [Galera] Node 0b02e9b1-ada4 state primary
2022-06-28T00:42:09.930954+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,0b02e9b1-ada4,1)
memb {
        0b02e9b1-ada4,0
        }
joined {
        }
left {
        }
partitioned {
        }
)
2022-06-28T00:42:09.930970+05:30 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-06-28T00:42:09.932635+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://192.168.1.2:4567
2022-06-28T00:42:09.932657+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x769a9e0
2022-06-28T00:42:09.932679+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://192.168.1.3:4567
2022-06-28T00:42:09.932691+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x769af90
2022-06-28T00:42:09.932706+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://192.168.1.1:4567
2022-06-28T00:42:09.932718+05:30 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x769b660
2022-06-28T00:42:09.932750+05:30 0 [Note] [MY-000000] [Galera] gcomm: connected
2022-06-28T00:42:09.932778+05:30 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2022-06-28T00:42:09.932880+05:30 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2022-06-28T00:42:09.932902+05:30 0 [Note] [MY-000000] [Galera] Opened channel 'pxcuat'
2022-06-28T00:42:09.933230+05:30 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2022-06-28T00:42:09.933459+05:30 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 0b0375c8-f64d-11ec-8493-7755149a1716
2022-06-28T00:42:09.933473+05:30 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2022-06-28T00:42:09.933506+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 0b0375c8-f64d-11ec-8493-7755149a1716
2022-06-28T00:42:09.933530+05:30 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 0b0375c8-f64d-11ec-8493-7755149a1716 from 0 (pxc1)
2022-06-28T00:42:09.933548+05:30 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2022-06-28T00:42:09.933576+05:30 0 [Note] [MY-000000] [Galera] Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 0,
        members    = 1/1 (primary/total),
        act_id     = 18,
        last_appl. = 18,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9
2022-06-28T00:42:09.933613+05:30 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2022-06-28T00:42:09.933626+05:30 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (19)
2022-06-28T00:42:09.933656+05:30 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc1) synced with group.
2022-06-28T00:42:09.933667+05:30 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 19)
2022-06-28T00:42:09.933678+05:30 2 [Note] [MY-000000] [Galera] ####### processing CC 19, local, ordered
2022-06-28T00:42:09.933698+05:30 2 [Note] [MY-000000] [Galera] Maybe drain monitors from 18 upto current CC event 19 upto:18
2022-06-28T00:42:09.933710+05:30 2 [Note] [MY-000000] [Galera] Drain monitors from 18 up to 18
2022-06-28T00:42:09.933729+05:30 2 [Note] [MY-000000] [Galera] Process first view: aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9 my uuid: 0b02e9b1-f64d-11ec-ada4-1bf1e9eaeec3
2022-06-28T00:42:09.933750+05:30 2 [Note] [MY-000000] [Galera] Server pxc1 connected to cluster at position aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:19 with ID 0b02e9b1-f64d-11ec-ada4-1bf1e9eaeec3
2022-06-28T00:42:09.933765+05:30 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2022-06-28T00:42:09.933780+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:09.933807+05:30 2 [Note] [MY-000000] [Galera] ####### My UUID: 0b02e9b1-f64d-11ec-ada4-1bf1e9eaeec3
2022-06-28T00:42:09.933826+05:30 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no
2022-06-28T00:42:09.933877+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-28T00:42:09.933908+05:30 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2022-06-28T00:42:09.933924+05:30 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2022-06-28T00:42:09.933938+05:30 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 19
2022-06-28T00:42:09.933961+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-28T00:42:09.935442+05:30 2 [Note] [MY-000000] [Galera] ================================================
View:
  id: aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9:19
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: 0b02e9b1-f64d-11ec-ada4-1bf1e9eaeec3, pxc1
=================================================
2022-06-28T00:42:09.935459+05:30 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-06-28T00:42:09.935473+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:09.935490+05:30 2 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing
2022-06-28T00:42:09.935500+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:09.936003+05:30 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
2022-06-28T00:42:09.938520+05:30 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-06-28T00:42:10.817340+05:30 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-06-28T00:42:10.904404+05:30 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2022-06-28T00:42:10.908995+05:30 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2022-06-28T00:42:10.993705+05:30 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2022-06-28T00:42:10.993723+05:30 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2022-06-28T00:42:10.994204+05:30 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-06-28T00:42:10.994235+05:30 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-06-28T00:42:11.001477+05:30 0 [Warning] [MY-010324] [Server] 'db' entry 'percona_schema mysql.pxc.sst.role@localhost' had database in mixed case that has been forced to lowercase because lower_case_table_names is set. It will not be possible to remove this privilege using REVOKE.
2022-06-28T00:42:11.007176+05:30 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2022-06-28T00:42:11.007199+05:30 0 [Note] [MY-000000] [Galera] Server initialized
2022-06-28T00:42:11.007213+05:30 0 [Note] [MY-000000] [WSREP] Server status change initializing -> initialized
2022-06-28T00:42:11.007228+05:30 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:11.007283+05:30 2 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
2022-06-28T00:42:11.010646+05:30 8 [Note] [MY-000000] [WSREP] Recovered cluster id aa86c6b2-f38e-11ec-b0ce-7aaf91441ec9
2022-06-28T00:42:11.012844+05:30 10 [Note] [MY-000000] [WSREP] Starting applier thread 10
2022-06-28T00:42:11.013325+05:30 11 [Note] [MY-000000] [WSREP] Starting applier thread 11
2022-06-28T00:42:11.013416+05:30 12 [Note] [MY-000000] [WSREP] Starting applier thread 12
2022-06-28T00:42:11.013462+05:30 16 [Note] [MY-000000] [WSREP] Starting applier thread 16
2022-06-28T00:42:11.013436+05:30 14 [Note] [MY-000000] [WSREP] Starting applier thread 14
2022-06-28T00:42:11.013381+05:30 13 [Note] [MY-000000] [WSREP] Starting applier thread 13
2022-06-28T00:42:11.013437+05:30 15 [Note] [MY-000000] [WSREP] Starting applier thread 15
2022-06-28T00:42:11.013969+05:30 17 [Note] [MY-000000] [WSREP] Starting applier thread 17
2022-06-28T00:42:11.013981+05:30 18 [Note] [MY-000000] [WSREP] Starting applier thread 18
2022-06-28T00:42:11.013997+05:30 19 [Note] [MY-000000] [WSREP] Starting applier thread 19
2022-06-28T00:42:11.014112+05:30 20 [Note] [MY-000000] [WSREP] Starting applier thread 20
2022-06-28T00:42:11.014057+05:30 2 [Note] [MY-000000] [WSREP] Server status change initialized -> joined
2022-06-28T00:42:11.014132+05:30 21 [Note] [MY-000000] [WSREP] Starting applier thread 21
2022-06-28T00:42:11.014172+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:11.014215+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-06-28T00:42:11.014233+05:30 22 [Note] [MY-000000] [WSREP] Starting applier thread 22
2022-06-28T00:42:11.020782+05:30 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql/mysqlx.sock
2022-06-28T00:42:11.020837+05:30 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27-18.1'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Percona XtraDB Cluster (GPL), Release rel18, Revision ac35177, WSREP version 26.4.3.
2022-06-28T00:42:11.025433+05:30 2 [Note] [MY-000000] [Galera] Recording CC from group: 19
2022-06-28T00:42:11.025452+05:30 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 19
2022-06-28T00:42:11.025460+05:30 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2022-06-28T00:42:11.025486+05:30 2 [Note] [MY-000000] [Galera] Server pxc1 synced with group
2022-06-28T00:42:11.025497+05:30 2 [Note] [MY-000000] [WSREP] Server status change joined -> synced
2022-06-28T00:42:11.025505+05:30 2 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2022-06-28T00:42:11.025514+05:30 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
[testuser@testserveruat ~]$

continuation …

LogFile of node3 - attempting to join

2022-06-28T00:53:43.456642+05:30 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-18.1) starting as process 29918
2022-06-28T00:53:43.459034+05:30 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2022-06-28T00:53:43.459057+05:30 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2022-06-28T00:53:43.459176+05:30 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_main. See below for the description of exact issue.
2022-06-28T00:53:43.459191+05:30 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL_CTX_new failed
2022-06-28T00:53:43.459210+05:30 0 [ERROR] [MY-000000] [WSREP] New joining cluster node didn't find all needed SSL artifacts
2022-06-28T00:53:43.459266+05:30 0 [ERROR] [MY-010119] [Server] Aborting
2022-06-28T00:53:43.459435+05:30 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.27-18.1)  Percona XtraDB Cluster (GPL), Release rel18, Revision ac35177, WSREP version 26.4.3.