Cluster version : PXC 8.0.27
OS : Centos 7
No . Of nodes : 3
Hi there ,
When i am trying to add the nod2e to the cluster ,below is the error reported …
Note : Already bootstrapped the node 1, trying to add the node 2(this is the first time iam trying to add ) .
[testuser@test3 etc]$ sudo systemctl status mysql.service
● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2022-06-24 18:14:14 IST; 58s ago
Process: 19152 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
Process: 16244 ExecStop=/usr/bin/mysql-systemd stop (code=exited, status=0/SUCCESS)
Process: 16166 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=0/SUCCESS)
Process: 16164 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 19148 ExecStart=/usr/sbin/mysqld $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 19097 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 19094 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 19053 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Main PID: 19148 (code=exited, status=1/FAILURE)
Status: "Server startup in progress"
Error: 2 (No such file or directory)
Jun 24 18:14:13 test3 systemd[1]: Starting Percona XtraDB Cluster...
Jun 24 18:14:14 test3 systemd[1]: mysql.service: main process exited, code=exited, status=1/FAILURE
Jun 24 18:14:14 test3 mysql-systemd[19152]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Jun 24 18:14:14 test3 mysql-systemd[19152]: WARNING: mysql may be already dead
Jun 24 18:14:14 test3 systemd[1]: Failed to start Percona XtraDB Cluster.
Jun 24 18:14:14 test3 systemd[1]: Unit mysql.service entered failed state.
Jun 24 18:14:14 test3 systemd[1]: mysql.service failed.
Logfile
2022-06-24T17:40:59.856198+05:30 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-18.1) starting as process 18565
2022-06-24T17:40:59.858771+05:30 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2022-06-24T17:40:59.858784+05:30 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2022-06-24T17:40:59.858910+05:30 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_main. See below for the description of exact issue.
2022-06-24T17:40:59.858935+05:30 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL_CTX_new failed
2022-06-24T17:40:59.858957+05:30 0 [Note] [MY-000000] [WSREP] New joining cluster node configured to use specified SSL artifacts
2022-06-24T17:40:59.859009+05:30 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/galera4/libgalera_smm.so initial position: 0b30b9d6-f3a7-11ec-a44d-67d27c33010c:1
2022-06-24T17:40:59.859052+05:30 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/galera4/libgalera_smm.so'
2022-06-24T17:40:59.860430+05:30 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.10(9728532) by Codership Oy <info@codership.com> (modified by Percona <https://percona.com/>) loaded successfully.
2022-06-24T17:40:59.860487+05:30 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
2022-06-24T17:40:59.861267+05:30 0 [Note] [MY-000000] [Galera] Found saved state: 0b30b9d6-f3a7-11ec-a44d-67d27c33010c:1, safe_to_bootstrap: 0
2022-06-24T17:40:59.861362+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: 0b30b9d6-f3a7-11ec-a44d-67d27c33010c
Seqno: 1 - 1
Offset: 1280
Synced: 1
2022-06-24T17:40:59.861394+05:30 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 0b30b9d6-f3a7-11ec-a44d-67d27c33010c, offset: 1280
2022-06-24T17:40:59.861466+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.
2022-06-24T17:40:59.861504+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2022-06-24T17:40:59.861533+05:30 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1
2022-06-24T17:40:59.861579+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/176 bytes) complete.
2022-06-24T17:40:59.861605+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (176/176 bytes) complete.
2022-06-24T17:40:59.861630+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers
2022-06-24T17:40:59.861657+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): free space: 134217552/134217728
2022-06-24T17:40:59.862325+05:30 0 [Warning] [MY-000000] [Galera] Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead.
2022-06-24T17:40:59.862731+05:30 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.1.3; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem; socket.ssl_reload = 1;
2022-06-24T17:40:59.866960+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-24T17:40:59.867069+05:30 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 0b30b9d6-f3a7-11ec-a44d-67d27c33010c:1, protocol version: -1
2022-06-24T17:40:59.867162+05:30 0 [Note] [MY-000000] [WSREP] Starting replication
2022-06-24T17:40:59.867187+05:30 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0
2022-06-24T17:40:59.867208+05:30 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 0b30b9d6-f3a7-11ec-a44d-67d27c33010c:1
2022-06-24T17:40:59.867275+05:30 0 [Note] [MY-000000] [Galera] protonet asio version 0
2022-06-24T17:40:59.867522+05:30 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2022-06-24T17:40:59.867552+05:30 0 [Note] [MY-000000] [Galera] backend: asio
2022-06-24T17:40:59.867604+05:30 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2022-06-24T17:40:59.867686+05:30 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2022-06-24T17:40:59.867704+05:30 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2022-06-24T17:40:59.867860+05:30 0 [Note] [MY-000000] [Galera] GMCast version 0
2022-06-24T17:40:59.867982+05:30 0 [Note] [MY-000000] [Galera] (b5a43317-98da, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2022-06-24T17:40:59.868000+05:30 0 [Note] [MY-000000] [Galera] (b5a43317-98da, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2022-06-24T17:40:59.868215+05:30 0 [Note] [MY-000000] [Galera] EVS version 1
2022-06-24T17:40:59.868302+05:30 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'PR-pxc-cluster-u', peer '192.168.1.1:,192.168.1.2:,192.168.1.3:'
2022-06-24T17:40:59.871767+05:30 0 [Note] [MY-000000] [Galera] (b5a43317-98da, 'ssl://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address ssl://192.168.1.3:4567
2022-06-24T17:41:02.869772+05:30 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout)
2022-06-24T17:41:02.869895+05:30 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2022-06-24T17:41:02.869925+05:30 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2022-06-24T17:41:02.869966+05:30 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible
2022-06-24T17:41:02.869996+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,b5a43317-98da,1)
memb {
b5a43317-98da,0
}
joined {
}
left {
}
partitioned {
}
)
2022-06-24T17:41:03.370116+05:30 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50188S), skipping check
2022-06-24T17:41:32.876682+05:30 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-06-24T17:41:32.876732+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-06-24T17:41:32.877101+05:30 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():161
2022-06-24T17:41:32.877131+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)
2022-06-24T17:41:33.877370+05:30 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-06-24T17:41:33.877436+05:30 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-06-24T17:41:33.877559+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1758: Failed to open channel 'PR-pxc-cluster-u' at 'gcomm://192.168.1.1,192.168.1.2,192.168.1.3': -110 (Connection timed out)
2022-06-24T17:41:33.877586+05:30 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
2022-06-24T17:41:33.877607+05:30 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.1.1,192.168.1.2,192.168.1.3) failed to establish connection with cluster (reason: 7)
2022-06-24T17:41:33.877632+05:30 0 [ERROR] [MY-010119] [Server] Aborting
2022-06-24T17:41:33.877916+05:30 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.27-18.1) Percona XtraDB Cluster (GPL), Release rel18, Revision ac35177, WSREP version 26.4.3.
2022-06-24T17:41:33.878569+05:30 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
2022-06-24T17:41:33.878619+05:30 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2022-06-24T17:41:33.879581+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-06-24T17:41:33.880359+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-06-24T17:41:33.881108+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0
2022-06-24T17:41:33.881128+05:30 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
2022-06-24T17:41:33.881134+05:30 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
2022-06-24T17:41:33.881139+05:30 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
2022-06-24T17:41:33.881146+05:30 0 [Note] [MY-000000] [Galera] avg deps dist 0
2022-06-24T17:41:33.881152+05:30 0 [Note] [MY-000000] [Galera] avg cert interval 0
2022-06-24T17:41:33.881182+05:30 0 [Note] [MY-000000] [Galera] cert index size 0
2022-06-24T17:41:33.881234+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-06-24T17:41:33.881275+05:30 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
2022-06-24T17:41:33.881290+05:30 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2022-06-24T17:41:33.881395+05:30 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)
2022-06-24T17:41:33.882274+05:30 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...