root@pxc-2:~# tail -150 /var/log/mysql/error.log 2020-10-28T13:28:55.343813+05:30 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown 2020-10-28T13:28:55.343922+05:30 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown 2020-10-28T13:28:55.344184+05:30 0 [Note] [MY-000000] [Galera] GMCast version 0 2020-10-28T13:28:55.344482+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2020-10-28T13:28:55.344595+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2020-10-28T13:28:55.344973+05:30 0 [Note] [MY-000000] [Galera] EVS version 1 2020-10-28T13:28:55.345162+05:30 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-cluster', peer '192.168.0.1:,192.168.0.2:,192.168.0.3:' 2020-10-28T13:28:55.347453+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://192.168.0.2:4567 2020-10-28T13:28:58.348139+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T13:28:58.348532+05:30 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout) 2020-10-28T13:28:58.348715+05:30 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1 2020-10-28T13:28:58.348914+05:30 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1 2020-10-28T13:28:58.349064+05:30 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible 2020-10-28T13:28:58.349213+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view (view_id(NON_PRIM,6d38082f,1) memb { 6d38082f,0 } joined { } left { } partitioned { } ) 2020-10-28T13:28:58.849512+05:30 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50453S), skipping check 2020-10-28T13:29:02.349399+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T13:29:06.350631+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T13:29:10.851998+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T13:29:14.853117+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T13:29:19.354205+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T13:29:23.855416+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T13:29:28.356665+05:30 0 [Note] [MY-000000] [Galera] (6d38082f, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T13:29:28.360534+05:30 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0 2020-10-28T13:29:28.360719+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view ((empty)) 2020-10-28T13:29:28.361214+05:30 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out) at gcomm/src/pc.cpp:connect():159 2020-10-28T13:29:28.361240+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out) 2020-10-28T13:29:28.361290+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1700: Failed to open channel 'pxc-cluster' at 'gcomm://192.168.0.1,192.168.0.2,192.168.0.3': -110 (Connection timed out) 2020-10-28T13:29:28.361312+05:30 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out 2020-10-28T13:29:28.361330+05:30 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.0.1,192.168.0.2,192.168.0.3) failed to establish connection with cluster (reason: 7) 2020-10-28T13:29:28.361348+05:30 0 [ERROR] [MY-010119] [Server] Aborting 2020-10-28T13:29:28.361510+05:30 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.20-11.2) Percona XtraDB Cluster (GPL), Release rel11, Revision 9132e55, WSREP version 26.4.3. 2020-10-28T13:29:28.361801+05:30 0 [Note] [MY-000000] [Galera] dtor state: CLOSED 2020-10-28T13:29:28.361833+05:30 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2020-10-28T13:29:28.364122+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2020-10-28T13:29:28.366339+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2020-10-28T13:29:28.368544+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2020-10-28T13:29:28.368564+05:30 0 [Note] [MY-000000] [Galera] cert index usage at exit 0 2020-10-28T13:29:28.368576+05:30 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0 2020-10-28T13:29:28.368585+05:30 0 [Note] [MY-000000] [Galera] deps set usage at exit 0 2020-10-28T13:29:28.368596+05:30 0 [Note] [MY-000000] [Galera] avg deps dist 0 2020-10-28T13:29:28.368605+05:30 0 [Note] [MY-000000] [Galera] avg cert interval 0 2020-10-28T13:29:28.368613+05:30 0 [Note] [MY-000000] [Galera] cert index size 0 2020-10-28T13:29:28.368650+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed. 2020-10-28T13:29:28.368676+05:30 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0 2020-10-28T13:29:28.368690+05:30 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2020-10-28T13:29:28.369495+05:30 0 [Note] [MY-000000] [Galera] Flushing memory map to disk... INFO: Skipping wsrep-recover for 3788d9c7-141e-11eb-969b-da352344b906:9 pair INFO: Assigning 3788d9c7-141e-11eb-969b-da352344b906:9 to wsrep_start_position 2020-10-28T14:06:46.200801+05:30 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11.2) starting as process 30068 2020-10-28T14:06:46.203149+05:30 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib/galera4/libgalera_smm.so initial position: 3788d9c7-141e-11eb-969b-da352344b906:9 2020-10-28T14:06:46.203200+05:30 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib/galera4/libgalera_smm.so' 2020-10-28T14:06:46.205228+05:30 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy loaded successfully. 2020-10-28T14:06:46.205281+05:30 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration. 2020-10-28T14:06:46.205577+05:30 0 [Note] [MY-000000] [Galera] Found saved state: 3788d9c7-141e-11eb-969b-da352344b906:9, safe_to_bootstrap: 1 2020-10-28T14:06:46.205719+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble: Version: 2 UUID: 3788d9c7-141e-11eb-969b-da352344b906 Seqno: 1 - 9 Offset: 1280 Synced: 1 2020-10-28T14:06:46.205767+05:30 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 3788d9c7-141e-11eb-969b-da352344b906, offset: 1280 2020-10-28T14:06:46.205883+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete. 2020-10-28T14:06:46.205949+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete. 2020-10-28T14:06:46.206011+05:30 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-9 2020-10-28T14:06:46.206086+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/1608 bytes) complete. 2020-10-28T14:06:46.206145+05:30 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (1608/1608 bytes) complete. 2020-10-28T14:06:46.206215+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/9 locked buffers 2020-10-28T14:06:46.206273+05:30 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 0/134217728 2020-10-28T14:06:46.207919+05:30 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.0.2; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; 2020-10-28T14:06:46.224304+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed. 2020-10-28T14:06:46.224436+05:30 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 3788d9c7-141e-11eb-969b-da352344b906:9, protocol version: -1 2020-10-28T14:06:46.224549+05:30 0 [Note] [MY-000000] [WSREP] Starting replication 2020-10-28T14:06:46.224640+05:30 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0 2020-10-28T14:06:46.224745+05:30 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 3788d9c7-141e-11eb-969b-da352344b906:9 2020-10-28T14:06:46.224889+05:30 0 [Note] [MY-000000] [Galera] protonet asio version 0 2020-10-28T14:06:46.225070+05:30 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums. 2020-10-28T14:06:46.225195+05:30 0 [Note] [MY-000000] [Galera] backend: asio 2020-10-28T14:06:46.225335+05:30 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 2020-10-28T14:06:46.225536+05:30 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown 2020-10-28T14:06:46.225669+05:30 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown 2020-10-28T14:06:46.225909+05:30 0 [Note] [MY-000000] [Galera] GMCast version 0 2020-10-28T14:06:46.226150+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2020-10-28T14:06:46.226253+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2020-10-28T14:06:46.226659+05:30 0 [Note] [MY-000000] [Galera] EVS version 1 2020-10-28T14:06:46.226857+05:30 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-cluster', peer '192.168.0.1:,192.168.0.2:,192.168.0.3:' 2020-10-28T14:06:46.229141+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://192.168.0.2:4567 2020-10-28T14:06:49.230006+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T14:06:49.230439+05:30 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout) 2020-10-28T14:06:49.230631+05:30 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1 2020-10-28T14:06:49.230840+05:30 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1 2020-10-28T14:06:49.231048+05:30 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible 2020-10-28T14:06:49.231185+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view (view_id(NON_PRIM,b6c51619,1) memb { b6c51619,0 } joined { } left { } partitioned { } ) 2020-10-28T14:06:49.731503+05:30 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50485S), skipping check 2020-10-28T14:06:53.731332+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T14:06:57.732538+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T14:07:02.233400+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T14:07:06.234355+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T14:07:10.735582+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T14:07:15.236430+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T14:07:19.237523+05:30 0 [Note] [MY-000000] [Galera] (b6c51619, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.0.1:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2020-10-28T14:07:19.244060+05:30 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0 2020-10-28T14:07:19.244249+05:30 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view ((empty)) 2020-10-28T14:07:19.244738+05:30 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out) at gcomm/src/pc.cpp:connect():159 2020-10-28T14:07:19.244765+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out) 2020-10-28T14:07:19.244823+05:30 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1700: Failed to open channel 'pxc-cluster' at 'gcomm://192.168.0.1,192.168.0.2,192.168.0.3': -110 (Connection timed out) 2020-10-28T14:07:19.244844+05:30 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out 2020-10-28T14:07:19.244862+05:30 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.0.1,192.168.0.2,192.168.0.3) failed to establish connection with cluster (reason: 7) 2020-10-28T14:07:19.244881+05:30 0 [ERROR] [MY-010119] [Server] Aborting 2020-10-28T14:07:19.245053+05:30 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.20-11.2) Percona XtraDB Cluster (GPL), Release rel11, Revision 9132e55, WSREP version 26.4.3. 2020-10-28T14:07:19.245356+05:30 0 [Note] [MY-000000] [Galera] dtor state: CLOSED 2020-10-28T14:07:19.245393+05:30 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2020-10-28T14:07:19.247629+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2020-10-28T14:07:19.249831+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2020-10-28T14:07:19.252093+05:30 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2020-10-28T14:07:19.252114+05:30 0 [Note] [MY-000000] [Galera] cert index usage at exit 0 2020-10-28T14:07:19.252124+05:30 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0 2020-10-28T14:07:19.252132+05:30 0 [Note] [MY-000000] [Galera] deps set usage at exit 0 2020-10-28T14:07:19.252145+05:30 0 [Note] [MY-000000] [Galera] avg deps dist 0 2020-10-28T14:07:19.252155+05:30 0 [Note] [MY-000000] [Galera] avg cert interval 0 2020-10-28T14:07:19.252164+05:30 0 [Note] [MY-000000] [Galera] cert index size 0 2020-10-28T14:07:19.252204+05:30 0 [Note] [MY-000000] [Galera] Service thread queue flushed. 2020-10-28T14:07:19.252232+05:30 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0 2020-10-28T14:07:19.252246+05:30 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2020-10-28T14:07:19.253009+05:30 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...