Percona-xtradb cluster in docker swarm timed out

vbvbrj · June 26, 2024, 10:35am

Hello.
Recently I created a stack with a docker-compose.yml to deploy in a docker swarm of 4 servers for percona-xtradb with etcd.
This is the docker-compose.yml:

# syntax=docker/dockerfile:1
version: '3.9'
services:

 percona_etcd:
  image: quay.io/coreos/etcd
  command:  etcd
  environment:
   - ETCD_DATA_DIR=/opt/etcd/data
   - ETCD_NAME=percona_etcd
   - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001
   - ETCD_ADVERTISE_CLIENT_URLS=http://percona_etcd:2379,http://percona_etcd:4001
   - ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
   - ETCD_INITIAL_ADVERTISE_PEER_URLS=http://percona_etcd:2380
   - ETCD-INITIAL-CLUSTER=etcd0=http://percona_etcd:2380
   - ETCD_INITIAL_CLUSTER_STATE=new
   - ETCD_INITIAL_CLUSTER_TOKEN=percona-etcd-cluster
  networks:
   - overlay
  deploy:
   mode: replicated
   replicas: 1
  volumes:
   - percona_etcd:/opt/etcd/data

 percona:
  image: percona/percona-xtradb-cluster:8.0
  networks:
   - overlay
  environment:
   - DISCOVERY_SERVICE=tasks.percona_etcd:2379
   - CLUSTER_NAME=percona
   - MYSQL_ALLOW_EMPTY_PASSWORD='true'
  cap_add:
   - SYS_NICE
  deploy:
   mode: replicated
   replicas: 3
   endpoint_mode: vip
   placement:
    max_replicas_per_node: 1
    constraints:
     - node.labels.percona == true
  volumes:
   - percona_my_cnf_d:/etc/my.cnf.d
   - percona_data:/var/lib/mysql
   - percona_log:/var/log/mysql

networks:
 overlay:
  external: true
volumes:
 percona_my_cnf_d:
  driver: glusterfs
 percona_etcd:
  driver: glusterfs
 percona_log:
  driver: glusterfs
 percona_data:
  driver: local
  driver_opts:
   o: bind
   type: none
   device: /data/percona

When first I deployed the cluster did go up and connection to percona worked as should with replication.
Second day after a stack is stopped and started, the pecona-xtradb services does not get in contact with each other. I just receive in logs following:

percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:34:44.837454+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48332160 cwnd: 1 last_queued_since: 48633226476207 last_delivered_since: 48633226476207 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:34:49.338012+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48336660 cwnd: 1 last_queued_since: 48637727024505 last_delivered_since: 48637727024505 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:34:53.838506+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48341160 cwnd: 1 last_queued_since: 48642227528558 last_delivered_since: 48642227528558 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:34:57.839023+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48345160 cwnd: 1 last_queued_since: 48646228029586 last_delivered_since: 48646228029586 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:35:02.339742+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48349664 cwnd: 1 last_queued_since: 48650728755275 last_delivered_since: 48650728755275 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:35:06.840280+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48354164 cwnd: 1 last_queued_since: 48655229290789 last_delivered_since: 48655229290789 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:35:10.840851+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48358164 cwnd: 1 last_queued_since: 48659229864294 last_delivered_since: 48659229864294 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:35:15.341442+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48362664 cwnd: 1 last_queued_since: 48663730459120 last_delivered_since: 48663730459120 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:35:19.841996+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48367164 cwnd: 1 last_queued_since: 48668231010175 last_delivered_since: 48668231010175 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:35:24.342706+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48371664 cwnd: 1 last_queued_since: 48672731720374 last_delivered_since: 48672731720374 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.1.1qtirby8sooe@5    | 2024-06-26T09:35:26.593500+03:00 0 [Note] [MY-000000] [Galera] (e6fe01a0-a37f, 'tcp://0.0.0.0:4567') reconnecting to 97a651ce-b4b7 (tcp://10.110.1.216:4567), attempt 600
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:35:28.343259+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48375668 cwnd: 1 last_queued_since: 48676732282195 last_delivered_since: 48676732282195 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.3.4nfwet49xkaf@6    | 2024-06-26T09:35:32.843742+03:00 0 [Note] [MY-000000] [Galera] (97a651ce-b4b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.8:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 48380168 cwnd: 1 last_queued_since: 48681232755452 last_delivered_since: 48681232755452 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.1.1qtirby8sooe@5    | 2024-06-26T09:36:59.111454+03:00 0 [Note] [MY-000000] [Galera] (e6fe01a0-a37f, 'tcp://0.0.0.0:4567') reconnecting to 97a651ce-b4b7 (tcp://10.110.1.216:4567), attempt 630

I can’t figure out why it worked first day and now percona services does not find each other anymore.
Can some one help me resolve this issue?

Blockquote

vbvbrj · June 26, 2024, 12:21pm

In case it is relevant:

percona_percona.1.jc7t3d4h8qp2@5    | Assigning ef7dae57-33b1-11ef-8fa0-b2e51da68967:27 to wsrep_start_position
percona_percona.1.jc7t3d4h8qp2@5    | + echo 'Skipping wsrep-recover for ef7dae57-33b1-11ef-8fa0-b2e51da68967:27 pair'
percona_percona.1.jc7t3d4h8qp2@5    | + echo 'Assigning ef7dae57-33b1-11ef-8fa0-b2e51da68967:27 to wsrep_start_position'
percona_percona.1.jc7t3d4h8qp2@5    | + wsrep_start_position_opt=--wsrep_start_position=ef7dae57-33b1-11ef-8fa0-b2e51da68967:27
percona_percona.1.jc7t3d4h8qp2@5    | + '[' -z --wsrep_start_position=ef7dae57-33b1-11ef-8fa0-b2e51da68967:27 -a -d /var/lib/mysql//mysql ']'
percona_percona.1.jc7t3d4h8qp2@5    | + /call-home.sh -f PRODUCT_FAMILY_PXC -v 8.0.36-28-1 -d DOCKER -s el8 -t 7 -c 2
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689063+03:00 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
percona_percona.2.zywef0zhu307@4    | view (view_id(NON_PRIM,c78ce0a8-b0dd,13)
percona_percona.2.zywef0zhu307@4    | memb {
percona_percona.2.zywef0zhu307@4    |         edbc8a03-8172,0
percona_percona.2.zywef0zhu307@4    |         }
percona_percona.2.zywef0zhu307@4    | joined {
percona_percona.2.zywef0zhu307@4    |         }
percona_percona.2.zywef0zhu307@4    | left {
percona_percona.2.zywef0zhu307@4    |         }
percona_percona.2.zywef0zhu307@4    | partitioned {
percona_percona.2.zywef0zhu307@4    |         b0d06926-aa8b,0
percona_percona.2.zywef0zhu307@4    |         c78ce0a8-b0dd,0
percona_percona.2.zywef0zhu307@4    |         }
percona_percona.2.zywef0zhu307@4    | )
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689204+03:00 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689210+03:00 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689241+03:00 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
percona_percona.2.zywef0zhu307@4    | view (view_id(NON_PRIM,edbc8a03-8172,14)
percona_percona.2.zywef0zhu307@4    | memb {
percona_percona.2.zywef0zhu307@4    |         edbc8a03-8172,0
percona_percona.2.zywef0zhu307@4    |         }
percona_percona.2.zywef0zhu307@4    | joined {
percona_percona.2.zywef0zhu307@4    |         }
percona_percona.2.zywef0zhu307@4    | left {
percona_percona.2.zywef0zhu307@4    |         }
percona_percona.2.zywef0zhu307@4    | partitioned {
percona_percona.2.zywef0zhu307@4    |         b0d06926-aa8b,0
percona_percona.2.zywef0zhu307@4    |         c78ce0a8-b0dd,0
percona_percona.2.zywef0zhu307@4    |         }
percona_percona.2.zywef0zhu307@4    | )
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689348+03:00 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689381+03:00 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689424+03:00 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689503+03:00 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689533+03:00 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689525+03:00 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 27 upto current CC event 27 upto:27
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689613+03:00 1 [Note] [MY-000000] [Galera] Drain monitors from 27 up to 27
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689655+03:00 1 [Note] [MY-000000] [Galera] ================================================
percona_percona.2.zywef0zhu307@4    | View:
percona_percona.2.zywef0zhu307@4    |   id: ef7dae57-33b1-11ef-8fa0-b2e51da68967:27
percona_percona.2.zywef0zhu307@4    |   status: non-primary
percona_percona.2.zywef0zhu307@4    |   protocol_version: -1
percona_percona.2.zywef0zhu307@4    |   capabilities:
percona_percona.2.zywef0zhu307@4    |   final: no
percona_percona.2.zywef0zhu307@4    |   own_index: 0
percona_percona.2.zywef0zhu307@4    |   members(1):
percona_percona.2.zywef0zhu307@4    |         0: edbc8a03-33b1-11ef-8172-5e75660da239, 9bfd5be17f76
percona_percona.2.zywef0zhu307@4    | =================================================
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689674+03:00 1 [Note] [MY-000000] [Galera] Non-primary view
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689693+03:00 1 [Note] [MY-000000] [WSREP] Server status change connected -> connected
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689726+03:00 1 [Note] [MY-000000] [WSREP] Detected Protocol version: -1 Changing pxc_maint_mode to MAINTENANCE.
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689755+03:00 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689799+03:00 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 27 upto current CC event 27 upto:27
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689825+03:00 1 [Note] [MY-000000] [Galera] Drain monitors from 27 up to 27
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689854+03:00 1 [Note] [MY-000000] [Galera] ================================================
percona_percona.2.zywef0zhu307@4    | View:
percona_percona.2.zywef0zhu307@4    |   id: ef7dae57-33b1-11ef-8fa0-b2e51da68967:27
percona_percona.2.zywef0zhu307@4    |   status: non-primary
percona_percona.2.zywef0zhu307@4    |   protocol_version: -1
percona_percona.2.zywef0zhu307@4    |   capabilities:
percona_percona.2.zywef0zhu307@4    |   final: no
percona_percona.2.zywef0zhu307@4    |   own_index: 0
percona_percona.2.zywef0zhu307@4    |   members(1):
percona_percona.2.zywef0zhu307@4    |         0: edbc8a03-33b1-11ef-8172-5e75660da239, 9bfd5be17f76
percona_percona.2.zywef0zhu307@4    | =================================================
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689872+03:00 1 [Note] [MY-000000] [Galera] Non-primary view
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689909+03:00 1 [Note] [MY-000000] [WSREP] Server status change connected -> connected
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689932+03:00 1 [Note] [MY-000000] [WSREP] Detected Protocol version: -1 Changing pxc_maint_mode to MAINTENANCE.
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.689958+03:00 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:06.869576+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.229:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68853572 cwnd: 1 last_queued_since: 69154215028316 last_delivered_since: 69154215028316 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.1.jc7t3d4h8qp2@5    | + exec mysqld --wsrep_start_position=ef7dae57-33b1-11ef-8fa0-b2e51da68967:27
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.024421+03:00 0 [Note] [MY-013667] [Server] Error-log destination "stderr" is not a file. Can not restore error log messages from previous run.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.016093+03:00 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.016112+03:00 0 [Warning] [MY-011070] [Server] 'binlog_format' is deprecated and will be removed in a future release.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.016141+03:00 0 [Warning] [MY-011068] [Server] The syntax 'wsrep_slave_threads' is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.018283+03:00 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.019549+03:00 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.019744+03:00 0 [Note] [MY-013932] [Server] BuildID[sha1]=8849a9bea52d6d7129f6c4910d67e380da94c3ed
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.019753+03:00 0 [Note] [MY-010949] [Server] Basedir set to /usr/.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.019765+03:00 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.36-28.1) starting as process 1
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.025585+03:00 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/galera4/libgalera_smm.so initial position: ef7dae57-33b1-11ef-8fa0-b2e51da68967:27
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.025681+03:00 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/galera4/libgalera_smm.so'
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.030029+03:00 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.17(aa2461d) by Codership Oy <info@codership.com> (modified by Percona <https://percona.com/>) loaded successfully.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.030160+03:00 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.032351+03:00 0 [Note] [MY-000000] [Galera] Found saved state: ef7dae57-33b1-11ef-8fa0-b2e51da68967:27, safe_to_bootstrap: 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.032728+03:00 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
percona_percona.1.jc7t3d4h8qp2@5    | Version: 2
percona_percona.1.jc7t3d4h8qp2@5    | UUID: ef7dae57-33b1-11ef-8fa0-b2e51da68967
percona_percona.1.jc7t3d4h8qp2@5    | Seqno: 1 - 27
percona_percona.1.jc7t3d4h8qp2@5    | Offset: 1280
percona_percona.1.jc7t3d4h8qp2@5    | Synced: 1
percona_percona.1.jc7t3d4h8qp2@5    | EncVersion: 1
percona_percona.1.jc7t3d4h8qp2@5    | Encrypted: 0
percona_percona.1.jc7t3d4h8qp2@5    | MasterKeyConst UUID: ef7a2891-33b1-11ef-a4c5-e2f46bff7fde
percona_percona.1.jc7t3d4h8qp2@5    | MasterKey UUID: 00000000-0000-0000-0000-000000000000
percona_percona.1.jc7t3d4h8qp2@5    | MasterKey ID: 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.032753+03:00 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: ef7dae57-33b1-11ef-8fa0-b2e51da68967, offset: 1280
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.033521+03:00 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.033673+03:00 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.033703+03:00 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-27
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.033835+03:00 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...  0.0% (      0/4076528 bytes) complete.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.033901+03:00 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (4076528/4076528 bytes) complete.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.033928+03:00 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 4/31 locked buffers
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.033952+03:00 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 130141976/134217728
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.036593+03:00 0 [Note] [MY-000000] [Galera] Passing config to GCS: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 10.110.1.237; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 11; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto;
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.049894+03:00 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.050045+03:00 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: ef7dae57-33b1-11ef-8fa0-b2e51da68967:27, protocol version: -1
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.050400+03:00 0 [Note] [MY-000000] [WSREP] Starting replication
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.050429+03:00 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.050489+03:00 0 [Note] [MY-000000] [Galera] Setting GCS initial position to ef7dae57-33b1-11ef-8fa0-b2e51da68967:27
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.050689+03:00 0 [Note] [MY-000000] [Galera] protonet asio version 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.050823+03:00 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.050879+03:00 0 [Note] [MY-000000] [Galera] backend: asio
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.051105+03:00 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.051541+03:00 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.051566+03:00 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.051978+03:00 0 [Note] [MY-000000] [Galera] GMCast version 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.052387+03:00 0 [Note] [MY-000000] [Galera] (de2016a7-a4dd, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.052415+03:00 0 [Note] [MY-000000] [Galera] (de2016a7-a4dd, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.053607+03:00 0 [Note] [MY-000000] [Galera] EVS version 1
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:08.053894+03:00 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'percona', peer '10.110.1.228:'
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:09.055605+03:00 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:09.055722+03:00 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:09.055996+03:00 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:09.056100+03:00 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
percona_percona.1.jc7t3d4h8qp2@5    | view (view_id(NON_PRIM,de2016a7-a4dd,1)
percona_percona.1.jc7t3d4h8qp2@5    | memb {
percona_percona.1.jc7t3d4h8qp2@5    |         de2016a7-a4dd,0
percona_percona.1.jc7t3d4h8qp2@5    |         }
percona_percona.1.jc7t3d4h8qp2@5    | joined {
percona_percona.1.jc7t3d4h8qp2@5    |         }
percona_percona.1.jc7t3d4h8qp2@5    | left {
percona_percona.1.jc7t3d4h8qp2@5    |         }
percona_percona.1.jc7t3d4h8qp2@5    | partitioned {
percona_percona.1.jc7t3d4h8qp2@5    |         }
percona_percona.1.jc7t3d4h8qp2@5    | )
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:09.556371+03:00 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT1.50286S), skipping check
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:10.370208+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68857072 cwnd: 1 last_queued_since: 69157715676565 last_delivered_since: 69157715676565 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:10.870322+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.229:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68857572 cwnd: 1 last_queued_since: 69158215786557 last_delivered_since: 69158215786557 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:11.057007+03:00 0 [Note] [MY-000000] [Galera] (de2016a7-a4dd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 69342044 cwnd: 1 last_queued_since: 69642697622679 last_delivered_since: 69642697622679 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:11.057386+03:00 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:14.370832+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68861072 cwnd: 1 last_queued_since: 69161716301567 last_delivered_since: 69161716301567 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:14.870955+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.229:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68861572 cwnd: 1 last_queued_since: 69162216420597 last_delivered_since: 69162216420597 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:18.371639+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68865076 cwnd: 1 last_queued_since: 69165717105489 last_delivered_since: 69165717105489 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:18.871770+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.229:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68865576 cwnd: 1 last_queued_since: 69166217225505 last_delivered_since: 69166217225505 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:22.372319+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68869076 cwnd: 1 last_queued_since: 69169717763371 last_delivered_since: 69169717763371 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:26.373049+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.227:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68873076 cwnd: 1 last_queued_since: 69173718510310 last_delivered_since: 69173718510310 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:26.873130+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 4000000 lost: 1 last_data_recv: 68873576 cwnd: 1 last_queued_since: 69174218596749 last_delivered_since: 69174218596749 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:30.373762+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.227:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68877076 cwnd: 1 last_queued_since: 69177719220656 last_delivered_since: 69177719220656 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:30.873892+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68877576 cwnd: 1 last_queued_since: 69178219359311 last_delivered_since: 69178219359311 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:34.874548+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68881576 cwnd: 1 last_queued_since: 69182220002193 last_delivered_since: 69182220002193 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:38.375201+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.227:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68885076 cwnd: 1 last_queued_since: 69185720669781 last_delivered_since: 69185720669781 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:38.875404+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68885576 cwnd: 1 last_queued_since: 69186220826060 last_delivered_since: 69186220826060 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:41.063195+03:00 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:41.063266+03:00 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
percona_percona.1.jc7t3d4h8qp2@5    | view ((empty))
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:41.064569+03:00 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)
percona_percona.1.jc7t3d4h8qp2@5    |          at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcomm/src/pc.cpp:connect():176
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:41.064608+03:00 0 [ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.066059+03:00 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.066127+03:00 0 [Note] [MY-000000] [Galera] gcomm: joining thread
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.066844+03:00 0 [ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs.cpp:gcs_open():1880: Failed to open channel 'percona' at 'gcomm://10.110.1.228': -110 (Connection timed out)
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.066881+03:00 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.066904+03:00 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://10.110.1.228) failed to establish connection with cluster (reason: 7)
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.066923+03:00 0 [ERROR] [MY-010119] [Server] Aborting
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.067017+03:00 0 [Note] [MY-010120] [Server] Binlog end
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.067169+03:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'daemon_keyring_proxy_plugin'
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.067353+03:00 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.36-28.1)  Percona XtraDB Cluster (GPL), Release rel28, Revision bfb687f, WSREP version 26.1.4.3.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.069343+03:00 0 [ERROR] [MY-010065] [Server] Failed to shutdown components infrastructure.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.069879+03:00 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.070260+03:00 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.073726+03:00 0 [Note] [MY-000000] [Galera] apply mon: entered 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.077172+03:00 0 [Note] [MY-000000] [Galera] apply mon: entered 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.080705+03:00 0 [Note] [MY-000000] [Galera] apply mon: entered 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081069+03:00 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081106+03:00 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081133+03:00 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081161+03:00 0 [Note] [MY-000000] [Galera] avg deps dist 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081180+03:00 0 [Note] [MY-000000] [Galera] avg cert interval 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081237+03:00 0 [Note] [MY-000000] [Galera] cert index size 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081368+03:00 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081430+03:00 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081767+03:00 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.081951+03:00 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)
percona_percona.1.jc7t3d4h8qp2@5    | 2024-06-26T15:16:42.084365+03:00 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:42.376012+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.227:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68889080 cwnd: 1 last_queued_since: 69189721462061 last_delivered_since: 69189721462061 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
percona_percona.2.zywef0zhu307@4    | 2024-06-26T15:16:42.876093+03:00 0 [Note] [MY-000000] [Galera] (edbc8a03-8172, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.110.1.228:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 68889580 cwnd: 1 last_queued_since: 69190221559906 last_delivered_since: 69190221559906 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)

przemek · June 28, 2024, 2:29pm

Hello,

Apparently, after the full stack restart, no node has been used to bootstrap the cluster. The logs show this node tried to join the cluster, but there was no primary component yet:

2024-06-26T15:16:09.055996+03:00 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible
2024-06-26T15:16:09.056100+03:00 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
 view (view_id(NON_PRIM,de2016a7-a4dd,1)
 memb {
         de2016a7-a4dd,0
         }
 joined {
         }
 left {
         }
 partitioned {
         }
 )

2024-06-26T15:16:41.064569+03:00 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)

After the full stop or all node crash, the most advanced member should be used to bootstrap the cluster before others can join.

vbvbrj · July 1, 2024, 5:02am

As the description of percona-xtradb-cluster says:

If you want to use the discovery service (right now only etcd is supported), set the address to DISCOVERY_SERVICE. The image will automatically find a running cluser by CLUSTER_NAME and join to the existing cluster (or start a new one).

Also, from percona documentation:

In recent Galera versions, the option pc.recovery (enabled by default) saves the cluster state into a file named gvwstate.dat on each member node.
Thanks to this new feature, the nodes will try to restore the primary component once all the members start to see each other.

But the nodes does not see each other and restarts continuously. How to make the nodes see each other and start function?

Topic		Replies	Views
problem trying to built pxc xtradb cluster 5.6 on docker Percona XtraDB Cluster 5.x	4	970	March 21, 2017
How Can I Running PXC_MySQL in a docker swarm mode by docker stack deploy Percona XtraDB Cluster 5.x mysql	5	1259	June 27, 2023
"Percona XtraDB Cluster in a Docker Container" not working Percona XtraDB Cluster 8.x	1	686	October 10, 2023
Percona XtraDB Cluster 5.7.18-29.20 is now available Percona XtraDB Cluster 5.x	0	413	June 3, 2017
XtraDB Docker Cluster Recovery From Replica Percona XtraDB Cluster 5.x	0	447	August 24, 2016

Percona-xtradb cluster in docker swarm timed out

Related topics