Rolling updates - re-created node cannot connect

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

Running: pxc-db:1.12.0, pxc-operator:1.12.1. On RKE 2
I make changes to my Helm Chart for pxc-db, say update a configuration, the last node (say node 2 (which is node 3 actually, 0,1,2)) is shutdown and restarted… and it endlessly loops trying to connect to the rest of the cluster:

PXC (2) pod logs: (trimmed)

+ '[' -n pxc-db-pxc-unready ']'
++ get_primary
++ peer-list -on-start=/var/lib/mysql/get-pxc-state -service=pxc-db-pxc-unready
++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
++ sort
++ tail -1
+ is_primary_exists=node:10-42-189-146.pxc-db-pxc-unready.percona.svc.cluster.local:wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
+ is_manual_recovery
+ set +o xtrace
+ [[ -z node:10-42-189-146.pxc-db-pxc-unready.percona.svc.cluster.local:wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary ]]
+ [[ -z node:10-42-189-146.pxc-db-pxc-unready.percona.svc.cluster.local:wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary ]]
+ [[ -z node:10-42-189-146.pxc-db-pxc-unready.percona.svc.cluster.local:wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary ]]
+ test -e /opt/percona/hookscript/hook.sh
+ exec mysqld --wsrep_start_position=7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813

PXC (2) Logs pod:

{"log":"2023-03-09T01:08:03.590827Z 0 [Warning] [MY-011068] [Server] The syntax 'wsrep_slave_threads' is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.592074Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.592639Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.592649Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29-21.1) starting as process 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.594707Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/galera4/libgalera_smm.so initial position: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.936869Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/galera4/libgalera_smm.so'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.937513Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.12(04bfb95) by Codership Oy <info@codership.com> (modified by Percona <https://percona.com/>) loaded successfully.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.937554Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.937831Z 0 [Note] [MY-000000] [Galera] Found saved state: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813, safe_to_bootstrap: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.937937Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:\nVersion: 2\nUUID: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67\nSeqno: 518 - 813\nOffset: 98749352\nSynced: 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.937966Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67, offset: 98749352\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.938316Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.939251Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.939284Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 518-813\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.939311Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/133643056 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.939350Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (133643056/133643056 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.939367Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 84/380 locked buffers\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:03.939386Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 1997464/134217728\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.029172Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.42.42.71; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.peer_timeout = PT6S; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038272Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038374Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813, protocol version: -1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038453Z 0 [Note] [MY-000000] [WSREP] Starting replication\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038469Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038484Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038536Z 0 [Note] [MY-000000] [Galera] protonet asio version 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038583Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038598Z 0 [Note] [MY-000000] [Galera] backend: asio\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038686Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038784Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.038845Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.039031Z 0 [Note] [MY-000000] [Galera] GMCast version 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.041158Z 0 [Note] [MY-000000] [Galera] (d7f9e056-9d26, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.041188Z 0 [Note] [MY-000000] [Galera] (d7f9e056-9d26, 'tcp://0.0.0.0:4567') multicast: , ttl: 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.041386Z 0 [Note] [MY-000000] [Galera] EVS version 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:04.041445Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-db-pxc', peer 'pxc-db-pxc-0.pxc-db-pxc:,pxc-db-pxc-1.pxc-db-pxc:'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:07.042084Z 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:07.042196Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:07.042212Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:07.042245Z 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:07.042263Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node\nview (view_id(NON_PRIM,d7f9e056-9d26,1)\nmemb {\n\td7f9e056-9d26,0\n\t}\njoined {\n\t}\nleft {\n\t}\npartitioned {\n\t}\n)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:07.542376Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50094S), skipping check\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:10.042449Z 0 [Note] [MY-000000] [Galera] (d7f9e056-9d26, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.182.52:4567 timed out, no messages seen in PT6S, socket stats: rtt: 7289 rttvar: 3644 rto: 208000 lost: 0 last_data_recv: 5992 cwnd: 10 last_queued_since: 5993441541 last_delivered_since: 5993441541 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:10.042896Z 0 [Note] [MY-000000] [Galera] (d7f9e056-9d26, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.189.146:4567 timed out, no messages seen in PT6S, socket stats: rtt: 428 rttvar: 214 rto: 200000 lost: 0 last_data_recv: 6004 cwnd: 10 last_queued_since: 6000663762 last_delivered_since: 6000663762 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:17.543083Z 0 [Note] [MY-000000] [Galera] (d7f9e056-9d26, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.182.52:4567 timed out, no messages seen in PT6S, socket stats: rtt: 769 rttvar: 384 rto: 204000 lost: 0 last_data_recv: 6000 cwnd: 10 last_queued_since: 5999423274 last_delivered_since: 5999423274 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:23.543590Z 0 [Note] [MY-000000] [Galera] (d7f9e056-9d26, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.189.146:4567 timed out, no messages seen in PT6S, socket stats: rtt: 402 rttvar: 201 rto: 200000 lost: 0 last_data_recv: 6000 cwnd: 10 last_queued_since: 5999793079 last_delivered_since: 5999793079 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:29.544030Z 0 [Note] [MY-000000] [Galera] (d7f9e056-9d26, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.182.52:4567 timed out, no messages seen in PT6S, socket stats: rtt: 1090 rttvar: 545 rto: 204000 lost: 0 last_data_recv: 6000 cwnd: 10 last_queued_since: 5998980679 last_delivered_since: 5998980679 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:36.044506Z 0 [Note] [MY-000000] [Galera] (d7f9e056-9d26, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.189.146:4567 timed out, no messages seen in PT6S, socket stats: rtt: 3034 rttvar: 1517 rto: 204000 lost: 0 last_data_recv: 6368 cwnd: 10 last_queued_since: 6365767669 last_delivered_since: 6365767669 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:37.178063Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:37.178120Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node\nview ((empty))\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:37.178401Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)\n\t at gcomm/src/pc.cpp:connect():161\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:37.178419Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.178588Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.178640Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.178699Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'pxc-db-pxc' at 'gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc': -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.178713Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.178725Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc) failed to establish connection with cluster (reason: 7)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.178739Z 0 [ERROR] [MY-010119] [Server] Aborting\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.178983Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.29-21.1) Percona XtraDB Cluster (GPL), Release rel21, Revision 250bc93, WSREP version 26.4.3.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.179808Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.179844Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.182500Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.184100Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185599Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185617Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185623Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185627Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185637Z 0 [Note] [MY-000000] [Galera] avg deps dist 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185641Z 0 [Note] [MY-000000] [Galera] avg cert interval 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185646Z 0 [Note] [MY-000000] [Galera] cert index size 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185697Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185736Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185746Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.185835Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:08:38.191780Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.030938Z 0 [Warning] [MY-011068] [Server] The syntax 'wsrep_slave_threads' is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.031746Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.032294Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.032303Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29-21.1) starting as process 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.033873Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/galera4/libgalera_smm.so initial position: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.033944Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/galera4/libgalera_smm.so'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.035488Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.12(04bfb95) by Codership Oy <info@codership.com> (modified by Percona <https://percona.com/>) loaded successfully.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.035535Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.035765Z 0 [Note] [MY-000000] [Galera] Found saved state: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813, safe_to_bootstrap: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.035840Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:\nVersion: 2\nUUID: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67\nSeqno: 518 - 813\nOffset: 98749352\nSynced: 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.035859Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67, offset: 98749352\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.036081Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.036940Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.036975Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 518-813\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.036995Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/133643056 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.037018Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (133643056/133643056 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.037028Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 84/380 locked buffers\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.037038Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 1997464/134217728\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.041031Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.42.42.71; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.peer_timeout = PT6S; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047303Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047413Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813, protocol version: -1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047496Z 0 [Note] [MY-000000] [WSREP] Starting replication\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047511Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047524Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 7f6414d7-bde4-11ed-baa8-16ee5f1bae67:813\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047576Z 0 [Note] [MY-000000] [Galera] protonet asio version 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047627Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047641Z 0 [Note] [MY-000000] [Galera] backend: asio\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047731Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047871Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.047904Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.048080Z 0 [Note] [MY-000000] [Galera] GMCast version 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.050933Z 0 [Note] [MY-000000] [Galera] (24df189b-b30b, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.050967Z 0 [Note] [MY-000000] [Galera] (24df189b-b30b, 'tcp://0.0.0.0:4567') multicast: , ttl: 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.051143Z 0 [Note] [MY-000000] [Galera] EVS version 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:13.051200Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-db-pxc', peer 'pxc-db-pxc-0.pxc-db-pxc:,pxc-db-pxc-1.pxc-db-pxc:'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:16.051779Z 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:16.051883Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:16.051899Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:16.051923Z 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:16.051939Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node\nview (view_id(NON_PRIM,24df189b-b30b,1)\nmemb {\n\t24df189b-b30b,0\n\t}\njoined {\n\t}\nleft {\n\t}\npartitioned {\n\t}\n)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:16.552057Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50088S), skipping check\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:19.052105Z 0 [Note] [MY-000000] [Galera] (24df189b-b30b, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.182.52:4567 timed out, no messages seen in PT6S, socket stats: rtt: 529 rttvar: 264 rto: 200000 lost: 0 last_data_recv: 6000 cwnd: 10 last_queued_since: 6000151034 last_delivered_since: 6000151034 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:19.052337Z 0 [Note] [MY-000000] [Galera] (24df189b-b30b, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.189.146:4567 timed out, no messages seen in PT6S, socket stats: rtt: 550 rttvar: 275 rto: 200000 lost: 0 last_data_recv: 6000 cwnd: 10 last_queued_since: 6000343327 last_delivered_since: 6000343327 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:10:26.552688Z 0 [Note] [MY-000000] [Galera] (24df189b-b30b, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.42.182.52:4567 timed out, no messages seen in PT6S, socket stats: rtt: 455 rttvar: 227 rto: 200000 lost: 0 last_data_recv: 6000 cwnd: 10 last_queued_since: 5999829888 last_delivered_since: 5999829888 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}

And just an endess loop. The other nodes are primary…

(Interestingly, while typing this) I see the other 2 nodes are saying they can’t resolve each other:

PXC 0:(trimmed)

{"log":"2023-03-09T00:50:02.784222Z 1101 [Warning] [MY-010056] [Server] Host name 'pxc-db-proxysql-2.pxc-db-proxysql-unready.percona.svc.cluster.local' could not be resolved: Name or service not known\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T00:51:06.729422Z 1235 [Warning] [MY-010058] [Server] Hostname 'pxc-db-proxysql-1.pxc-db-proxysql-unready.percona.svc.cluster.local' does not resolve to '10.42.42.68'.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:09:10.062374Z 1182 [Warning] [MY-010056] [Server] Host name 'pxc-db-proxysql-0.pxc-db-proxysql-unready.percona.svc.cluster.local' could not be resolved: Name or service not known\n","file":"/var/lib/mysql/mysqld-error.log"}

PXC 1: (trimmed)

{"log":"2023-03-09T01:09:10.029619Z 472 [Warning] [MY-010056] [Server] Host name 'pxc-db-proxysql-0.pxc-db-proxysql-unready.percona.svc.cluster.local' could not be resolved: Name or service not known\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-03-09T01:09:10.029619Z 472 [Warning] [MY-010056] [Server] Host name 'pxc-db-proxysql-0.pxc-db-proxysql-unready.percona.svc.cluster.local' could not be resolved: Name or service not known\n","file":"/var/lib/mysql/mysqld-error.log"}

It seems the operator breaks something when trying to roll an update to the cluster… I haven’t had this problem with Bitnami’s MariaDB Galera Cluster…
I happened to also notice that EndpointSlice for pxc-db-pxc-s6jgg - the IP’s for each node doesn’t match… A “quick fix” for this breaking issue for rolling updates to config was set “RollingUpdate” for updateStrategy" and “redeploy” on the Statefulset myself. It doesn’t get messed up then. Annoyingly it doesn’t actually re-deploy ALL of them, so I have to delete the others myself.

2

I’ve set: disableTLS: true
And I’ve not seen this issue come up again…