Same issue here.
Deploying Percona PXC-Operator & PXC-DB (v1.15.1) with ArgoCD, Kustomize and the Helm-Chart.
values:
default values except backup.storages for minio and pxc.persistence for custom storageClass.
pxc-db-pxc-2 is getting into a CrashLoopBackOff after a while. pxc-db-pxc-0 & pxc-db-pxc-1 are fine.
pxc-db-pxc-0 & pxc-db-pxc-1 logs:
{"log":"2024-11-26T11:35:06.285631Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert unknown ca\n","file":"/var/lib/mysql/mysqld-error.log"}
pxc-db-pxc-2 logs:
{"log":"2024-11-26T11:34:37.719545Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.719562Z 0 [Warning] [MY-011070] [Server] 'binlog_format' is deprecated and will be removed in a future release.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.719575Z 0 [Warning] [MY-011068] [Server] The syntax 'wsrep_slave_threads' is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.721445Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.721950Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.36-28.1) starting as process 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.753314Z 0 [Warning] [MY-010068] [Server] CA certificate /etc/mysql/ssl-internal/ca.crt is self signed.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.753379Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.753394Z 0 [Note] [MY-000000] [WSREP] New joining cluster node configured to use specified SSL artifacts\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.753438Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/galera4/libgalera_smm.so initial position: c21fc1f6-ab45-11ef-a4f7-8a3a8ccbbe93:27\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.753461Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/galera4/libgalera_smm.so'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.754374Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.17(aa2461d) by Codership Oy <info@codership.com> (modified by Percona <https://percona.com/>) loaded successfully.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.754453Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.754684Z 0 [Warning] [MY-000000] [Galera] SSL compression is not effective. The option socket.ssl_compression is deprecated and will be removed in future releases.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.754701Z 0 [Warning] [MY-000000] [Galera] Parameter 'socket.ssl_compression' is deprecated and will be removed in future versions\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755220Z 0 [Note] [MY-000000] [Galera] Found saved state: c21fc1f6-ab45-11ef-a4f7-8a3a8ccbbe93:27, safe_to_bootstrap: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755346Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:\nVersion: 2\nUUID: c21fc1f6-ab45-11ef-a4f7-8a3a8ccbbe93\nSeqno: 22 - 27\nOffset: 1920\nSynced: 1\nEncVersion: 1\nEncrypted: 0\nMasterKeyConst UUID: 23f4cc5f-ab46-11ef-9ce2-625d193e566d\nMasterKey UUID: 00000000-0000-0000-0000-000000000000\nMasterKey ID: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755365Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: c21fc1f6-ab45-11ef-a4f7-8a3a8ccbbe93, offset: 1920\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755466Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755517Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755530Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 22-27\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755562Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/2576 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755595Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (2576/2576 bytes) complete.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755604Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 2/8 locked buffers\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.755613Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 134215792/134217728\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.758659Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 10.244.5.89; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 10; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 11; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = /etc/mysql/ssl-internal/ca.crt; socket.ssl_cert = /etc/mysql/ssl-internal/tls.crt; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = /etc/mysql/ssl-internal/tls.key; socket.ssl_reload = 1; \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.776124Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.776252Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: c21fc1f6-ab45-11ef-a4f7-8a3a8ccbbe93:27, protocol version: -1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.776344Z 0 [Note] [MY-000000] [WSREP] Starting replication\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.776365Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.776380Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to c21fc1f6-ab45-11ef-a4f7-8a3a8ccbbe93:27\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.776458Z 0 [Note] [MY-000000] [Galera] protonet asio version 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.777008Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.777047Z 0 [Note] [MY-000000] [Galera] backend: asio\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.777128Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.777229Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.777252Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.777445Z 0 [Note] [MY-000000] [Galera] GMCast version 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.779802Z 0 [Note] [MY-000000] [Galera] (6b021eca-8f10, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.779848Z 0 [Note] [MY-000000] [Galera] (6b021eca-8f10, 'ssl://0.0.0.0:4567') multicast: , ttl: 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.780111Z 0 [Note] [MY-000000] [Galera] EVS version 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:37.780197Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-db-pxc', peer 'pxc-db-pxc-0.pxc-db-pxc:,pxc-db-pxc-1.pxc-db-pxc:'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:38.780843Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:38.780923Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:38.780953Z 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:38.780985Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node\nview (view_id(NON_PRIM,6b021eca-8f10,1)\nmemb {\n\t6b021eca-8f10,0\n\t}\njoined {\n\t}\nleft {\n\t}\npartitioned {\n\t}\n)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:39.281154Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT1.50105S), skipping check\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:34:40.782159Z 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:10.792550Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:10.792618Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node\nview ((empty))\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:10.792705Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)\n\t at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcomm/src/pc.cpp:connect():176\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:10.792729Z 0 [ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.792904Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.792973Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.793087Z 0 [ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs.cpp:gcs_open():1880: Failed to open channel 'pxc-db-pxc' at 'gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc': -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.793109Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.793120Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc) failed to establish connection with cluster (reason: 7)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.793131Z 0 [ERROR] [MY-010119] [Server] Aborting\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.793418Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.36-28.1) Percona XtraDB Cluster (GPL), Release rel28, Revision bfb687f, WSREP version 26.1.4.3.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.794001Z 0 [ERROR] [MY-010065] [Server] Failed to shutdown components infrastructure.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.794135Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.794168Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.795787Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.797293Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.798816Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.798849Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.798867Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.798882Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.798898Z 0 [Note] [MY-000000] [Galera] avg deps dist 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.798908Z 0 [Note] [MY-000000] [Galera] avg cert interval 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.798916Z 0 [Note] [MY-000000] [Galera] cert index size 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.798995Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.799083Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.799102Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.799243Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T11:35:11.802495Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...\n","file":"/var/lib/mysql/mysqld-error.log"}