K8s XtraDB PXC Cluster, Third node fail to start

jw12314 · March 3, 2024, 12:13am

Hi.
I have a problem to start a cluster,
third node always fail to start

kubectl get pods -n pxctest -o wide
NAME                                               READY   STATUS    RESTARTS        AGE   IP              NODE    NOMINATED NODE   READINESS GATES
percona-xtradb-cluster-operator-55db8df4c9-4rpxp   1/1     Running   0               86m   10.182.139.23   node6   <none>           <none>
pxctest-haproxy-0                       3/3     Running   0               27m   10.182.139.29   node6   <none>           <none>
pxctest-haproxy-1                       3/3     Running   0               25m   10.182.33.154   node5   <none>           <none>
pxctest-haproxy-2                       3/3     Running   0               25m   10.182.47.52    node4   <none>           <none>
pxctest-pxc-0                           4/4     Running   1 (27m ago)     27m   10.182.139.30   node6   <none>           <none>
pxctest-pxc-1                           4/4     Running   0               26m   10.182.33.153   node5   <none>           <none>
pxctest-pxc-2                           3/4     Running   1 (9m24s ago)   25m   10.182.47.54    node4   <none>           <none>

No idea how to approach this. Please help.

EDIT:
I tried operator v1.13.0 and also master branch, same issue.

When I attempt to recreate this cluster, it schedules pods on different nodes, but it fails in the same way regardless of which node the last pod is scheduled to. So the first two pods are always working well on all nodes, the third one is always failing on all nodes.

Debian 12 is the system installed on all nodes with ZFS root
and the PVC provisioner is a openebs-zfspv, tested on both zfs and ext4 fstype flag
also tested with no provisioner (hostPath) same exact issue, third node refuse to start.

I literally tried everything to make it work, looking for help here is my last hope.

Tomislav_Plavcic · March 3, 2024, 4:11pm

Hi @jw12314 !
Which container is stuck on pod pxctest-pxc-2 (I guess pxc, but just in case)?
Can you share some logs:

kubectl describe pod pxctest-pxc-2
kubectl logs pxctest-pxc-2 -clogs
kubectl logs pxctest-pxc-2 -cpxc
kubectl get events
kubectl logs percona-xtradb-cluster-operator-55db8df4c9-4rpxp

also if possible share your cr.yaml

jw12314 · March 3, 2024, 5:07pm

All containers seems to be working, I can shell exec into all of them, but mysql is not starting.

# kubectl describe pod pxctest-pxc-2 -n pxctest
Name:             pxctest-pxc-2
Namespace:        pxctest
Priority:         0
Service Account:  default
Node:             node4/REDACTED
Start Time:       Sun, 03 Mar 2024 18:44:21 +0100
Labels:           app.kubernetes.io/component=pxc
                  app.kubernetes.io/instance=pxctest
                  app.kubernetes.io/managed-by=percona-xtradb-cluster-operator
                  app.kubernetes.io/name=percona-xtradb-cluster
                  app.kubernetes.io/part-of=percona-xtradb-cluster
                  apps.kubernetes.io/pod-index=2
                  controller-revision-hash=pxctest-pxc-74fd756ddd
                  statefulset.kubernetes.io/pod-name=pxctest-pxc-2
Annotations:      cni.projectcalico.org/containerID: 76b104196b7809468734be926426282335043c5e20e65426ad3fcba572c014c3
                  cni.projectcalico.org/podIP: 10.182.47.0/32
                  cni.projectcalico.org/podIPs: 10.182.47.0/32
                  percona.com/configuration-hash: d41d8cd98f00b204e9800998ecf8427e
                  percona.com/ssl-hash: 92cc6c63fcfa0ba09e345aea0c338e3c
                  percona.com/ssl-internal-hash: cf5a6c0387ddbd0102a1e37906be1c4a
Status:           Running
IP:               10.182.47.0
IPs:
  IP:           10.182.47.0
Controlled By:  StatefulSet/pxctest-pxc
Init Containers:
  pxc-init:
    Container ID:  docker://e126e371faaa2f9b4a6714c6059cb5e7dccd947f314eb0f3bce080f7d14b8779
    Image:         perconalab/percona-xtradb-cluster-operator:1.13.0
    Image ID:      docker-pullable://percona/percona-xtradb-cluster-operator@sha256:c674d63242f1af521edfbaffae2ae02fb8d010c0557a67a9c42d2b4a50db5243
    Port:          <none>
    Host Port:     <none>
    Command:
      /pxc-init-entrypoint.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 03 Mar 2024 18:44:26 +0100
      Finished:     Sun, 03 Mar 2024 18:44:26 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        1m
      memory:     256M
    Environment:  <none>
    Mounts:
      /var/lib/mysql from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xsgsc (ro)
Containers:
  pmm-client:
    Container ID:   docker://ae0fa86d66ccccc1ef369f882190225664929d3aa53c2391eb24cd37dc4d26ee
    Image:          percona/pmm-client:2.38.0
    Image ID:       docker-pullable://percona/pmm-client@sha256:e034bf7ea7ce20bce5898f60db01c5703d555149dd5b208ce8e82103e1f0b12a
    Ports:          7777/TCP, 30100/TCP, 30101/TCP, 30102/TCP, 30103/TCP, 30104/TCP, 30105/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Sun, 03 Mar 2024 18:44:28 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     1m
      memory:  150M
    Liveness:  http-get http://:7777/local/Status delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment Variables from:
      pxctest-env-vars-pxc  Secret  Optional: true
    Environment:
      PMM_SERVER:                     REDACTED
      CLIENT_PORT_LISTEN:             7777
      CLIENT_PORT_MIN:                30100
      CLIENT_PORT_MAX:                30105
      POD_NAME:                       pxctest-pxc-2 (v1:metadata.name)
      POD_NAMESPASE:                  pxctest (v1:metadata.namespace)
      PMM_AGENT_SERVER_ADDRESS:       REDACTED
      PMM_AGENT_SERVER_USERNAME:      api_key
      PMM_AGENT_SERVER_PASSWORD:      <set to the key 'pmmserverkey' in secret 'internal-pxctest'>  Optional: false
      PMM_AGENT_LISTEN_PORT:          7777
      PMM_AGENT_PORTS_MIN:            30100
      PMM_AGENT_PORTS_MAX:            30105
      PMM_AGENT_CONFIG_FILE:          /usr/local/percona/pmm2/config/pmm-agent.yaml
      PMM_AGENT_SERVER_INSECURE_TLS:  1
      PMM_AGENT_LISTEN_ADDRESS:       0.0.0.0
      PMM_AGENT_SETUP_METRICS_MODE:   push
      PMM_AGENT_SETUP:                1
      PMM_AGENT_SETUP_FORCE:          1
      PMM_AGENT_SETUP_NODE_TYPE:      container
      PMM_AGENT_SETUP_NODE_NAME:      $(POD_NAMESPASE)-$(POD_NAME)
      DB_TYPE:                        mysql
      DB_USER:                        monitor
      DB_PASSWORD:                    <set to the key 'monitor' in secret 'internal-pxctest'>  Optional: false
      DB_ARGS:                        --query-source=perfschema
      DB_CLUSTER:                     pxc
      DB_HOST:                        localhost
      DB_PORT:                        33062
      CLUSTER_NAME:                   pxctest
      PMM_ADMIN_CUSTOM_PARAMS:
      PMM_AGENT_PRERUN_SCRIPT:        /var/lib/mysql/pmm-prerun.sh
      PMM_AGENT_SIDECAR:              true
      PMM_AGENT_SIDECAR_SLEEP:        5
    Mounts:
      /var/lib/mysql from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xsgsc (ro)
  logs:
    Container ID:   docker://5319eea3c3a4e4bbfbcb5d59b02f06e79a58a43d4979c4fba0a5383effa29d2d
    Image:          percona/percona-xtradb-cluster-operator:1.13.0-logcollector
    Image ID:       docker-pullable://percona/percona-xtradb-cluster-operator@sha256:cf9bdfb046852895fb11cf505c8379367d716e25c0d52ba85c7b03cf4b6ed94e
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sun, 03 Mar 2024 18:44:30 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     1m
      memory:  100M
    Environment Variables from:
      pxctest-log-collector  Secret  Optional: true
    Environment:
      LOG_DATA_DIR:   /var/lib/mysql
      POD_NAMESPASE:  pxctest (v1:metadata.namespace)
      POD_NAME:       pxctest-pxc-2 (v1:metadata.name)
    Mounts:
      /var/lib/mysql from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xsgsc (ro)
  logrotate:
    Container ID:  docker://fe1225c6047f91fa3731520afe676cab856e0ff97b23e57022a050b639588980
    Image:         percona/percona-xtradb-cluster-operator:1.13.0-logcollector
    Image ID:      docker-pullable://percona/percona-xtradb-cluster-operator@sha256:cf9bdfb046852895fb11cf505c8379367d716e25c0d52ba85c7b03cf4b6ed94e
    Port:          <none>
    Host Port:     <none>
    Args:
      logrotate
    State:          Running
      Started:      Sun, 03 Mar 2024 18:44:33 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     1m
      memory:  100M
    Environment:
      SERVICE_TYPE:      mysql
      MONITOR_PASSWORD:  <set to the key 'monitor' in secret 'internal-pxctest'>  Optional: false
    Mounts:
      /var/lib/mysql from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xsgsc (ro)
  pxc:
    Container ID:  docker://22d3e8eb51bcc009a37b460fa4ee411559f1c0a706c31a29db20f6b2a436bc2e
    Image:         percona/percona-xtradb-cluster:8.0.32-24.2
    Image ID:      docker-pullable://percona/percona-xtradb-cluster@sha256:1f978ab8912e1b5fc66570529cb7e7a4ec6a38adbfce1ece78159b0fcfa7d47a
    Ports:         3306/TCP, 4444/TCP, 4567/TCP, 4568/TCP, 33062/TCP, 33060/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      /var/lib/mysql/pxc-entrypoint.sh
    Args:
      mysqld
    State:          Running
      Started:      Sun, 03 Mar 2024 18:44:35 +0100
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      1m
      memory:   256M
    Liveness:   exec [/var/lib/mysql/liveness-check.sh] delay=300s timeout=5s period=10s #success=1 #failure=3
    Readiness:  exec [/var/lib/mysql/readiness-check.sh] delay=15s timeout=15s period=30s #success=1 #failure=5
    Environment Variables from:
      pxctest-env-vars-pxc  Secret  Optional: true
    Environment:
      PXC_SERVICE:                    pxctest-pxc-unready
      MONITOR_HOST:                   %
      MYSQL_ROOT_PASSWORD:            <set to the key 'root' in secret 'internal-pxctest'>        Optional: false
      XTRABACKUP_PASSWORD:            <set to the key 'xtrabackup' in secret 'internal-pxctest'>  Optional: false
      MONITOR_PASSWORD:               <set to the key 'monitor' in secret 'internal-pxctest'>     Optional: false
      LOG_DATA_DIR:                   /var/lib/mysql
      IS_LOGCOLLECTOR:                yes
      CLUSTER_HASH:                   3046112
      OPERATOR_ADMIN_PASSWORD:        <set to the key 'operator' in secret 'internal-pxctest'>  Optional: false
      LIVENESS_CHECK_TIMEOUT:         5
      READINESS_CHECK_TIMEOUT:        15
      DEFAULT_AUTHENTICATION_PLUGIN:  caching_sha2_password
    Mounts:
      /etc/my.cnf.d from auto-config (rw)
      /etc/mysql/mysql-users-secret from mysql-users-secret-file (rw)
      /etc/mysql/ssl from ssl (rw)
      /etc/mysql/ssl-internal from ssl-internal (rw)
      /etc/mysql/vault-keyring-secret from vault-keyring-secret (rw)
      /etc/percona-xtradb-cluster.conf.d from config (rw)
      /tmp from tmp (rw)
      /var/lib/mysql from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xsgsc (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-pxctest-pxc-2
    ReadOnly:   false
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pxctest-pxc
    Optional:  true
  ssl-internal:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pxctest-ssl-internal
    Optional:    true
  ssl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pxctest-ssl
    Optional:    false
  auto-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      auto-pxctest-pxc
    Optional:  true
  vault-keyring-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pxctest-vault
    Optional:    true
  mysql-users-secret-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  internal-pxctest
    Optional:    false
  kube-api-access-xsgsc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  10m                    default-scheduler  Successfully assigned pxctest/pxctest-pxc-2 to node4
  Normal   Pulling    9m58s                  kubelet            Pulling image "perconalab/percona-xtradb-cluster-operator:1.13.0"
  Normal   Pulled     9m57s                  kubelet            Successfully pulled image "perconalab/percona-xtradb-cluster-operator:1.13.0" in 1.116s (1.116s including waiting)
  Normal   Created    9m56s                  kubelet            Created container pxc-init
  Normal   Started    9m56s                  kubelet            Started container pxc-init
  Normal   Pulling    9m56s                  kubelet            Pulling image "percona/pmm-client:2.38.0"
  Normal   Pulled     9m55s                  kubelet            Successfully pulled image "percona/pmm-client:2.38.0" in 1.044s (1.044s including waiting)
  Normal   Created    9m55s                  kubelet            Created container pmm-client
  Normal   Started    9m54s                  kubelet            Started container pmm-client
  Normal   Pulling    9m54s                  kubelet            Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector"
  Normal   Pulled     9m53s                  kubelet            Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector" in 1.114s (1.115s including waiting)
  Normal   Created    9m53s                  kubelet            Created container logs
  Normal   Started    9m52s                  kubelet            Started container logs
  Normal   Pulling    9m52s                  kubelet            Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector"
  Normal   Pulled     9m51s                  kubelet            Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector" in 1.092s (1.092s including waiting)
  Normal   Created    9m49s                  kubelet            Created container logrotate
  Normal   Started    9m49s                  kubelet            Started container logrotate
  Normal   Pulling    9m49s                  kubelet            Pulling image "percona/percona-xtradb-cluster:8.0.32-24.2"
  Normal   Pulled     9m47s                  kubelet            Successfully pulled image "percona/percona-xtradb-cluster:8.0.32-24.2" in 1.443s (1.443s including waiting)
  Normal   Created    9m47s                  kubelet            Created container pxc
  Normal   Started    9m47s                  kubelet            Started container pxc
  Warning  Unhealthy  7m59s (x5 over 9m29s)  kubelet            Readiness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '10.182.47.0:33062' (111)
+ [[ '' == \P\r\i\m\a\r\y ]]
+ exit 1
  Warning  Unhealthy  4m39s  kubelet  Liveness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '10.182.47.0:33062' (111)
+ [[ -n '' ]]
+ exit 1

# kubectl logs pxctest-pxc-2 -clogs -n pxctest
+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/fluent-bit/bin
+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/fluent-bit/bin
+ '[' fluent-bit = logrotate ']'
+ '[' fluent-bit = fluent-bit ']'
+ fluentbit_opt+='-c /etc/fluentbit/fluentbit.conf'
+ test -e /opt/percona/hookscript/hook.sh
+ exec fluent-bit -c /etc/fluentbit/fluentbit.conf
Fluent Bit v2.1.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

# kubectl logs pxctest-pxc-2 -cpxc -n pxctest
+ trap exit SIGTERM
+ '[' m = - ']'
+ CFG=/etc/mysql/node.cnf
+ wantHelp=
+ for arg in "$@"
+ case "$arg" in
++ mysqld -V
++ awk '{print $3}'
++ awk -F. '{print $1"."$2}'
+ MYSQL_VERSION=8.0
++ mysqld -V
++ awk '{print $3}'
++ awk -F. '{print $3}'
++ awk -F- '{print $1}'
+ MYSQL_PATCH_VERSION=32
+ vault_secret=/etc/mysql/vault-keyring-secret/keyring_vault.conf
+ '[' -f /etc/mysql/vault-keyring-secret/keyring_vault.conf ']'
+ '[' -f /usr/lib64/mysql/plugin/binlog_utils_udf.so ']'
+ sed -i '/\[mysqld\]/a plugin_load="binlog_utils_udf=binlog_utils_udf.so"' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a gtid-mode=ON' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a enforce-gtid-consistency' /etc/mysql/node.cnf
+ grep -q '^progress=' /etc/mysql/node.cnf
+ sed -i 's|^progress=.*|progress=1|' /etc/mysql/node.cnf
+ grep -q '^\[sst\]' /etc/mysql/node.cnf
+ grep -q '^cpat=' /etc/mysql/node.cnf
+ sed '/^\[sst\]/a cpat=.*\\.pem$\\|.*init\\.ok$\\|.*galera\\.cache$\\|.*wsrep_recovery_verbose\\.log$\\|.*readiness-check\\.sh$\\|.*liveness-check\\.sh$\\|.*get-pxc-state$\\|.*sst_in_progress$\\|.*pmm-prerun\\.sh$\\|.*sst-xb-tmpdir$\\|.*\\.sst$\\|.*gvwstate\\.dat$\\|.*grastate\\.dat$\\|.*\\.err$\\|.*\\.log$\\|.*RPM_UPGRADE_MARKER$\\|.*RPM_UPGRADE_HISTORY$\\|.*pxc-entrypoint\\.sh$\\|.*unsafe-bootstrap\\.sh$\\|.*pxc-configure-pxc\\.sh\\|.*peer-list$\\|.*auth_plugin$' /etc/mysql/node.cnf
+ [[ 8.0 == \8\.\0 ]]
+ [[ 32 -ge 26 ]]
+ grep -q '^skip_replica_start=ON' /etc/mysql/node.cnf
+ sed -i '/\[mysqld\]/a skip_replica_start=ON' /etc/mysql/node.cnf
+ auth_plugin=caching_sha2_password
+ [[ -f /var/lib/mysql/auth_plugin ]]
++ cat /var/lib/mysql/auth_plugin
+ prev_auth_plugin=caching_sha2_password
+ [[ caching_sha2_password != \m\y\s\q\l\_\n\a\t\i\v\e\_\p\a\s\s\w\o\r\d ]]
+ [[ caching_sha2_password == \m\y\s\q\l\_\n\a\t\i\v\e\_\p\a\s\s\w\o\r\d ]]
+ [[ -z caching_sha2_password ]]
+ [[ 8.0 == \5\.\7 ]]
+ echo caching_sha2_password
+ sed -i /default_authentication_plugin/d /etc/mysql/node.cnf
+ [[ 8.0 == \8\.\0 ]]
+ [[ 32 -ge 27 ]]
+ sed -i '/\[mysqld\]/a authentication_policy=caching_sha2_password,,' /etc/mysql/node.cnf
+ file_env XTRABACKUP_PASSWORD xtrabackup xtrabackup
Percona XtraDB Cluster: Finding peers
2024/03/03 18:00:07 Peer finder enter
2024/03/03 18:00:07 Determined Domain to be pxctest.svc.cluster.local
2024/03/03 18:00:07 Peer list updated
was []
now [10-182-139-39.pxctest-pxc-unready.pxctest.svc.cluster.local 10-182-33-159.pxctest-pxc-unready.pxctest.svc.cluster.local 10-182-47-0.pxctest-pxc-unready.pxctest.svc.cluster.local]
2024/03/03 18:00:07 execing: /var/lib/mysql/pxc-configure-pxc.sh with stdin: 10-182-139-39.pxctest-pxc-unready.pxctest.svc.cluster.local
10-182-33-159.pxctest-pxc-unready.pxctest.svc.cluster.local
10-182-47-0.pxctest-pxc-unready.pxctest.svc.cluster.local

jw12314 · March 3, 2024, 6:06pm

# kubectl get events -n pxctest
LAST SEEN   TYPE      REASON                  OBJECT                                                  MESSAGE
37m         Normal    LeaderElection          lease/08db1feb.percona.com                              percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6_4cea8c81-9d5c-4f81-a5b2-b4847d0d112a became leader
22m         Normal    WaitForFirstConsumer    persistentvolumeclaim/datadir-pxctest-pxc-0             waiting for first consumer to be created before binding
22m         Normal    ExternalProvisioning    persistentvolumeclaim/datadir-pxctest-pxc-0             Waiting for a volume to be created either by the external provisioner 'zfs.csi.openebs.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
22m         Normal    Provisioning            persistentvolumeclaim/datadir-pxctest-pxc-0             External provisioner is provisioning volume for claim "pxctest/datadir-pxctest-pxc-0"
22m         Normal    ProvisioningSucceeded   persistentvolumeclaim/datadir-pxctest-pxc-0             Successfully provisioned volume pvc-26f58993-2236-49ed-bea2-6d1eab4f33e7
20m         Normal    WaitForFirstConsumer    persistentvolumeclaim/datadir-pxctest-pxc-1             waiting for first consumer to be created before binding
20m         Normal    ExternalProvisioning    persistentvolumeclaim/datadir-pxctest-pxc-1             Waiting for a volume to be created either by the external provisioner 'zfs.csi.openebs.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
20m         Normal    Provisioning            persistentvolumeclaim/datadir-pxctest-pxc-1             External provisioner is provisioning volume for claim "pxctest/datadir-pxctest-pxc-1"
20m         Normal    ProvisioningSucceeded   persistentvolumeclaim/datadir-pxctest-pxc-1             Successfully provisioned volume pvc-99ba2bbb-6db1-42ba-b767-28f93293eb77
19m         Normal    WaitForFirstConsumer    persistentvolumeclaim/datadir-pxctest-pxc-2             waiting for first consumer to be created before binding
19m         Normal    ExternalProvisioning    persistentvolumeclaim/datadir-pxctest-pxc-2             Waiting for a volume to be created either by the external provisioner 'zfs.csi.openebs.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
19m         Normal    Provisioning            persistentvolumeclaim/datadir-pxctest-pxc-2             External provisioner is provisioning volume for claim "pxctest/datadir-pxctest-pxc-2"
19m         Normal    ProvisioningSucceeded   persistentvolumeclaim/datadir-pxctest-pxc-2             Successfully provisioned volume pvc-67125998-93c6-4a0d-b462-92c822fc3490
37m         Normal    Scheduled               pod/percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6    Successfully assigned pxctest/percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6 to node6
37m         Normal    Pulling                 pod/percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6    Pulling image "perconalab/percona-xtradb-cluster-operator:main"
37m         Normal    Pulled                  pod/percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6    Successfully pulled image "perconalab/percona-xtradb-cluster-operator:main" in 1.184s (1.184s including waiting)
37m         Normal    Created                 pod/percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6    Created container percona-xtradb-cluster-operator
37m         Normal    Started                 pod/percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6    Started container percona-xtradb-cluster-operator
37m         Warning   Unhealthy               pod/percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6    Liveness probe failed: Get "http://10.182.139.37:8080/metrics": dial tcp 10.182.139.37:8080: connect: connection refused
37m         Normal    SuccessfulCreate        replicaset/percona-xtradb-cluster-operator-5f7b5c4f86   Created pod: percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6
37m         Normal    ScalingReplicaSet       deployment/percona-xtradb-cluster-operator              Scaled up replica set percona-xtradb-cluster-operator-5f7b5c4f86 to 1
22m         Normal    Scheduled               pod/pxctest-haproxy-0                                   Successfully assigned pxctest/pxctest-haproxy-0 to node6
22m         Normal    Pulling                 pod/pxctest-haproxy-0                                   Pulling image "perconalab/percona-xtradb-cluster-operator:1.13.0"
22m         Normal    Pulled                  pod/pxctest-haproxy-0                                   Successfully pulled image "perconalab/percona-xtradb-cluster-operator:1.13.0" in 1.103s (1.103s including waiting)
22m         Normal    Created                 pod/pxctest-haproxy-0                                   Created container pxc-init
22m         Normal    Started                 pod/pxctest-haproxy-0                                   Started container pxc-init
22m         Normal    Pulling                 pod/pxctest-haproxy-0                                   Pulling image "percona/pmm-client:2.38.0"
22m         Normal    Pulled                  pod/pxctest-haproxy-0                                   Successfully pulled image "percona/pmm-client:2.38.0" in 1.357s (1.357s including waiting)
22m         Normal    Created                 pod/pxctest-haproxy-0                                   Created container pmm-client
22m         Normal    Started                 pod/pxctest-haproxy-0                                   Started container pmm-client
22m         Normal    Pulling                 pod/pxctest-haproxy-0                                   Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy"
22m         Normal    Pulled                  pod/pxctest-haproxy-0                                   Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy" in 1.11s (1.11s including waiting)
22m         Normal    Created                 pod/pxctest-haproxy-0                                   Created container haproxy
22m         Normal    Started                 pod/pxctest-haproxy-0                                   Started container haproxy
22m         Normal    Pulling                 pod/pxctest-haproxy-0                                   Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy"
22m         Normal    Pulled                  pod/pxctest-haproxy-0                                   Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy" in 1.008s (1.516s including waiting)
22m         Normal    Created                 pod/pxctest-haproxy-0                                   Created container pxc-monit
22m         Normal    Started                 pod/pxctest-haproxy-0                                   Started container pxc-monit
21m         Warning   Unhealthy               pod/pxctest-haproxy-0                                   Readiness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2
20m         Normal    Scheduled               pod/pxctest-haproxy-1                                   Successfully assigned pxctest/pxctest-haproxy-1 to node5
20m         Normal    Pulling                 pod/pxctest-haproxy-1                                   Pulling image "perconalab/percona-xtradb-cluster-operator:1.13.0"
20m         Normal    Pulled                  pod/pxctest-haproxy-1                                   Successfully pulled image "perconalab/percona-xtradb-cluster-operator:1.13.0" in 1.038s (1.563s including waiting)
20m         Normal    Created                 pod/pxctest-haproxy-1                                   Created container pxc-init
20m         Normal    Started                 pod/pxctest-haproxy-1                                   Started container pxc-init
20m         Normal    Pulling                 pod/pxctest-haproxy-1                                   Pulling image "percona/pmm-client:2.38.0"
20m         Normal    Pulled                  pod/pxctest-haproxy-1                                   Successfully pulled image "percona/pmm-client:2.38.0" in 1.012s (1.012s including waiting)
20m         Normal    Created                 pod/pxctest-haproxy-1                                   Created container pmm-client
20m         Normal    Started                 pod/pxctest-haproxy-1                                   Started container pmm-client
20m         Normal    Pulling                 pod/pxctest-haproxy-1                                   Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy"
20m         Normal    Pulled                  pod/pxctest-haproxy-1                                   Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy" in 990ms (990ms including waiting)
20m         Normal    Created                 pod/pxctest-haproxy-1                                   Created container haproxy
20m         Normal    Started                 pod/pxctest-haproxy-1                                   Started container haproxy
20m         Normal    Pulling                 pod/pxctest-haproxy-1                                   Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy"
20m         Normal    Pulled                  pod/pxctest-haproxy-1                                   Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy" in 1.35s (1.35s including waiting)
20m         Normal    Created                 pod/pxctest-haproxy-1                                   Created container pxc-monit
20m         Normal    Started                 pod/pxctest-haproxy-1                                   Started container pxc-monit
20m         Normal    Scheduled               pod/pxctest-haproxy-2                                   Successfully assigned pxctest/pxctest-haproxy-2 to terem
19m         Normal    Pulling                 pod/pxctest-haproxy-2                                   Pulling image "perconalab/percona-xtradb-cluster-operator:1.13.0"
19m         Normal    Pulled                  pod/pxctest-haproxy-2                                   Successfully pulled image "perconalab/percona-xtradb-cluster-operator:1.13.0" in 1.233s (1.233s including waiting)
19m         Normal    Created                 pod/pxctest-haproxy-2                                   Created container pxc-init
19m         Normal    Started                 pod/pxctest-haproxy-2                                   Started container pxc-init
19m         Normal    Pulling                 pod/pxctest-haproxy-2                                   Pulling image "percona/pmm-client:2.38.0"
19m         Normal    Pulled                  pod/pxctest-haproxy-2                                   Successfully pulled image "percona/pmm-client:2.38.0" in 1.176s (1.176s including waiting)
19m         Normal    Created                 pod/pxctest-haproxy-2                                   Created container pmm-client
19m         Normal    Started                 pod/pxctest-haproxy-2                                   Started container pmm-client
19m         Normal    Pulling                 pod/pxctest-haproxy-2                                   Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy"
19m         Normal    Pulled                  pod/pxctest-haproxy-2                                   Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy" in 1.132s (1.132s including waiting)
19m         Normal    Created                 pod/pxctest-haproxy-2                                   Created container haproxy
19m         Normal    Started                 pod/pxctest-haproxy-2                                   Started container haproxy
19m         Normal    Pulling                 pod/pxctest-haproxy-2                                   Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy"
19m         Normal    Pulled                  pod/pxctest-haproxy-2                                   Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-haproxy" in 1.084s (1.084s including waiting)
19m         Normal    Created                 pod/pxctest-haproxy-2                                   Created container pxc-monit
19m         Normal    Started                 pod/pxctest-haproxy-2                                   Started container pxc-monit
19m         Normal    Pulled                  pod/pxctest-haproxy-2                                   Successfully pulled image "percona/pmm-client:2.38.0" in 1.442s (1.442s including waiting)
22m         Normal    SuccessfulCreate        statefulset/pxctest-haproxy                             create Pod pxctest-haproxy-0 in StatefulSet pxctest-haproxy successful
20m         Normal    SuccessfulCreate        statefulset/pxctest-haproxy                             create Pod pxctest-haproxy-1 in StatefulSet pxctest-haproxy successful
20m         Normal    SuccessfulCreate        statefulset/pxctest-haproxy                             create Pod pxctest-haproxy-2 in StatefulSet pxctest-haproxy successful
22m         Normal    Scheduled               pod/pxctest-pxc-0                                       Successfully assigned pxctest/pxctest-pxc-0 to node6
22m         Normal    Pulling                 pod/pxctest-pxc-0                                       Pulling image "perconalab/percona-xtradb-cluster-operator:1.13.0"
22m         Normal    Pulled                  pod/pxctest-pxc-0                                       Successfully pulled image "perconalab/percona-xtradb-cluster-operator:1.13.0" in 1.033s (2.05s including waiting)
22m         Normal    Created                 pod/pxctest-pxc-0                                       Created container pxc-init
22m         Normal    Started                 pod/pxctest-pxc-0                                       Started container pxc-init
22m         Normal    Pulling                 pod/pxctest-pxc-0                                       Pulling image "percona/pmm-client:2.38.0"
22m         Normal    Pulled                  pod/pxctest-pxc-0                                       Successfully pulled image "percona/pmm-client:2.38.0" in 1.071s (1.277s including waiting)
22m         Normal    Created                 pod/pxctest-pxc-0                                       Created container pmm-client
22m         Normal    Started                 pod/pxctest-pxc-0                                       Started container pmm-client
22m         Normal    Pulling                 pod/pxctest-pxc-0                                       Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector"
22m         Normal    Pulled                  pod/pxctest-pxc-0                                       Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector" in 1.104s (1.104s including waiting)
22m         Normal    Created                 pod/pxctest-pxc-0                                       Created container logs
22m         Normal    Started                 pod/pxctest-pxc-0                                       Started container logs
22m         Normal    Pulling                 pod/pxctest-pxc-0                                       Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector"
22m         Normal    Pulled                  pod/pxctest-pxc-0                                       Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector" in 1.036s (1.036s including waiting)
21m         Normal    Created                 pod/pxctest-pxc-0                                       Created container logrotate
21m         Normal    Started                 pod/pxctest-pxc-0                                       Started container logrotate
21m         Normal    Pulling                 pod/pxctest-pxc-0                                       Pulling image "percona/percona-xtradb-cluster:8.0.32-24.2"
21m         Normal    Pulled                  pod/pxctest-pxc-0                                       Successfully pulled image "percona/percona-xtradb-cluster:8.0.32-24.2" in 1.028s (1.028s including waiting)
21m         Normal    Created                 pod/pxctest-pxc-0                                       Created container pxc
21m         Normal    Started                 pod/pxctest-pxc-0                                       Started container pxc
21m         Warning   Unhealthy               pod/pxctest-pxc-0                                       Readiness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '10.182.139.39:33062' (111)...
20m         Normal    Scheduled               pod/pxctest-pxc-1                                       Successfully assigned pxctest/pxctest-pxc-1 to node5
20m         Normal    Pulling                 pod/pxctest-pxc-1                                       Pulling image "perconalab/percona-xtradb-cluster-operator:1.13.0"
20m         Normal    Pulled                  pod/pxctest-pxc-1                                       Successfully pulled image "perconalab/percona-xtradb-cluster-operator:1.13.0" in 1.054s (1.054s including waiting)
20m         Normal    Created                 pod/pxctest-pxc-1                                       Created container pxc-init
20m         Normal    Started                 pod/pxctest-pxc-1                                       Started container pxc-init
20m         Normal    Pulling                 pod/pxctest-pxc-1                                       Pulling image "percona/pmm-client:2.38.0"
20m         Normal    Pulled                  pod/pxctest-pxc-1                                       Successfully pulled image "percona/pmm-client:2.38.0" in 1.028s (1.028s including waiting)
20m         Normal    Created                 pod/pxctest-pxc-1                                       Created container pmm-client
20m         Normal    Started                 pod/pxctest-pxc-1                                       Started container pmm-client
20m         Normal    Pulling                 pod/pxctest-pxc-1                                       Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector"
20m         Normal    Pulled                  pod/pxctest-pxc-1                                       Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector" in 1.078s (1.078s including waiting)
20m         Normal    Created                 pod/pxctest-pxc-1                                       Created container logs
20m         Normal    Started                 pod/pxctest-pxc-1                                       Started container logs
20m         Normal    Pulling                 pod/pxctest-pxc-1                                       Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector"
20m         Normal    Pulled                  pod/pxctest-pxc-1                                       Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector" in 1.061s (1.061s including waiting)
20m         Normal    Created                 pod/pxctest-pxc-1                                       Created container logrotate
20m         Normal    Started                 pod/pxctest-pxc-1                                       Started container logrotate
20m         Normal    Pulling                 pod/pxctest-pxc-1                                       Pulling image "percona/percona-xtradb-cluster:8.0.32-24.2"
20m         Normal    Pulled                  pod/pxctest-pxc-1                                       Successfully pulled image "percona/percona-xtradb-cluster:8.0.32-24.2" in 1.028s (1.028s including waiting)
20m         Normal    Created                 pod/pxctest-pxc-1                                       Created container pxc
20m         Normal    Started                 pod/pxctest-pxc-1                                       Started container pxc
20m         Warning   Unhealthy               pod/pxctest-pxc-1                                       Readiness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '10.182.33.159:33062' (111)...
19m         Normal    Scheduled               pod/pxctest-pxc-2                                       Successfully assigned pxctest/pxctest-pxc-2 to terem
19m         Normal    Pulling                 pod/pxctest-pxc-2                                       Pulling image "perconalab/percona-xtradb-cluster-operator:1.13.0"
19m         Normal    Pulled                  pod/pxctest-pxc-2                                       Successfully pulled image "perconalab/percona-xtradb-cluster-operator:1.13.0" in 1.116s (1.116s including waiting)
19m         Normal    Created                 pod/pxctest-pxc-2                                       Created container pxc-init
19m         Normal    Started                 pod/pxctest-pxc-2                                       Started container pxc-init
19m         Normal    Pulling                 pod/pxctest-pxc-2                                       Pulling image "percona/pmm-client:2.38.0"
19m         Normal    Pulled                  pod/pxctest-pxc-2                                       Successfully pulled image "percona/pmm-client:2.38.0" in 1.044s (1.044s including waiting)
19m         Normal    Created                 pod/pxctest-pxc-2                                       Created container pmm-client
19m         Normal    Started                 pod/pxctest-pxc-2                                       Started container pmm-client
19m         Normal    Pulling                 pod/pxctest-pxc-2                                       Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector"
19m         Normal    Pulled                  pod/pxctest-pxc-2                                       Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector" in 1.114s (1.115s including waiting)
19m         Normal    Created                 pod/pxctest-pxc-2                                       Created container logs
19m         Normal    Started                 pod/pxctest-pxc-2                                       Started container logs
19m         Normal    Pulling                 pod/pxctest-pxc-2                                       Pulling image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector"
19m         Normal    Pulled                  pod/pxctest-pxc-2                                       Successfully pulled image "percona/percona-xtradb-cluster-operator:1.13.0-logcollector" in 1.092s (1.092s including waiting)
19m         Normal    Created                 pod/pxctest-pxc-2                                       Created container logrotate
19m         Normal    Started                 pod/pxctest-pxc-2                                       Started container logrotate
19m         Normal    Pulling                 pod/pxctest-pxc-2                                       Pulling image "percona/percona-xtradb-cluster:8.0.32-24.2"
19m         Normal    Pulled                  pod/pxctest-pxc-2                                       Successfully pulled image "percona/percona-xtradb-cluster:8.0.32-24.2" in 1.443s (1.443s including waiting)
19m         Normal    Created                 pod/pxctest-pxc-2                                       Created container pxc
19m         Normal    Started                 pod/pxctest-pxc-2                                       Started container pxc
4m1s        Warning   Unhealthy               pod/pxctest-pxc-2                                       Readiness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '10.182.47.0:33062' (111)...
14m         Warning   Unhealthy               pod/pxctest-pxc-2                                       Liveness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '10.182.47.0:33062' (111)...
22m         Normal    SuccessfulCreate        statefulset/pxctest-pxc                                 create Claim datadir-pxctest-pxc-0 Pod pxctest-pxc-0 in StatefulSet pxctest-pxc success
22m         Normal    SuccessfulCreate        statefulset/pxctest-pxc                                 create Pod pxctest-pxc-0 in StatefulSet pxctest-pxc successful
20m         Normal    SuccessfulCreate        statefulset/pxctest-pxc                                 create Claim datadir-pxctest-pxc-1 Pod pxctest-pxc-1 in StatefulSet pxctest-pxc success
20m         Normal    SuccessfulCreate        statefulset/pxctest-pxc                                 create Pod pxctest-pxc-1 in StatefulSet pxctest-pxc successful
19m         Normal    SuccessfulCreate        statefulset/pxctest-pxc                                 create Claim datadir-pxctest-pxc-2 Pod pxctest-pxc-2 in StatefulSet pxctest-pxc success
19m         Normal    SuccessfulCreate        statefulset/pxctest-pxc                                 create Pod pxctest-pxc-2 in StatefulSet pxctest-pxc successful

jw12314 · March 3, 2024, 6:08pm

# kubectl logs percona-xtradb-cluster-operator-5f7b5c4f86-8ljl6 -n pxctest
2024-03-03T17:26:01.501Z        INFO    setup   Runs on {"platform": "kubernetes", "version": "v1.28.6"}
2024-03-03T17:26:01.501Z        INFO    setup   Manager starting up     {"gitCommit": "1769cb021918b0a921e806ae07bbfb263554c437", "gitBranch": "main", "buildTime": "2024-02-24T13:08:52Z", "goVersion": "go1.21.7", "os": "linux", "arch": "amd64"}
2024-03-03T17:26:01.503Z        INFO    setup   Registering Components.
2024-03-03T17:26:11.306Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/validate-percona-xtradbcluster"}
2024-03-03T17:26:11.306Z        INFO    setup   Starting the Cmd.
2024-03-03T17:26:11.307Z        INFO    controller-runtime.metrics      Starting metrics server
2024-03-03T17:26:11.307Z        INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-03-03T17:26:11.307Z        INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-03-03T17:26:11.308Z        INFO    controller-runtime.webhook      Starting webhook server
2024-03-03T17:26:11.309Z        INFO    controller-runtime.certwatcher  Updated current TLS certificate
2024-03-03T17:26:11.310Z        INFO    controller-runtime.webhook      Serving webhook server  {"host": "", "port": 9443}
2024-03-03T17:26:11.310Z        INFO    controller-runtime.certwatcher  Starting certificate watcher
2024-03-03T17:26:11.310Z        INFO    attempting to acquire leader lease pxctest/08db1feb.percona.com...
2024-03-03T17:26:11.408Z        INFO    successfully acquired lease pxctest/08db1feb.percona.com
2024-03-03T17:26:11.409Z        INFO    Starting EventSource    {"controller": "pxcrestore-controller", "source": "kind source: *v1.PerconaXtraDBClusterRestore"}
2024-03-03T17:26:11.409Z        INFO    Starting EventSource    {"controller": "pxc-controller", "source": "kind source: *v1.PerconaXtraDBCluster"}
2024-03-03T17:26:11.409Z        INFO    Starting Controller     {"controller": "pxcrestore-controller"}
2024-03-03T17:26:11.409Z        INFO    Starting EventSource    {"controller": "pxcbackup-controller", "source": "kind source: *v1.PerconaXtraDBClusterBackup"}
2024-03-03T17:26:11.409Z        INFO    Starting Controller     {"controller": "pxcbackup-controller"}
2024-03-03T17:26:11.409Z        INFO    Starting Controller     {"controller": "pxc-controller"}
2024-03-03T17:26:11.999Z        INFO    Starting workers        {"controller": "pxc-controller", "worker count": 1}
2024-03-03T17:26:11.999Z        INFO    Starting workers        {"controller": "pxcbackup-controller", "worker count": 1}
2024-03-03T17:26:12.104Z        INFO    Starting workers        {"controller": "pxcrestore-controller", "worker count": 1}
2024-03-03T17:41:41.730Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a6314f58-dc60-4313-9c96-8ad637b55281"}
2024-03-03T17:41:41.980Z        INFO    KubeAPIWarningLogger    .metadata.ownerReferences contains duplicate entries; API server dedups owner references in 1.20+, and may reject such requests as early as 1.24; please fix your requests; duplicate UID(s) observed: 7c9d7a25-f70d-42be-86aa-4c96369c78d5
2024-03-03T17:41:41.990Z        INFO    Creating or updating backup job {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a6314f58-dc60-4313-9c96-8ad637b55281", "name": "ceb18-daily-backup", "schedule": "0 0 * * *"}
2024-03-03T17:41:41.990Z        INFO    add new job     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a6314f58-dc60-4313-9c96-8ad637b55281", "name": "ensure-version/pxctest/pxctest", "schedule": "0 4 * * *"}
2024-03-03T17:41:41.990Z        INFO    add new job     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a6314f58-dc60-4313-9c96-8ad637b55281", "name": "telemetry/pxctest/pxctest", "schedule": "25 * * * *"}
2024-03-03T17:41:42.745Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "5f3e88e2-bac7-4fdc-9bdc-ba485810cc56"}
2024-03-03T17:41:47.872Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "c06b5f96-0a70-455a-86ed-6fa858f723e4"}
2024-03-03T17:41:52.968Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "93052836-2bbf-4e60-a27e-5daefc7f0614"}
2024-03-03T17:41:58.125Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "c9fccb03-8606-4624-bcee-3cc66cae7716"}
2024-03-03T17:42:03.326Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "e87cab47-7dfd-4d3e-b7af-a12716900546"}
2024-03-03T17:42:08.420Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "86447f88-4101-4954-9007-42cad1468741"}
2024-03-03T17:42:13.517Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "5bb3e56a-2da9-4357-a70d-89d6ec450b0c"}
2024-03-03T17:42:18.726Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "29ddcdd6-444c-42d3-9c07-10285184f0b1"}
2024-03-03T17:42:23.922Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "f9702259-cab9-4ee6-8b3f-f55d1522b78d"}
2024-03-03T17:42:29.176Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "1c7ec298-c703-42b6-b3db-f4ceb5081e06"}
2024-03-03T17:42:34.415Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "36653e95-1abe-433e-a7a8-44aa972cd713"}
2024-03-03T17:42:39.628Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "0c4a9fac-7932-4117-a4ad-45d044ec742e"}
2024-03-03T17:42:44.948Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "6ec671f9-6c72-4241-93cc-3989acbacef5"}
2024-03-03T17:42:50.105Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "3386dc26-1781-4645-aa49-04be323f0443"}
2024-03-03T17:42:55.231Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "69c4a0fe-8e14-4b07-b7e0-42bce232abc7"}
2024-03-03T17:43:00.390Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "686f1b7c-6fda-4e42-ba89-768b2ad7b469"}
2024-03-03T17:43:05.501Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "afacefdb-32a4-46b5-a651-fb482c2e987f"}
2024-03-03T17:43:10.618Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "82cef414-c325-47f8-ae6d-4c1c77d2c5fc"}
2024-03-03T17:43:15.718Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "937fdb42-335b-4ea7-8aa7-032bcf8aa2a0"}
2024-03-03T17:43:16.431Z        INFO    Password expiration policy updated      {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee", "user": "operator"}
2024-03-03T17:43:16.752Z        INFO    Password expiration policy updated      {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee", "user": "monitor"}
2024-03-03T17:43:17.211Z        INFO    User monitor: granted privileges        {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee"}
2024-03-03T17:43:17.347Z        INFO    monitor user privileges granted {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee"}
2024-03-03T17:43:17.558Z        INFO    Password expiration policy updated      {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee", "user": "xtrabackup"}
2024-03-03T17:43:17.882Z        INFO    User xtrabackup: granted privileges     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee"}
2024-03-03T17:43:18.010Z        INFO    Password expiration policy updated      {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee", "user": "replication"}
2024-03-03T17:43:18.135Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee"}
2024-03-03T17:43:21.219Z        INFO    reconcile replication error     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "a4df5b4e-6cdf-47c4-bd56-eeabc4c425ee", "err": "get primary pxc pod: failed to get proxy connection: dial tcp 10.100.247.150:3306: connect: connection refused"}
2024-03-03T17:43:21.601Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "19e7f9f4-a4b3-4c91-bfab-60010e2fa008"}
2024-03-03T17:43:24.620Z        INFO    reconcile replication error     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "19e7f9f4-a4b3-4c91-bfab-60010e2fa008", "err": "get primary pxc pod: failed to get proxy connection: dial tcp 10.100.247.150:3306: connect: connection refused"}
2024-03-03T17:43:26.349Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "17ec1914-607e-468b-aca3-eeda598520a9"}
2024-03-03T17:43:27.858Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "c5c3f5a9-0e8b-4a18-b44b-55f4b74ddf92"}
2024-03-03T17:43:32.818Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "cb16d331-38d0-4ca1-9c22-7da2819eb85e"}
2024-03-03T17:43:38.422Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "e57135f7-5672-475e-bd04-b3a1e4a0445a"}
2024-03-03T17:43:44.036Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "8eba95a9-19c7-45f1-a519-51685445dc7f"}
2024-03-03T17:43:49.516Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "941a734d-39b6-41a3-b4cb-c47127994ede"}
2024-03-03T17:43:55.016Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "ad16913e-77e3-4fc0-8a2d-1f56ffb0716d"}
2024-03-03T17:43:55.642Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "87203378-b276-4bd6-adea-b32efb4280bc"}
2024-03-03T17:44:00.562Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "d49e4776-ef1f-48e5-8def-686afcac043e"}
2024-03-03T17:44:05.853Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "809d4e0c-635b-4766-9440-ec593c16c0b4"}
2024-03-03T17:44:11.239Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "dfa12d3a-6f7d-4e8c-a3d9-460d5de0ae4f"}
2024-03-03T17:44:16.700Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "d581c553-d5a3-4814-bba4-c54466d1405b"}
2024-03-03T17:44:22.059Z        INFO    Waiting for HAProxy to be ready before smart update     {"controller": "pxc-controller", "namespace": "pxctest", "name": "pxctest", "reconcileID": "55e007b9-bf31-48a0-a079-06e2494fe997"}

jw12314 · March 3, 2024, 6:09pm

apiVersion: v1
kind: Secret
metadata:
  name: pxctest-minio-s3
type: Opaque
data:
  AWS_ACCESS_KEY_ID: REDACTED
  AWS_SECRET_ACCESS_KEY: REDACTED
---
apiVersion: v1
kind: Secret
metadata:
  name: pxctest-secrets
type: Opaque
stringData:
  root: REDACTED
  xtrabackup: REDACTED
  monitor: REDACTED
  proxyadmin: REDACTED
  pmmserverkey: REDACTED
  operator: REDACTED
  replication: REDACTED
---
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: pxctest
  finalizers:
    - delete-pxc-pods-in-order
spec:
  crVersion: 1.13.0
  allowUnsafeConfigurations: false
  pause: false
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: "0 4 * * *"
  pxc:
    size: 3
    image: percona/percona-xtradb-cluster:8.0.32-24.2
    autoRecovery: true
    expose:
      enabled: true
      type: LoadBalancer
    replicationChannels:
    - name: pxc1_to_pxc2
      isSource: true
    resources:
      requests:
        memory: 256M
        cpu: 1m
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 1
    volumeSpec:
      persistentVolumeClaim:
        storageClassName: openebs-zfspv-ext4-v2
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 128G
    gracePeriod: 600
  haproxy:
    enabled: true
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.13.0-haproxy
    serviceType: ClusterIP
    resources:
      requests:
        memory: 128M
        cpu: 1m
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 1
    gracePeriod: 30
  proxysql:
    enabled: false
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.13.0-proxysql
    resources:
      requests:
        memory: 1G
        cpu: 1m
    priorityClassName: high-priority
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    volumeSpec:
      persistentVolumeClaim:
        storageClassName: openebs-zfspv-ext4-v2
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 2G
    podDisruptionBudget:
      maxUnavailable: 1
    gracePeriod: 30
  logcollector:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.13.0-logcollector
    resources:
      requests:
        memory: 100M
        cpu: 1m
  pmm:
    enabled: true
    image: percona/pmm-client:2.38.0
    serverHost: REDACTED
    resources:
      requests:
        memory: 150M
        cpu: 1m
  backup:
    image: percona/percona-xtradb-cluster-operator:1.13.0-pxc8.0-backup-pxb8.0.32
    pitr:
      enabled: false
      storageName: minio
      timeBetweenUploads: 10
    storages:
      minio:
        type: s3
        verifyTLS: false
        s3:
          bucket: pxctest-backup
          credentialsSecret: pxctest-minio-s3
          endpointUrl: REDACTED
      s3-us-west:
        type: s3
        verifyTLS: true
        s3:
          bucket: S3-BACKUP-BUCKET-NAME-HERE
          credentialsSecret: my-cluster-name-backup-s3
          region: us-west-2
      azure-blob:
        type: azure
        azure:
          credentialsSecret: azure-secret
          container: test
      fs-pvc:
        type: filesystem
        volume:
          persistentVolumeClaim:
            storageClassName: openebs-zfspv-ext4-v2
            accessModes: [ "ReadWriteOnce" ]
            resources:
              requests:
                storage: 128G
    schedule:
      - name: "daily-backup"
        schedule: "0 0 * * *"
        keep: 5
        storageName: fs-pvc

jw12314 · March 5, 2024, 11:03am

Soooo it cannot be fixed, it cannot be diagnosed, its ready for production? :>

quirkypurplebeanbag · March 28, 2024, 2:34pm

I’m not a 100% sure we’ve been experiencing the same problem, but I’ve managed to find the culprit in my Percona XtraDB PXC Cluster setup. The Percona XtraDB Cluster (PXC) StatefulSet pod pxc-2 seems to always crash after a while. Pods pxc-0 and pxc-1 seem to keep on running fine without crashing.

After some debugging I can see the following trace in the pxc-2 pod just before it shuts down:

pxc 2024-03-21T01:25:03.473726Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50485S), skipping check
pxc 2024-03-21T01:25:32.990043Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
pxc 2024-03-21T01:25:32.990078Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
pxc view ((empty))
pxc 2024-03-21T01:25:32.990240Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)
pxc      at gcomm/src/pc.cpp:connect():161
pxc 2024-03-21T01:25:32.990251Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)
pxc 2024-03-21T01:25:33.990526Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
pxc 2024-03-21T01:25:33.990584Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
pxc 2024-03-21T01:25:33.991017Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'pxc-db-pxc' at 'gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc': -110 (Connection timed out)
pxc 2024-03-21T01:25:33.991067Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
pxc 2024-03-21T01:25:33.991090Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc) failed to establish connection with cluster (reason: 7)
pxc 2024-03-21T01:25:33.991098Z 0 [ERROR] [MY-010119] [Server] Aborting
pxc 2024-03-21T01:25:33.991583Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.32-24.2)  Percona XtraDB Cluster (GPL), Release rel24, Revision 2119e75, WSREP version 26.1.4.3.
pxc 2024-03-21T01:25:33.993694Z 0 [ERROR] [MY-010065] [Server] Failed to shutdown components infrastructure.
pxc 2024-03-21T01:25:33.994197Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED
pxc 2024-03-21T01:25:33.994459Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0
pxc 2024-03-21T01:25:33.997250Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
pxc 2024-03-21T01:25:34.000048Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
pxc 2024-03-21T01:25:34.005141Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0
pxc 2024-03-21T01:25:34.005172Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0
pxc 2024-03-21T01:25:34.005232Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0
pxc 2024-03-21T01:25:34.005239Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0
pxc 2024-03-21T01:25:34.005252Z 0 [Note] [MY-000000] [Galera] avg deps dist 0
pxc 2024-03-21T01:25:34.005296Z 0 [Note] [MY-000000] [Galera] avg cert interval 0
pxc 2024-03-21T01:25:34.005304Z 0 [Note] [MY-000000] [Galera] cert index size 0
pxc 2024-03-21T01:25:34.005366Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
pxc 2024-03-21T01:25:34.005422Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0
pxc 2024-03-21T01:25:34.005456Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
pxc 2024-03-21T01:25:34.005680Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0)
pxc 2024-03-21T01:25:34.006746Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...

The main culprit seems to be the following error:

pxc 2024-03-21T01:25:32.990240Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)

and:

pxc 2024-03-21T01:25:33.991017Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'pxc-db-pxc' at 'gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc': -110 (Connection timed out)
pxc 2024-03-21T01:25:33.991067Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out
pxc 2024-03-21T01:25:33.991090Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://pxc-db-pxc-0.pxc-db-pxc,pxc-db-pxc-1.pxc-db-pxc) failed to establish connection with cluster (reason: 7)
pxc 2024-03-21T01:25:33.991098Z 0 [ERROR] [MY-010119] [Server] Aborting

So it attempts to connect to the pxc-0 and pxc-1 pods. Having a look at the pxc-1 pod’s logs at that exact timeframe results in the following events:

2024-03-21T01:25:01.472335Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert unknown ca
2024-03-21T01:25:02.975503Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert unknown ca
2024-03-21T01:25:04.477549Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert unknown ca
[..snip..]
2024-03-21T01:25:29.983752Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert unknown ca
2024-03-21T01:25:31.483706Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert unknown ca
2024-03-21T01:25:32.988761Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert unknown ca

So actually what seems to go on here it that something goes wrong with the valid certificate generation in the cluster. This is something currently handled by the Helm chart of the PXC cluster itself, but can also be handled with cert-manager:

github.com

percona/percona-helm-charts/blob/pxc-db-1.14.1/charts/pxc-db/values.yaml#L174


      
            ##
            # storageClass: "-"
            accessMode: ReadWriteOnce
            size: 8Gi
          
          # If you set this to true the cluster will be created without TLS
          disableTLS: false
          
          # disable Helm creating TLS certificates if you want to let the operator
          # request certificates from cert-manager
          certManager: false
          
          # If this is set will not create secrets from values and will instead try to use
          # a pre-existing secret of the same name.
          # clusterSecretName: cluster1-secrets
          readinessProbes:
            initialDelaySeconds: 15
            timeoutSeconds: 15
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 5

In my opinion doing hacky tacky certificate stuff manually is a recipe for disaster anyways and is better handled by something like cert-manager in a proper central automated way. After enabling the cert-manager variable in the Helm chart I haven’t seen any crashes anymore. Good luck!

Topic		Replies	Views
Percona XtraDB Node not able to start up - k8s Percona XtraDB Cluster 5.x kubernetes	1	156	March 18, 2024
Three-node cluster failed to start, please help me, thanks Percona XtraDB Cluster 5.x	1	559	July 19, 2016
Can't set up 1 node pxc on kubernetes cluster Percona XtraDB Cluster 5.x mysql , percona	2	195	February 1, 2024
Kubernetes XtraDB Cluster Operator frequently restarts Percona Operator for MySQL	2	138	March 8, 2024
Not able to scale PXC on kubernetes Percona XtraDB Cluster 8.x mysql , percona	4	709	June 1, 2022

K8s XtraDB PXC Cluster, Third node fail to start

Related Topics