Hey,
I’ve been trying to set up percona pxc cluster for CI purposes for a few days now.
I use kind to setup the k8s cluster
Env details:
os: Ubuntu 22.04
k8s: v1.25.2
kind: 0.16.0
I’ve installed the percona pxc following the blog post here https://www.percona.com/blog/testing-percona-distribution-for-mysql-operator-locally-with-kind/ (the difference is that I use Ubuntu) without any modifications.
It seems that there is a networking issue although I’m not 100% sure.
As you can see I installed the cilium CNI but it hadn’t been working before with the kindnet too and basically I’d been getting the same errors.
It seems that mysqld started correctly (cluster1-pxc-0 logs):
2023-02-12T23:33:05.996245Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict locatio │
│ 2023-02-12T23:33:05.996290Z 0 [Warning] [MY-000000] [WSREP] Node is not a cluster node. Disabling pxc_strict_mode │
│ 2023-02-12T23:33:05.997592Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29-21.1) starting as process 1 │
│ 2023-02-12T23:33:05.997669Z 0 [ERROR] [MY-010338] [Server] Can't find error-message file '/usr/share/percona-xtradb-cluster/errmsg.sys'. Check err │
│ 2023-02-12T23:33:06.001039Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. │
│ 2023-02-12T23:33:06.001096Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported │
│ 2023-02-12T23:33:06.008611Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. │
│ 2023-02-12T23:33:06.237670Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. │
│ 2023-02-12T23:33:06.342052Z 1 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil) │
│ 2023-02-12T23:33:06.347982Z 1 [System] [MY-000000] [WSREP] PXC upgrade completed successfully │
│ 2023-02-12T23:33:06.493585Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. │
│ 2023-02-12T23:33:06.493614Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported │
│ 2023-02-12T23:33:06.499217Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/lib/mysql' in the path is acces │
│ 2023-02-12T23:33:06.513909Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2 │
│ 2023-02-12T23:33:06.513940Z 0 [Note] [MY-000000] [Galera] Loading provider none initial position: 00000000-0000-0000-0000-000000000000:-1 │
│ 2023-02-12T23:33:06.513953Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library 'none' │
│ 2023-02-12T23:33:06.514877Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql │
│ 2023-02-12T23:33:06.514953Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.29-21.1' socket: '/var/lib/mys │
│ Stream closed EOF for default/cluster1-pxc-0 (pxc)
The errors that I’m getting:
- cluster1-haproxy-0:pxc-monit
│ 2023/02/12 23:36:12 lookup cluster1-pxc on 10.96.0.10:53: no such host │
│ 2023/02/12 23:36:13 lookup cluster1-pxc on 10.96.0.10:53: no such host
- Events from the pod:
Normal Pulled 56m kubelet Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy" in 1.025621 │
│ 516s │
│ Normal Pulling 31m (x11 over 71m) kubelet Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-haproxy" │
│ Warning Unhealthy 16m (x460 over 71m) kubelet Readiness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initia │
│ l communication packet', system error: 2 │
│ Normal Pulled 6m35s (x8 over 42m) kubelet (combined from similar events): Successfully pulled image "percona/percona-xtradb-cluster-oper │
│ ator:1.12.0-haproxy" in 1.118248913s │
│ Warning BackOff 86s (x110 over 47m) kubelet Back-off restarting failed container │
│
- Events from the cluster1-pxc-0 pod:
Events: │
│ Type Reason Age From Message │
│ ---- ------ ---- ---- ------- │
│ Normal Pulling 18m (x11 over 73m) kubelet Pulling image "percona/percona-xtradb-cluster:8.0.29-21.1" │
│ Warning Unhealthy 2m35s (x203 over 72m) kubelet Readiness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '10.244.2.46:33 │
│ 062' (111) │
│ + [[ '' == \P\r\i\m\a\r\y ]] │
│ + exit 1 │
│
I’ve spotted a similar issue here -Fresh instance with Percona XtraDB Cluster Operator v1.8.0 not starting completly under OKD - #21 by pavloos but it was 2 year ago but I was trying everything suggested and it seems that the workaround has already been applied.
I’ve restarted the coredns, installed and installed celium CNI instead of kindnet, but ended up facing the same issues.
Could you please take a look or at least give me some advice on how to debug it further?
Best regards,
Tomek