# Random pxc node fails with gcomm issues

**URL:** <https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548>\
**Category:** Percona Operator for MySQL\
**Tags:** mysql, percona\
**Created:** [January 10, 2023, 10:17am UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548 "2023-01-10T10:17:03Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ilford](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ilford/32/9063_2.png) [@Ilford](https://forums.percona.com/u/Ilford)\
**Post date:** [January 10, 2023, 10:17am UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548/1 "2023-01-10T10:17:03Z")

</div>

Hello,

I’m deploying on my k8s cluster the pxc-opertor and pxc-db helm charts, both in version 1.12.0.

It works great but after some time one of the cluster pod crashes and in the logs I have this issues :

```auto
[ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)\n\t at gcomm/src/pc.cpp:connect():161\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-01-10T10:11:42.739343Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"} 

```

As if the pods crashes for some reason and then it cannot synchronize with other pods of the cluster.

Just for you to know, I enabled persistence in helm values and I had to set pxc\_strict\_mode to PERMISSIVE to support some legacy apps, I hope this is not related.

Does someone have a clue about how to fix this ?

Thanks 🙂

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [January 10, 2023, 10:40am UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548/2 "2023-01-10T10:40:08Z")

</div>

Hey @Ilford,

Do you know why pods crash? Do you have any logs of these crashes?

---

<div class="post-metadata">

**Author:** ![Ilford](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ilford/32/9063_2.png) [@Ilford](https://forums.percona.com/u/Ilford)\
**Post date:** [January 10, 2023, 11:00am UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548/3 "2023-01-10T11:00:58Z")

</div>

The first errors that appears before a crash are these (I’m not sure that this is always this case)

```auto
{"log":"2023-01-10T10:22:32.865988Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)\n\t at gcomm/src/pc.cpp:connect():161\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-01-10T10:22:32.866038Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-01-10T10:22:33.866582Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'siam-pxc-db-pxc' at 'gcomm://siam-pxc-db-pxc-0.siam-pxc-db-pxc,siam-pxc-db-pxc-1.siam-pxc-db-pxc': -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-01-10T10:22:33.866625Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-01-10T10:22:33.866653Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://siam-pxc-db-pxc-0.siam-pxc-db-pxc,siam-pxc-db-pxc-1.siam-pxc-db-pxc) failed to establish connection with cluster (reason: 7)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-01-10T10:22:33.866688Z 0 [ERROR] [MY-010119] [Server] Aborting\n","file":"/var/lib/mysql/mysqld-error.log"}

```

---

<div class="post-metadata">

**Author:** ![Ilford](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ilford/32/9063_2.png) [@Ilford](https://forums.percona.com/u/Ilford)\
**Post date:** [January 10, 2023, 11:43am UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548/4 "2023-01-10T11:43:54Z")

</div>

Ok, I managed to catch a crash on live, this appears in the logs:

```auto
[2023/01/10 11:38:30] [engine] caught signal (SIGTERM)
{"log":"2023-01-10T11:38:30.561075Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.29-21.1).\n","file":"/var/lib/mysql/mysqld-error.log"}

```

And then, impossible to sync with the cluster.

I did not do something special to send this signal to the engine.

For you to know I have multiple pxc-cluster and operators in different namespaces. I did not deployed the operator “cluster wide” but I have the feeling that there is something wrong with the operators, the crashed appeared when I deployed an pxc-cluster in a different namespace. I not sure if it is related.

---

<div class="post-metadata">

**Author:** ![Ilford](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ilford/32/9063_2.png) [@Ilford](https://forums.percona.com/u/Ilford)\
**Post date:** [January 10, 2023, 11:51am UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548/5 "2023-01-10T11:51:02Z")

</div>

I can trigger the crash by deleting the pod, then the statefulset recreates it, but same error appears (with gcomm timeout)

---

<div class="post-metadata">

**Author:** ![Ilford](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ilford/32/9063_2.png) [@Ilford](https://forums.percona.com/u/Ilford)\
**Post date:** [January 10, 2023, 12:02pm UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548/6 "2023-01-10T12:02:09Z")

</div>

I’m playing a lot with my pxc clusters and I’m definitely convinced that there is some relations between two cluster+operators deployed on two separate namespaces. I don’t know how this is possible tough, but each time I act on a cluster (like deleting, restoring a backup etc,), something happens on a other cluster (a pods crashed or a crashed pod fixes itself…)

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [January 10, 2023, 7:03pm UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548/7 "2023-01-10T19:03:31Z")

</div>

Hi,

> [@Ilford](#):
>
> ```auto
> [2023/01/10 11:38:30] [engine] caught signal (SIGTERM)
> {"log":"2023-01-10T11:38:30.561075Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.29-21.1).\n","file":"/var/lib/mysql/mysqld-error.log"}
> 
> ```

I think a liveness probe kills the container or you have a problem with resources and k8s killed it. Do you see any ‘OOMKilled’ errors?

Please check the output of the following commands:

```auto
kubectl get events --sort-by=.metadata.creationTimestamp -w
kubectl describe pods <your_pod_name>

```

Please provide your CR as well.

> [@Ilford](#):
>
> For you to know I have multiple pxc-cluster and operators in different namespaces. I did not deployed the operator “cluster wide” but I have the feeling that there is something wrong with the operators, the crashed appeared when I deployed an pxc-cluster in a different namespace. I not sure if it is related.

I am not sure how possible it is if you do not use CW. I have an idea if e.g. your new cluster takes some resources from k8s cluster and you do not have any limits by a namespace it can have an influence on existing clusters.

---

<div class="post-metadata">

**Author:** ![Ilford](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ilford/32/9063_2.png) [@Ilford](https://forums.percona.com/u/Ilford)\
**Post date:** [January 12, 2023, 12:38pm UTC](https://forums.percona.com/t/random-pxc-node-fails-with-gcomm-issues/19548/8 "2023-01-12T12:38:38Z")

</div>

Hi,

I tried to inspect the logs but I didn’t see OOM errors.

As I suspected some wrong communications between services, to make some changes I enabled certManager with this values :

```auto
pxc-db:
  pxc:  
    certManager: true
  tls:
    issuerConf:
      name: my-issuer
      kind: ClusterIssuer

```

And now everything works very fluently. If I act as a chaos monkey and I delete some pods manually, every pods synchronizes with the cluster very quickly. I waited a couple days to reply here to be sure but for now I have no crashes (I had systematically a crash after a couple of minutes after the cluster was bootstrapped before).

I cannot explain how this config solved my issue, but I’m glad it did 😀

Thanks for your time !