# SmartUpdate breaks pxc pod when "applying changes"

**URL:** https://forums.percona.com/t/smartupdate-breaks-pxc-pod-when-applying-changes/32527
**Category:** Percona Operator for MySQL
**Created:** [August 23, 2024, 10:58am UTC](https://forums.percona.com/t/smartupdate-breaks-pxc-pod-when-applying-changes/32527 "2024-08-23T10:58:52Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![dkorbginski](https://avatars.discourse-cdn.com/v4/letter/d/b9bd4f/32.png) [@dkorbginski](https://forums.percona.com/u/dkorbginski)
#### Post date: [August 23, 2024, 10:58am UTC](https://forums.percona.com/t/smartupdate-breaks-pxc-pod-when-applying-changes/32527/1 "2024-08-23T10:58:52Z")

</div>

## Description:

I’m running the operator and cluster deployed via helm. Initially, the cluster is running without problems, but after a few moments, the operator wants to “apply changes to secondary pod” according to its logs. This restarts the targeted pod, but it’s not able to reach a working state. The `mysqld --wsrep_start_position=...` command fails.

In the pod logs, I see these errors:

> **pxc pod logs**
>
> ```auto
> {"log":"2024-08-23T10:48:09.639919Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0\n","file":"/var/lib/mysql/mysqld-error.log"}
> {"log":"2024-08-23T10:48:09.640023Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node\nview ((empty))\n","file":"/var/lib/mysql/mysqld-error.log"}
> {"log":"2024-08-23T10:48:09.647832Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)\n\t at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcomm/src/pc.cpp:connect():176\n","file":"/var/lib/mysql/mysqld-error.log"}
> {"log":"2024-08-23T10:48:09.653509Z 0 [ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"}
> {"log":"2024-08-23T10:48:10.658832Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread\n","file":"/var/lib/mysql/mysqld-error.log"}
> {"log":"2024-08-23T10:48:10.658894Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread\n","file":"/var/lib/mysql/mysqld-error.log"}
> {"log":"2024-08-23T10:48:10.658996Z 0 [ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs.cpp:gcs_open():1880: Failed to open channel 'testapp-db-pxc' at 'gcomm://testapp-db-pxc-0.testapp-db-pxc,testapp-db-pxc-1.testapp-db-pxc': -110 (Connection timed out)\n","file":"/var/lib/mysql/mysqld-error.log"}
> {"log":"2024-08-23T10:48:10.665159Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out\n","file":"/var/lib/mysql/mysqld-error.log"}
> {"log":"2024-08-23T10:48:10.665196Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://testapp-db-pxc-0.testapp-db-pxc,testapp-db-pxc-1.testapp-db-pxc) failed to establish connection with cluster (reason: 7)\n","file":"/var/lib/mysql/mysqld-error.log"}
> 
> ```

I don’t know/understand what kind of update the operator is trying to do here. Any ideas what it might be and why it causes mysql to fail to start?

## Version:

Helm charts “pxc-operator” and “pxc-db” 1.15.0

## Helm values:

pxc-operator:

```auto
watchAllNamespaces: true

```

pxc-db:

```auto
pxc:
  persistence:
    storageClass: rook-ceph-block
    size: 2Gi
  disableTLS: true

```

---

<div class="post-metadata">

### Author: ![yunus.uyanik](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/yunus.uyanik/32/19250_2.png) [@yunus.uyanik](https://forums.percona.com/u/yunus.uyanik)
#### Post date: [September 25, 2024, 11:56am UTC](https://forums.percona.com/t/smartupdate-breaks-pxc-pod-when-applying-changes/32527/2 "2024-09-25T11:56:43Z")

</div>

Hi @dkorbginski,

> I don’t know/understand what kind of update the operator is trying to do here. Any ideas what it might be and why it causes mysql to fail to start?

The cluster setting by following variable [wsrep\_start\_position](https://docs.percona.com/percona-xtradb-cluster/8.0/wsrep-system-index.html#wsrep_start_position) starting up MySQL processes.

Regarding the pod logs;  
`{"log":"2024-08-23T10:48:10.665159Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out\n","file":"/var/lib/mysql/mysqld-error.log"}`  
You should ensure all nodes can communicate with each other.