# PXC cluster CrashLoopBackOff **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Tags:** mysql, percona, kubernetes\ **Created:** [November 22, 2021, 9:17am UTC](https://forums.percona.com/t/pxc-cluster-crashloopbackoff/13151 "2021-11-22T09:17:41Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Antoine](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@Antoine](https://forums.percona.com/u/Antoine)\ **Post date:** [November 22, 2021, 9:17am UTC](https://forums.percona.com/t/pxc-cluster-crashloopbackoff/13151/1 "2021-11-22T09:17:41Z")
Hi! My PXC 5.7 cluster is crash looping with the following error: ```auto 2021-11-22T08:57:27.796235Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.34-37 started; log sequence number 2264189188 2021-11-22T08:57:27.796281Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery. 2021-11-22T08:57:27.797037Z 0 [Note] Plugin 'FEDERATED' is disabled. 2021-11-22T08:57:27.809704Z 0 [Note] InnoDB: Starting recovery for XA transactions... 2021-11-22T08:57:27.809727Z 0 [Note] InnoDB: Transaction 12760 in prepared state after recovery 2021-11-22T08:57:27.809731Z 0 [Note] InnoDB: Transaction contains changes to 1 rows 2021-11-22T08:57:27.809736Z 0 [Note] InnoDB: 1 transactions in prepared state after recovery 2021-11-22T08:57:27.809739Z 0 [Note] Found 1 prepared transaction(s) in InnoDB 2021-11-22T08:57:27.809753Z 0 [Warning] WSREP: Discovered discontinuity in recovered wsrep transaction XIDs. Truncating the recovery list to 0 entries 2021-11-22T08:57:27.809757Z 0 [Note] WSREP: Last wsrep seqno to be recovered 2656 2021-11-22T08:57:27.809852Z 0 [ERROR] Found 1 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions. 2021-11-22T08:57:27.809862Z 0 [ERROR] Aborting ``` I don’t know what to do, since I’m using the operator installed with the chart pxc-operator and an instance installed with the chart pxc-db. So, what should I do? And why the operator does not handle automatically this use case? Why only 1 replica of 3 is crash looping while others are OK? And why all the haproxy in front of PXC instances are unready (there is no HA so?)? ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [November 22, 2021, 3:48pm UTC](https://forums.percona.com/t/pxc-cluster-crashloopbackoff/13151/2 "2021-11-22T15:48:03Z")
Hello @Antoine, I would manually destroy that pod and let the operator recreate it so that it forces a fresh SST from one of the other nodes. Yes, I think the operator should handle this. Can you please open a bug report at [https://jira.percona.com](https://jira.percona.com) with all the config files and other info? ---
**Author:** ![Antoine](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@Antoine](https://forums.percona.com/u/Antoine)\ **Post date:** [November 22, 2021, 5:02pm UTC](https://forums.percona.com/t/pxc-cluster-crashloopbackoff/13151/3 "2021-11-22T17:02:15Z")
Hello @matthewb, thanks for your reply! I tried to delete the pod, but it still crashes looping with the same error. I will open a bug request yes, thanks. I’m still a help taker. 🙂 ---
**Author:** ![Antoine](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@Antoine](https://forums.percona.com/u/Antoine)\ **Post date:** [November 24, 2021, 2:55pm UTC](https://forums.percona.com/t/pxc-cluster-crashloopbackoff/13151/4 "2021-11-24T14:55:33Z")
I see the default value of innodb\_flush\_log\_at\_trx\_commit is 0. Could this be the problem? I have a very long transaction each day (about an hour). I think the instance crashed during this one. The other problem that worries me is that all the haproxy in front of the 3 instances are in CrashLoopBackOff. Why? I think they should be OK because there are 2 of 3 PXC instances which are ready. ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [November 30, 2021, 9:22am UTC](https://forums.percona.com/t/pxc-cluster-crashloopbackoff/13151/5 "2021-11-30T09:22:47Z")
Hey @Antoine , thanks for raising this. Could you please share your values.yaml from helm? I would love to try to reproduce this. ---
**Author:** ![Antoine](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@Antoine](https://forums.percona.com/u/Antoine)\ **Post date:** [November 30, 2021, 1:08pm UTC](https://forums.percona.com/t/pxc-cluster-crashloopbackoff/13151/6 "2021-11-30T13:08:59Z")
Hi @Sergey_Pronin, The values for the operator are: ```auto resources: {} affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: operator app.kubernetes.io/instance: pxc-operator app.kubernetes.io/name: pxc-operator app.kubernetes.io/part-of: pxc-operator topologyKey: kubernetes.io/hostname ``` The values for the pxc instance are: ```auto finalizers: - delete-pxc-pods-in-order # Can delete proxysql PVCs, they're recreatable. - delete-proxysql-pvc # Don't delete database PVCs. # - delete-pxc-pvc upgradeOptions: apply: 5.7-recommended pxc: resources: requests: cpu: "2" memory: 4Gi limits: cpu: "2" memory: 4Gi affinity: advanced: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: pxc app.kubernetes.io/instance: pxc-db app.kubernetes.io/managed-by: percona-xtradb-cluster-operator app.kubernetes.io/name: percona-xtradb-cluster app.kubernetes.io/part-of: percona-xtradb-cluster topologyKey: kubernetes.io/hostname antiAffinityTopologyKey: kubernetes.io/hostname persistence: enabled: true storageClass: vsphere-delete accessMode: ReadWriteOnce size: 100Gi disableTLS: true clusterSecretName: pxc-custom-secret haproxy: resources: requests: cpu: "1" memory: 1Gi limits: cpu: "1" memory: 1Gi affinity: advanced: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: haproxy app.kubernetes.io/instance: pxc-db app.kubernetes.io/managed-by: percona-xtradb-cluster-operator app.kubernetes.io/name: percona-xtradb-cluster app.kubernetes.io/part-of: percona-xtradb-cluster topologyKey: kubernetes.io/hostname antiAffinityTopologyKey: kubernetes.io/hostname logcollector: resources: requests: cpu: "10m" memory: 16Mi limits: cpu: "100m" memory: "128Mi" backup: enabled: true storages: mys3: type: s3 s3: credentialsSecret: pxc-backups-custom-secret bucket: mybucket region: myregion endpointUrl: https://minio-endpoint resources: requests: cpu: "2" memory: 4Gi limits: cpu: "2" memory: 4Gi schedule: - name: s3-daily-backup schedule: "0 0 * * *" keep: 10 storageName: mys3 ``` I manually created the `pxc-custom-secret` which contains all users/passwords and `pxc-backups-custom-secret` for the S3 storage. I use the chart version 1.9.1 for the operator and the pxc instance. ---
**Author:** ![Elman\_Torres](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/elman_torres/32/14732_2.png) [@Elman\_Torres](https://forums.percona.com/u/Elman_Torres)\ **Post date:** [February 27, 2024, 4:54am UTC](https://forums.percona.com/t/pxc-cluster-crashloopbackoff/13151/7 "2024-02-27T04:54:35Z")
Hello @Antoine , have you solved the problem? I’m having this issue right now.