Can not delete pxc operator

Description:

I have tried to deploy the mysql pxc operator on my arm64 cluster.
I didn’t know that openebs didn’t provide pv or pvc and thought everything just didn’t work.
I have later installed rancher’s local path thingy.
Anyhow,
I tried to deploy the mysql operator or pxc cluster, but it failed, so I deleted the namespace mysql-operator.
However

kubectl get pxc -n mysql-operator 
NAME              ENDPOINT                                 STATUS         PXC   PROXYSQL   HAPROXY   AGE
mysql-db-pxc-db   mysql-db-pxc-db-haproxy.mysql-operator   initializing                              24h

I can’t delete this resource or whatever it’s supposed to be.
I would like to delete it so the namespace can be deleted.

Steps to Reproduce:

I can’t retrace the actions I did

Version:

latest

Logs:

apiVersion: v1
items:
- apiVersion: pxc.percona.com/v1
  kind: PerconaXtraDBCluster
  metadata:
    annotations:
      meta.helm.sh/release-name: mysql-db
      meta.helm.sh/release-namespace: mysql-operator
    creationTimestamp: "2025-01-30T14:48:11Z"
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2025-01-30T14:51:20Z"
    finalizers:
    - percona.com/delete-pxc-pods-in-order
    generation: 2
    labels:
      app.kubernetes.io/instance: mysql-db
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: pxc-db
      app.kubernetes.io/version: 1.16.1
      helm.sh/chart: pxc-db-1.16.1
    name: mysql-db-pxc-db
    namespace: mysql-operator
    resourceVersion: "187238"
    uid: 69474c7d-f606-4fe2-8d4b-58709a72a6b3
  spec:
    backup:
      image: percona/percona-xtradb-cluster-operator:1.16.1-pxc8.0-backup-pxb8.0.35
      pitr:
        enabled: false
      schedule: []
      storages: {}
    crVersion: 1.16.1
    enableCRValidationWebhook: false
    enableVolumeExpansion: false
    haproxy:
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      annotations: {}
      enabled: true
      gracePeriod: 30
      image: percona/haproxy:2.8.11
      labels: {}
      livenessDelaySec: 300
      livenessProbes:
        failureThreshold: 4
        initialDelaySeconds: 60
        periodSeconds: 30
        successThreshold: 1
        timeoutSeconds: 5
      nodeSelector: {}
      podDisruptionBudget:
        maxUnavailable: 1
      readinessDelaySec: 15
      readinessProbes:
        failureThreshold: 3
        initialDelaySeconds: 15
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 1
      resources:
        limits: {}
        requests:
          cpu: 600m
          memory: 1G
      sidecarPVCs: []
      sidecarResources:
        limits: {}
        requests: {}
      sidecarVolumes: []
      sidecars: []
      size: 3
      tolerations: []
      volumeSpec:
        emptyDir: {}
    logCollectorSecretName: mysql-db-pxc-db-log-collector
    logcollector:
      enabled: true
      image: percona/percona-xtradb-cluster-operator:1.16.1-logcollector-fluentbit3.2.2
      resources:
        limits: {}
        requests:
          cpu: 200m
          memory: 100M
    pause: false
    pmm:
      enabled: false
    proxysql:
      enabled: false
    pxc:
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      annotations: {}
      autoRecovery: true
      gracePeriod: 600
      image: percona/percona-xtradb-cluster:8.0.39-30.1
      labels: {}
      livenessDelaySec: 300
      livenessProbes:
        failureThreshold: 3
        initialDelaySeconds: 300
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      nodeSelector: {}
      podDisruptionBudget:
        maxUnavailable: 1
      readinessDelaySec: 15
      readinessProbes:
        failureThreshold: 5
        initialDelaySeconds: 15
        periodSeconds: 30
        successThreshold: 1
        timeoutSeconds: 15
      resources:
        limits: {}
        requests:
          cpu: 600m
          memory: 1G
      sidecarPVCs: []
      sidecarResources:
        limits: {}
        requests: {}
      sidecarVolumes: []
      sidecars: []
      size: 3
      tolerations: []
      volumeSpec:
        persistentVolumeClaim:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 8Gi
    secretsName: mysql-db-pxc-db-secrets
    sslInternalSecretName: mysql-db-pxc-db-ssl-internal
    sslSecretName: mysql-db-pxc-db-ssl
    tls:
      enabled: true
    updateStrategy: SmartUpdate
    upgradeOptions:
      apply: disabled
      schedule: 0 4 * * *
      versionServiceEndpoint: https://check.percona.com
    vaultSecretName: mysql-db-pxc-db-vault
  status:
    backup: {}
    conditions:
    - lastTransitionTime: "2025-01-30T14:48:12Z"
      status: enabled
      type: tls
    - lastTransitionTime: "2025-01-30T14:48:15Z"
      status: "True"
      type: initializing
    haproxy:
      labelSelectorPath: app.kubernetes.io/component=haproxy,app.kubernetes.io/instance=mysql-db-pxc-db,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster
      message: 'haproxy: back-off 1m20s restarting failed container=haproxy pod=mysql-db-pxc-db-haproxy-0_mysql-operator(a57b2d0f-8bd4-432d-b2bd-ca5cd1d751b0);
        pxc-monit: back-off 1m20s restarting failed container=pxc-monit pod=mysql-db-pxc-db-haproxy-0_mysql-operator(a57b2d0f-8bd4-432d-b2bd-ca5cd1d751b0); '
      size: 3
      status: initializing
    host: mysql-db-pxc-db-haproxy.mysql-operator
    logcollector: {}
    message:
    - 'pxc: 0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims.
      preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.'
    - 'haproxy: haproxy: back-off 1m20s restarting failed container=haproxy pod=mysql-db-pxc-db-haproxy-0_mysql-operator(a57b2d0f-8bd4-432d-b2bd-ca5cd1d751b0);
      pxc-monit: back-off 1m20s restarting failed container=pxc-monit pod=mysql-db-pxc-db-haproxy-0_mysql-operator(a57b2d0f-8bd4-432d-b2bd-ca5cd1d751b0); '
    observedGeneration: 1
    pmm: {}
    proxysql: {}
    pxc:
      labelSelectorPath: app.kubernetes.io/component=pxc,app.kubernetes.io/instance=mysql-db-pxc-db,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster
      message: '0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims.
        preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.'
      size: 3
      status: initializing
    ready: 0
    size: 6
    state: initializing
kind: List
metadata:
  resourceVersion: ""

Expected Result:

The namespace with all its resources gets deleted

Actual Result:

mysql-db-pxc-db remains

Additional Information:

PXC DB cluster does not support arm64. You can deploy only the operator pod on the arm64 cluster. You will be able to do it as soon as we have ProxySQL and HaProxy images with arm64 support. I hope we can expect it in Q2 or Q3.

The question is:
How do I delete the resource?
It’s stuck.
The documentation isn’t helping.

How do I delete the resource?
mysql-operator Terminating 2d5h

Your soltion has been the worst of all of them.
You’re even hold resources hostage that can’t be deleted and you refuse to help!

@dalu, Have you investigated why the resources cannot be deleted? Did you try to remove finalizers? like kubectl patch pxc/mysql-db-pxc-db -p '{"metadata":{"finalizers":[]}}' --type=merge and then delete it kubectl delete pxc mysql-db-pxc-db.
P.S. Here’s a great article on debugging and fixing this issue: Kubernetes Pods Stuck in Terminating – A Resolution Guide."

And we have a page in our official doc about it Delete the Operator - Percona Operator for MySQL

1 Like