Helm uninstall not working

MySQL & MariaDB Percona Operator for MySQL
1

I can use Helm for uninstalling (the charts is removed from helm list) but It seems nothing happen with the pods It remains in k8s.

Installation was based on this: Install with Helm - Percona Operator for MySQL based on Percona XtraDB Cluster

Before:

root@kubernetes-master-ubuntu:~# helm list -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
ingress-nginx   default         2               2022-08-18 02:10:46.471275227 +0000 UTC deployed        ingress-nginx-4.2.1     1.3.0
my-db           my-namespace    1               2022-09-29 20:54:38.257264002 +0000 UTC deployed        pxc-db-1.11.0           1.11.0
my-db           default         1               2022-09-29 22:03:08.556752683 +0000 UTC deployed        pxc-db-1.11.5           1.11.0
my-op           default         1               2022-09-29 20:47:32.901839549 +0000 UTC deployed        pxc-operator-1.11.0     1.11.0

After:

root@kubernetes-master-ubuntu:~# helm list -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
ingress-nginx   default         2               2022-08-18 02:10:46.471275227 +0000 UTC deployed        ingress-nginx-4.2.1     1.3.0

root@kubernetes-master-ubuntu:~# k get all -ALL | grep pxc
default                pod/my-db-pxc-db-haproxy-0                             1/2     Running   125 (2m20s ago)   2d20h
default                pod/my-db-pxc-db-haproxy-1                             1/2     Running   121 (113s ago)    2d20h
default                pod/my-db-pxc-db-haproxy-2                             0/2     Pending   0                 2d20h
default                pod/my-db-pxc-db-pxc-0                                 3/3     Running   1 (36h ago)       2d20h
default                pod/my-db-pxc-db-pxc-1                                 3/3     Running   1 (36h ago)       2d20h
default                pod/my-db-pxc-db-pxc-2                                 0/3     Pending   0                 2d20h
default                service/my-db-pxc-db-haproxy                      ClusterIP      10.109.124.94    <none>        3306/TCP,3309/TCP,33062/TCP,33060/TCP   2d20h
default                service/my-db-pxc-db-haproxy-replicas             ClusterIP      10.101.223.216   <none>        3306/TCP                                2d20h
default                service/my-db-pxc-db-pxc                          ClusterIP      None             <none>        3306/TCP,33062/TCP,33060/TCP            2d20h
default                service/my-db-pxc-db-pxc-unready                  ClusterIP      None             <none>        3306/TCP,33062/TCP,33060/TCP            2d20h
default     statefulset.apps/my-db-pxc-db-haproxy            0/3     2d20h
default     statefulset.apps/my-db-pxc-db-pxc                2/3     2d20h

root@kubernetes-master-ubuntu:~# k version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

root@kubernetes-master-ubuntu:~# helm version
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.17.13"}
root@kubernetes-master-ubuntu:~#
2

@Beci_Roboz I need to see your commands which you used to uninstall the operator and cluster.

I have used the following commands and everything works.

❯ kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
my-db-pxc-db-haproxy-0                      2/2     Running   0          4m15s
my-db-pxc-db-haproxy-1                      2/2     Running   0          3m9s
my-db-pxc-db-haproxy-2                      2/2     Running   0          2m49s
my-db-pxc-db-pxc-0                          3/3     Running   0          4m15s
my-db-pxc-db-pxc-1                          3/3     Running   0          3m15s
my-db-pxc-db-pxc-2                          3/3     Running   0          2m14s
my-operator-pxc-operator-6f4984b6dc-g2nvk   1/1     Running   0          12m

❯ helm uninstall my-db 
❯ helm uninstall my-operator

❯ kubectl get pods
No resources found in ps2 namespace.
3

I guess I used these, but I don’t remember well, I try to do that again.

Thanks: Bela

4

Dear @Slava_Sarzhan,

I completely messed things up. Because I couldn’t use helm uninstall I tried to do this:

But I didn’t find deploy/cr.yaml file so I run this from here:
~/.cache/helm/repository/pxc-db-1.11.0.tgz/utar://pxc-db/crds/crd.yaml
kubectl delete -f crds.yaml
It was freeze at a point, so I restarted all nodes :frowning:

But now I get this message during the reinstall process:

root@kubernetes-master-ubuntu:~# helm install my-db percona/pxc-db
Error: INSTALLATION FAILED: cannot re-use a name that is still in use

root@kubernetes-master-ubuntu:~# helm install my-db2 percona/pxc-db
Error: INSTALLATION FAILED: create not allowed while custom resource definition is terminating

Is there any (manual) method with which I can remove everything related to percona(operator) from the k8s?

Thank you so much!

5

I try to do uninstall and reinstall process without success :frowning:

root@kubernetes-master-ubuntu:~/.cache/helm/repository# helm list -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
ingress-nginx   default         2               2022-08-18 02:10:46.471275227 +0000 UTC deployed        ingress-nginx-4.2.1     1.3.0
my-db2          default         1               2022-10-02 20:18:46.426932839 +0000 UTC failed          pxc-db-1.11.5           1.11.0
my-op           default         1               2022-10-02 20:08:31.26771953 +0000 UTC  deployed        pxc-operator-1.11.1     1.11.0

root@kubernetes-master-ubuntu:~/.cache/helm/repository# helm uninstall my-db2
release "my-db2" uninstalled

root@kubernetes-master-ubuntu:~/.cache/helm/repository# helm uninstall my-op
release "my-op" uninstalled

root@kubernetes-master-ubuntu:~/.cache/helm/repository# helm list -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
ingress-nginx   default         2               2022-08-18 02:10:46.471275227 +0000 UTC deployed        ingress-nginx-4.2.1     1.3.0

root@kubernetes-master-ubuntu:~/.cache/helm/repository# k get all -ALL | grep pxc

root@kubernetes-master-ubuntu:~/.cache/helm/repository# helm install my-op percona/pxc-operator
NAME: my-op
LAST DEPLOYED: Sun Oct  2 20:40:13 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. pxc-operator deployed.
  If you would like to deploy an pxc-cluster set cluster.enabled to true in values.yaml
  Check the pxc-operator logs
    export POD=$(kubectl get pods -l app.kubernetes.io/name=pxc-operator --namespace default --output name)
    kubectl logs $POD --namespace=default

root@kubernetes-master-ubuntu:~/.cache/helm/repository# helm install my-db percona/pxc-db
Error: INSTALLATION FAILED: create not allowed while custom resource definition is terminating
6

I could remove the problematic “custom resource definition”-s on this way:

root@kubernetes-master-ubuntu:~# kubectl get crd
NAME                                           CREATED AT
...
perconaxtradbclusterbackups.pxc.percona.com    2022-10-02T20:08:30Z
perconaxtradbclusterrestores.pxc.percona.com   2022-10-02T20:08:30Z
perconaxtradbclusters.pxc.percona.com          2022-09-29T20:47:29Z
...

root@kubernetes-master-ubuntu:~# k delete crd perconaxtradbclusterbackups.pxc.percona.com
customresourcedefinition.apiextensions.k8s.io "perconaxtradbclusterbackups.pxc.percona.com" deleted

root@kubernetes-master-ubuntu:~# k delete crd perconaxtradbclusterrestores.pxc.percona.com
customresourcedefinition.apiextensions.k8s.io "perconaxtradbclusterrestores.pxc.percona.com" deleted

root@kubernetes-master-ubuntu:~# k delete crd perconaxtradbclusters.pxc.percona.com
customresourcedefinition.apiextensions.k8s.io "perconaxtradbclusters.pxc.percona.com" deleted

The last deletion was freeze, so I manually edited its configuration and I removed the finalizer rows

root@kubernetes-master-ubuntu:~# k edit  crd perconaxtradbclusters.pxc.percona.com
customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com edited
root@kubernetes-master-ubuntu:~# k delete crd perconaxtradbclusters.pxc.percona.com
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "perconaxtradbclusters.pxc.percona.com" not found

After the previous I could reinstall the Percona(K8S):

root@kubernetes-master-ubuntu:~# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "percona" chart repository
...Successfully got an update from the "rook-release" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
root@kubernetes-master-ubuntu:~# helm install my-op percona/pxc-operator
NAME: my-op
LAST DEPLOYED: Sun Oct  2 21:09:38 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. pxc-operator deployed.
  If you would like to deploy an pxc-cluster set cluster.enabled to true in values.yaml
  Check the pxc-operator logs
    export POD=$(kubectl get pods -l app.kubernetes.io/name=pxc-operator --namespace default --output name)
    kubectl logs $POD --namespace=default
root@kubernetes-master-ubuntu:~# helm install my-db percona/pxc-db
NAME: my-db
LAST DEPLOYED: Sun Oct  2 21:09:49 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. To get a MySQL prompt inside your new cluster you can run:

    ROOT_PASSWORD=`kubectl -n default get secrets my-db-pxc-db -o jsonpath="{.data.root}" | base64 --decode`
    kubectl -n default exec -ti \
      my-db-pxc-db-pxc-0 -- mysql -uroot -p"$ROOT_PASSWORD"

2. To connect an Application running in the same Kubernetes cluster you can connect with:


$ kubectl run -i --tty --rm percona-client --image=percona --restart=Never \
  -- mysql -h my-db-pxc-db-proxysql.default.svc.cluster.local -uroot -p"$ROOT_PASSWORD"

I hope everything is correct now!

Thanks: Bela

7

As a check I tried to uninstall the Percona again:

root@kubernetes-master-ubuntu:~# helm list -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
ingress-nginx   default         2               2022-08-18 02:10:46.471275227 +0000 UTC deployed        ingress-nginx-4.2.1     1.3.0
my-db           default         1               2022-10-02 21:09:49.24400466 +0000 UTC  deployed        pxc-db-1.11.5           1.11.0
my-op           default         1               2022-10-02 21:09:38.940941394 +0000 UTC deployed        pxc-operator-1.11.1     1.11.0

root@kubernetes-master-ubuntu:~# helm uninstall my-db
release "my-db" uninstalled

root@kubernetes-master-ubuntu:~# helm uninstall my-op
release "my-op" uninstalled

But These remains in the system:

root@kubernetes-master-ubuntu:~# k get all -ALL | grep pxc
default                pod/my-db-pxc-db-haproxy-0                             1/2     Running   1 (96s ago)     10m
default                pod/my-db-pxc-db-haproxy-1                             1/2     Running   1 (91s ago)     9m31s
default                pod/my-db-pxc-db-haproxy-2                             0/2     Pending   0               9m11s
default                service/my-db-pxc-db-haproxy                      ClusterIP      10.105.18.161    <none>        3306/TCP,3309/TCP,33062/TCP,33060/TCP   10m
default                service/my-db-pxc-db-haproxy-replicas             ClusterIP      10.97.24.32      <none>        3306/TCP                                10m
default                service/my-db-pxc-db-pxc                          ClusterIP      None             <none>        3306/TCP,33062/TCP,33060/TCP            10m
default                service/my-db-pxc-db-pxc-unready                  ClusterIP      None             <none>        3306/TCP,33062/TCP,33060/TCP            10m
default     statefulset.apps/my-db-pxc-db-haproxy            0/3     10m

root@kubernetes-master-ubuntu:~# k get crd
NAME                                           CREATED AT
...
perconaxtradbclusterbackups.pxc.percona.com    2022-10-02T21:09:36Z
perconaxtradbclusterrestores.pxc.percona.com   2022-10-02T21:09:36Z
perconaxtradbclusters.pxc.percona.com          2022-10-02T21:09:36Z
...

where did i mess something up?

Thanks: Bela

1 Like
8

Finally the problem solved, I guess I had to wait more for K8S to delete the pods and related things.

2 Likes
9

@Beci_Roboz yes, it is not 5 seconds things, you need to wait a little bit. Do you need any additional help regarding this question ?

2 Likes
10

No, you are really kind, thank you so much!

2 Likes