Hello,
I trying to upgrade my prod cluster on Kubernetes to version 1.10.0.
So I applied all crds and now I tryed to recreate the first node - but it is not starting anymore.
I get following message in the logs:
+ grep -v wsrep_sst_auth /etc/mysql/node.cnf
++ cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
cat: /var/run/secrets/kubernetes.io/serviceaccount/namespace: No such file or directory
+ POD_NAMESPACE=
So i look in the manifest and compares the new one with the old one:
Here is a diff between two yamls:
5,8c5,8
< cni.projectcalico.org/containerID: 862b04e92d79a229d9f392eefea58091fc3502e555f880b4fa4eaaa232408253
< cni.projectcalico.org/podIP: 10.42.244.151/32
< cni.projectcalico.org/podIPs: 10.42.244.151/32
< percona.com/configuration-hash: 1545f67177e26d53feff59304ced4b38
---
> cni.projectcalico.org/containerID: 195167b91394bdcf549b3027dd7f009662a1ec48a650dc668e712f58e1d0b799
> cni.projectcalico.org/podIP: 10.42.159.229/32
> cni.projectcalico.org/podIPs: 10.42.159.229/32
> percona.com/configuration-hash: 5a0596201f198c11bdbf4e4e38ac06aa
11c11
< creationTimestamp: "2021-12-16T10:30:24Z"
---
> creationTimestamp: "2021-12-24T09:01:27Z"
19,20c19,20
< controller-revision-hash: mysql-cluster-prod-px-pxc-5678fc4c87
< statefulset.kubernetes.io/pod-name: mysql-cluster-prod-px-pxc-0
---
> controller-revision-hash: mysql-cluster-prod-px-pxc-b5cb7988c
> statefulset.kubernetes.io/pod-name: mysql-cluster-prod-px-pxc-2
257a258,265
> k:{"name":"PMM_AGENT_SIDECAR"}:
> .: {}
> f:name: {}
> f:value: {}
> k:{"name":"PMM_AGENT_SIDECAR_SLEEP"}:
> .: {}
> f:name: {}
> f:value: {}
625c633
< time: "2021-12-16T10:30:24Z"
---
> time: "2021-12-24T09:01:27Z"
636c644
< time: "2021-12-16T10:30:58Z"
---
> time: "2021-12-24T09:01:32Z"
645a654,655
> f:message: {}
> f:reason: {}
657a668,669
> f:message: {}
> f:reason: {}
667c679
< k:{"ip":"10.42.244.151"}:
---
> k:{"ip":"10.42.159.229"}:
673,674c685,686
< time: "2021-12-16T10:33:54Z"
< name: mysql-cluster-prod-px-pxc-0
---
> time: "2021-12-24T09:01:44Z"
> name: mysql-cluster-prod-px-pxc-2
683,684c695,696
< resourceVersion: "145412352"
< uid: e019a4db-a854-43df-9a04-213eecb5c2b0
---
> resourceVersion: "151478105"
> uid: de45be5b-8492-4c98-8807-d28d27eb81b1
778c790
< pmm-admin add $(DB_TYPE) $(PMM_ADMIN_CUSTOM_PARAMS) --skip-connection-check --metrics-mode=push --username=$(DB_USER) --password=$(DB_PASSWORD) --cluster=$(CLUSTER_NAME) --service-name=$(PMM_AGENT_SETUP_NODE_NAME) --host=$(POD_NAME) --port=$(DB_PORT) $(DB_ARGS);
---
> pmm-admin add $(DB_TYPE) $(PMM_ADMIN_CUSTOM_PARAMS) --skip-connection-check --metrics-mode=push --username=$(DB_USER) --password=$(DB_PASSWORD) --cluster=$(CLUSTER_NAME) --service-name=$(PMM_AGENT_SETUP_NODE_NAME) --host=$(POD_NAME) --port=$(DB_PORT) $(DB_ARGS);
779a792,795
> - name: PMM_AGENT_SIDECAR
> value: "true"
> - name: PMM_AGENT_SIDECAR_SLEEP
> value: "5"
784c800
< image: percona/pmm-client:2.18.0
---
> image: percona/pmm-client:2.23.0
822,823c838,839
< cpu: "0"
< memory: 1G
---
> cpu: 300m
> memory: 150M
829,831d844
< - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
< name: kube-api-access-tksmq
< readOnly: true
849c862
< image: percona/percona-xtradb-cluster-operator:1.9.0-logcollector
---
> image: percona/percona-xtradb-cluster-operator:1.10.0-logcollector
854,855c867,868
< cpu: "0"
< memory: 1G
---
> cpu: 200m
> memory: 100M
861,863d873
< - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
< name: kube-api-access-tksmq
< readOnly: true
874c884
< image: percona/percona-xtradb-cluster-operator:1.9.0-logcollector
---
> image: percona/percona-xtradb-cluster-operator:1.10.0-logcollector
879,880c889,890
< cpu: "0"
< memory: 1G
---
> cpu: 200m
> memory: 100M
886,888d895
< - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
< name: kube-api-access-tksmq
< readOnly: true
975c982
< memory: 1G
---
> memory: 1500M
995,997d1001
< - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
< name: kube-api-access-tksmq
< readOnly: true
1000c1004
< hostname: mysql-cluster-prod-px-pxc-0
---
> hostname: mysql-cluster-prod-px-pxc-2
1006c1010
< image: percona/percona-xtradb-cluster-operator:1.9.0
---
> image: percona/percona-xtradb-cluster-operator:1.10.0
1012c1016
< memory: 1G
---
> memory: 1500M
1018,1021c1022
< - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
< name: kube-api-access-tksmq
< readOnly: true
< nodeName: docker04
---
> nodeName: docker01
1046c1047
< claimName: datadir-mysql-cluster-prod-px-pxc-0
---
> claimName: datadir-mysql-cluster-prod-px-pxc-2
1079,1096d1079
< - name: kube-api-access-tksmq
< projected:
< defaultMode: 420
< sources:
< - serviceAccountToken:
< expirationSeconds: 3607
< path: token
< - configMap:
< items:
< - key: ca.crt
< path: ca.crt
< name: kube-root-ca.crt
< - downwardAPI:
< items:
< - fieldRef:
< apiVersion: v1
< fieldPath: metadata.namespace
< path: namespace
1100c1083
< lastTransitionTime: "2021-12-16T10:31:10Z"
---
> lastTransitionTime: "2021-12-24T09:01:36Z"
1104,1105c1087,1090
< lastTransitionTime: "2021-12-16T10:33:54Z"
< status: "True"
---
> lastTransitionTime: "2021-12-24T09:01:28Z"
> message: 'containers with unready status: [pxc]'
> reason: ContainersNotReady
> status: "False"
1108,1109c1093,1096
< lastTransitionTime: "2021-12-16T10:33:54Z"
< status: "True"
---
> lastTransitionTime: "2021-12-24T09:01:28Z"
> message: 'containers with unready status: [pxc]'
> reason: ContainersNotReady
> status: "False"
1112c1099
< lastTransitionTime: "2021-12-16T10:30:24Z"
---
> lastTransitionTime: "2021-12-24T09:01:27Z"
1116,1118c1103,1105
< - containerID: docker://c87240648ee51cb395f576a4ad027d63c2554bf9045016baa85beeaf90668724
< image: percona/percona-xtradb-cluster-operator:1.9.0-logcollector
< imageID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:efb0b84fd5b5231c041579284baaa74a849c1bb0bc7659432925497f88e5bffa
---
> - containerID: docker://cc0d8bd54b77d81600fc9dc53dbc4f51e484169fe07e531a9dcb922c92021b64
> image: percona/percona-xtradb-cluster-operator:1.10.0-logcollector
> imageID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:d4fd89c4d1a2c880bb85d405c053ba8d70caf673e91371c0bfd4aa8261ef2ecc
1126,1129c1113,1116
< startedAt: "2021-12-16T10:31:42Z"
< - containerID: docker://47858fbcc667d55c9a8a6b4dcc556737e20cac534280d9db97f02f4437b1cee8
< image: percona/percona-xtradb-cluster-operator:1.9.0-logcollector
< imageID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:efb0b84fd5b5231c041579284baaa74a849c1bb0bc7659432925497f88e5bffa
---
> startedAt: "2021-12-24T09:01:42Z"
> - containerID: docker://d082233229bb3b94aee2ce4c70b8001095e93fd47e112d8ec50cca51c4552ca6
> image: percona/percona-xtradb-cluster-operator:1.10.0-logcollector
> imageID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:d4fd89c4d1a2c880bb85d405c053ba8d70caf673e91371c0bfd4aa8261ef2ecc
1137,1140c1124,1127
< startedAt: "2021-12-16T10:31:40Z"
< - containerID: docker://a8dd2adcb6445822255c414a11e306c0e7bd42374f24a5e1db98778447b12219
< image: percona/pmm-client:2.18.0
< imageID: docker-pullable://percona/pmm-client@sha256:4fd309bf3d9336e9bf853ec5e9b578562c4a69cbc577bf1c2511a3b135922830
---
> startedAt: "2021-12-24T09:01:40Z"
> - containerID: docker://2036a9e03a450e491022a881e741df371ada01ebae11f192c7e96ec96d62a8a2
> image: percona/pmm-client:2.23.0
> imageID: docker-pullable://percona/pmm-client@sha256:8fa0e45f740fa8564cbfbdf5d9a5507a07e331f8f40ea022d3a64d7278478eac
1148,1149c1135,1136
< startedAt: "2021-12-16T10:31:12Z"
< - containerID: docker://443923c333a9085ba0e77b19892fcdd0bec36903316a101f7ba81cfed918295b
---
> startedAt: "2021-12-24T09:01:38Z"
> - containerID: docker://6848a92502c72742bf3ce2162c812aa3427879084c5086fd9e414c9cfacb7f2f
1152c1139,1145
< lastState: {}
---
> lastState:
> terminated:
> containerID: docker://6848a92502c72742bf3ce2162c812aa3427879084c5086fd9e414c9cfacb7f2f
> exitCode: 1
> finishedAt: "2021-12-24T09:33:49Z"
> reason: Error
> startedAt: "2021-12-24T09:33:47Z"
1154,1156c1147,1149
< ready: true
< restartCount: 0
< started: true
---
> ready: false
> restartCount: 11
> started: false
1158,1160c1151,1154
< running:
< startedAt: "2021-12-16T10:32:51Z"
< hostIP: 172.25.31.243
---
> waiting:
> message: back-off 5m0s restarting failed container=pxc pod=mysql-cluster-prod-px-pxc-2_mysql-cluster(de45be5b-8492-4c98-8807-d28d27eb81b1)
> reason: CrashLoopBackOff
> hostIP: 172.25.31.240
1162,1164c1156,1158
< - containerID: docker://1d9cc9730a330e3f2fa5cbd856fcf25a5e924faa12b72ba3e8e23fe4f922b377
< image: percona/percona-xtradb-cluster-operator:1.9.0
< imageID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:3783173df8b833028184de686cc98890179a249c7ed5cde8391cdfd751cf990d
---
> - containerID: docker://9081ef451b8e92aeeff0f7ffb026b40cd0ca97f564626b8fd7e960da3aeaca6d
> image: percona/percona-xtradb-cluster-operator:1.10.0
> imageID: docker-pullable://percona/percona-xtradb-cluster-operator@sha256:73d2266258b700a691db6196f4b5c830845d34d57bdef5be5ffbd45e88407309
1171c1165
< containerID: docker://1d9cc9730a330e3f2fa5cbd856fcf25a5e924faa12b72ba3e8e23fe4f922b377
---
> containerID: docker://9081ef451b8e92aeeff0f7ffb026b40cd0ca97f564626b8fd7e960da3aeaca6d
1173c1167
< finishedAt: "2021-12-16T10:31:10Z"
---
> finishedAt: "2021-12-24T09:01:35Z"
1175c1169
< startedAt: "2021-12-16T10:31:10Z"
---
> startedAt: "2021-12-24T09:01:35Z"
1177c1171
< podIP: 10.42.244.151
---
> podIP: 10.42.159.229
1179c1173
< - ip: 10.42.244.151
---
> - ip: 10.42.159.229
1181c1175
< startTime: "2021-12-16T10:30:24Z"
---
> startTime: "2021-12-24T09:01:28Z"
The new yaml misses the kube-api-access and Namespace mounts.
Here:
- name: kube-api-access-tksmq
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
Why is it missing? I used the helm charts for deploy the cluster - how can I fix that?
I like to upgrade the cluster without any downtime.