# I am getting CrashLookBackOff error for Percona XtraDB Cluster 8.0.20 in Kubernetes

**URL:** <https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253>\
**Category:** Percona Distribution for MySQL\
**Tags:** percona\
**Created:** [November 9, 2020, 7:08am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253 "2020-11-09T07:08:01Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Laxminarayana](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@Laxminarayana](https://forums.percona.com/u/Laxminarayana)\
**Post date:** [November 9, 2020, 7:08am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/1 "2020-11-09T07:08:01Z")

</div>

I am getting CrashLookBackOff error for Percona XtraDB Cluster 8.0.20 in Kubernetes. We are deploying through helm chart. Is it stable version for helm chart, Can someone help on this.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [November 9, 2020, 2:22pm UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/2 "2020-11-09T14:22:23Z")

</div>

Hi @Laxminarayana Can’t help much if you don’t provide any logs or more detailed information. Have you looked at the mysql error log on the pod with the issue? Have you looked at other K8S logs? Please provide.

---

<div class="post-metadata">

**Author:** ![Laxminarayana](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@Laxminarayana](https://forums.percona.com/u/Laxminarayana)\
**Post date:** [November 11, 2020, 3:33am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/3 "2020-11-11T03:33:11Z")

</div>

@matthewb Thanks for response. Please find below manifest file and generated logs.

apiVersion: v1

kind: Pod

metadata:

creationTimestamp: ‘2020-11-11T08:48:29Z’

generateName: percona-xtradb-cluster-pxc-

labels:

app: percona-xtradb-cluster-pxc

chart: percona-xtradb-cluster-1.0.3

controller-revision-hash: percona-xtradb-cluster-pxc-c4cb996c8

heritage: Tiller

release: percona-xtradb-cluster

[statefulset.kubernetes.io/pod-name:](http://statefulset.kubernetes.io/pod-name:) percona-xtradb-cluster-pxc-0

name: percona-xtradb-cluster-pxc-0

namespace: default

ownerReferences:

- apiVersion: apps/v1

blockOwnerDeletion: true

controller: true

kind: StatefulSet

name: percona-xtradb-cluster-pxc

uid: ab89b4bf-23fa-11eb-9a3d-0242376f5115

resourceVersion: ‘33575’

selfLink: /api/v1/namespaces/default/pods/percona-xtradb-cluster-pxc-0

uid: ab958682-23fa-11eb-9a3d-0242376f5115

spec:

containers:

- command:

env:

```
- name: MYSQL_ROOT_PASSWORD

 valueFrom:

  secretKeyRef:

   key: mysql-root-password

   name: percona-xtradb-cluster-pxc

- name: MYSQL_PASSWORD

 valueFrom:

  secretKeyRef:

   key: mysql-password

   name: percona-xtradb-cluster-pxc

- name: XTRABACKUP_PASSWORD

 valueFrom:

  secretKeyRef:

   key: xtrabackup-password

   name: percona-xtradb-cluster-pxc

- name: MYSQL_USER

 valueFrom:

  secretKeyRef:

   key: mysql-user

   name: percona-xtradb-cluster-pxc

- name: MYSQL_DATABASE

 valueFrom:

  secretKeyRef:

   key: mysql-database

   name: percona-xtradb-cluster-pxc

- name: ALLOW_ROOT_FROM

 value: '%'

- name: CLUSTER_NAME

 value: percona-xtradb-cluster-pxc

- name: SHORT_CLUSTER_NAME

 value: percona-xtradb-cluster-pxc

- name: K8S_SERVICE_NAME

 value: percona-xtradb-cluster-pxc-repl

- name: PXC_STRICT_MODE

 value: MASTER

- name: DEBUG

 value: 'true'

```

image: ‘local/percona-xtradb-cluster-server:8.0.20-11.3’

imagePullPolicy: Never

livenessProbe:

```
exec:

 command:

  - /bin/bash

  - '-c'

  - mysqladmin ping || test -e /var/lib/mysql/sst_in_progress

failureThreshold: 3

initialDelaySeconds: 30

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 2

```

name: database

ports:

```
- containerPort: 3306

 name: mysql

 protocol: TCP

- containerPort: 4567

 name: galera-repl

 protocol: TCP

- containerPort: 4568

 name: state-transfer

 protocol: TCP

- containerPort: 4444

 name: state-snapshot

 protocol: TCP

```

readinessProbe:

```
exec:

 command:

  - mysql

  - '-h'

  - 127.0.0.1

  - '-e'

  - SELECT 1

failureThreshold: 3

initialDelaySeconds: 30

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 2

```

resources: {}

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

volumeMounts:

```
- mountPath: /var/lib/mysql

 name: mysql-data

- mountPath: /startup-scripts

 name: mysql-startup-scripts

- mountPath: /etc/mysql/conf.d

 name: mysql-config-files

- mountPath: /root

 name: slash-root

- mountPath: /var/log

 name: var-log

- mountPath: /backup-data

 name: backup-data

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

 name: default-token-hl4xt

 readOnly: true

```

- command:

image: ‘local/debian:jessie’

imagePullPolicy: Never

name: audit-logs

resources: {}

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

volumeMounts:

```
- mountPath: /var/log

 name: var-log

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

 name: default-token-hl4xt

 readOnly: true

```

dnsPolicy: ClusterFirst

enableServiceLinks: true

hostname: percona-xtradb-cluster-pxc-0

initContainers:

- command:

image: ‘local/debian:jessie’

imagePullPolicy: Never

name: init

resources: {}

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

volumeMounts:

```
- mountPath: /var/lib/mysql

 name: mysql-data

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

 name: default-token-hl4xt

 readOnly: true

```

nodeName: minikube

priority: 0

restartPolicy: Always

schedulerName: default-scheduler

securityContext:

fsGroup: 1001

serviceAccount: default

serviceAccountName: default

subdomain: percona-xtradb-cluster-pxc

terminationGracePeriodSeconds: 30

tolerations:

- effect: NoExecute

key: [node.kubernetes.io/not-ready](http://node.kubernetes.io/not-ready)

operator: Exists

tolerationSeconds: 300

- effect: NoExecute

key: [node.kubernetes.io/unreachable](http://node.kubernetes.io/unreachable)

operator: Exists

tolerationSeconds: 300

volumes:

- name: mysql-data

persistentVolumeClaim:

```
claimName: mysql-data-percona-xtradb-cluster-pxc-0

```

- emptyDir: {}

name: slash-root

- emptyDir: {}

name: var-log

- configMap:

name: mysql-config-files

- configMap:

name: mysql-startup-scripts

- emptyDir: {}

name: backup-data

- emptyDir: {}

name: rsync-disk

- name: default-token-hl4xt

secret:

```
defaultMode: 420

secretName: default-token-hl4xt

```

status:

conditions:

- lastProbeTime: null

lastTransitionTime: ‘2020-11-11T08:48:31Z’

status: ‘True’

type: Initialized

- lastProbeTime: null

lastTransitionTime: ‘2020-11-11T08:48:29Z’

message: ‘containers with unready status: [database]’

reason: ContainersNotReady

status: ‘False’

type: Ready

- lastProbeTime: null

lastTransitionTime: ‘2020-11-11T08:48:29Z’

message: ‘containers with unready status: [database]’

reason: ContainersNotReady

status: ‘False’

type: ContainersReady

- lastProbeTime: null

lastTransitionTime: ‘2020-11-11T08:48:29Z’

status: ‘True’

type: PodScheduled

containerStatuses:

- containerID: \>-

image: ‘debian:jessie’

imageID: \>-

```
docker-pullable://debian@sha256:62459e0ef769f129957fed32a1a4512b1415f09b794454acf012c82412ddf834

```

lastState:

```
terminated:

 containerID: >-

  docker://60621f5d5c1d56ac759c36a2cb7c7c7b72e0989e443510ba62f02ab98023efa5

 exitCode: 1

 finishedAt: '2020-11-11T08:48:48Z'

 reason: Error

 startedAt: '2020-11-11T08:48:48Z'

```

name: audit-logs

ready: true

restartCount: 3

state:

```
running:

 startedAt: '2020-11-11T08:49:15Z'

```

- containerID: \>-

image: ‘local/percona-xtradb-cluster-server:8.0.20-11.3’

imageID: \>-

```
docker-pullable://percona/percona-xtradb-cluster@sha256:08ef5b75e12dbbf7fba35c9b6598908e139d9d18227b05eaa06b55b18f574754

```

lastState:

```
terminated:

 containerID: >-

  docker://eb49f2b69500dc3159bce2cc7bc32a94e65a35e560bf0191b0f44180918944b8

 exitCode: 2

 finishedAt: '2020-11-11T08:55:05Z'

 reason: Error

 startedAt: '2020-11-11T08:55:01Z'

```

name: database

ready: false

restartCount: 6

state:

```
waiting:

 message: >-

  Back-off 5m0s restarting failed container=database

  pod=percona-xtradb-cluster-pxc-0_default(ab958682-23fa-11eb-9a3d-0242376f5115)

 reason: CrashLoopBackOff

```

hostIP: 192.168.49.2

initContainerStatuses:

- containerID: \>-

image: ‘debian:jessie’

imageID: \>-

```
docker-pullable://debian@sha256:62459e0ef769f129957fed32a1a4512b1415f09b794454acf012c82412ddf834

```

lastState: {}

name: init

ready: true

restartCount: 0

state:

```
terminated:

 containerID: >-

  docker://97caad90ea2f251639d4eabc23a44a91a6a328ae2baa2d90e776bf1f142a1948

 exitCode: 0

 finishedAt: '2020-11-11T08:48:30Z'

 reason: Completed

 startedAt: '2020-11-11T08:48:30Z'

```

phase: Running

podIP: 172.17.0.10

qosClass: BestEffort

startTime: ‘2020-11-11T08:48:29Z’

---

<div class="post-metadata">

**Author:** ![Laxminarayana](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@Laxminarayana](https://forums.percona.com/u/Laxminarayana)\
**Post date:** [November 11, 2020, 3:33am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/4 "2020-11-11T03:33:30Z")

</div>

Logs:

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$ kubectl logs percona-xtradb-cluster-pxc-0 -c database -n default

- . /startup-scripts/functions.sh

++ hostname -i

++ awk ’ { print $1 } ’

- ipaddr=172.17.0.10

++ hostname

- hostname=percona-xtradb-cluster-pxc-0

- echo ‘I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10’

I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10

- ‘[’ ‘’ = - ‘]’

++ resolveip -s percona-xtradb-cluster-pxc-repl

/startup-scripts/entrypoint.sh: line 19: resolveip: command not found

++ echo ‘’

- cluster\_join=

- [[-z ‘’]]

- echo ‘I am the Primary Node’

I am the Primary Node

- init\_mysql

- DATADIR=/var/lib/mysql

- ‘[’ ‘!’ -e /var/lib/mysql/mysql ‘]’

- write\_password\_file

- [[-n ohchohtaigheecaemeeDaivo3wai3Dae]]

- cat

- exec mysqld --user=mysql --wsrep\_cluster\_name=percona-xtradb-cluster-pxc --wsrep\_node\_name=percona-xtradb-cluster-pxc-0 --wsrep\_cluster\_address=gcomm:// --wsrep\_sst\_method=xtrabackup-v2 --wsrep\_sst\_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar --wsrep\_node\_address=172.17.0.10 --pxc\_strict\_mode=MASTER

2020-11-11T08:55:01.770215Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes and --log-slow-slave-statements have no effect if --slow-query-log is not set

2020-11-11T08:55:01.777677Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11.3) starting as process 1

2020-11-11T08:55:01.786110Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/libgalera\_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1

2020-11-11T08:55:01.786220Z 0 [Note] [MY-000000] [Galera] wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’

2020-11-11T08:55:01.791275Z 0 [Note] [MY-000000] [Galera] wsrep\_load(): Galera 4.3(r752664d) by Codership Oy [info@codership.com](mailto:info@codership.com) loaded successfully.

2020-11-11T08:55:01.791490Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.

2020-11-11T08:55:01.794837Z 0 [Note] [MY-000000] [Galera] Found saved state: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe:-1, safe\_to\_bootstrap: 1

2020-11-11T08:55:01.796754Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:

Version: 2

UUID: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe

Seqno: -1 - -1

Offset: -1

Synced: 0

2020-11-11T08:55:01.797015Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe, offset: -1

2020-11-11T08:55:01.797560Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% ( 0/134217752 bytes) complete.

2020-11-11T08:55:01.902603Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan…100.0% (134217752/134217752 bytes) complete.

2020-11-11T08:55:01.902810Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1

2020-11-11T08:55:01.902987Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% ( 0/184 bytes) complete.

2020-11-11T08:55:01.903113Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan…100.0% (184/184 bytes) complete.

2020-11-11T08:55:01.903239Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers

2020-11-11T08:55:01.903357Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 0/134217728

2020-11-11T08:55:01.904638Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base\_dir = /var/lib/mysql/; base\_host = 172.17.0.10; base\_port = 4567; cert.log\_conflicts = no; cert.optimistic\_pa = no; debug = no; evs.auto\_evict = 0; evs.delay\_margin = PT1S; evs.delayed\_keep\_period = PT30S; evs.inactive\_check\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 10; evs.stats\_report\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 4; evs.view\_forget\_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze\_purge\_at\_seqno = -1; gcache.keep\_pages\_count = 0; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = galera.cache; gcache.page\_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread\_prio = ; gcs.fc\_debug = 0; gcs.fc\_factor = 1.0; gcs.fc\_limit = 100; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignore\_quorum = false; pc.ignore\_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait\_prim = true; pc.wait\_prim\_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal\_read\_timeout = PT30S; repl.commit\_order = 3; repl.key\_format = FLAT8; repl.max\_ws\_size = 2147483647; repl.proto\_max = 10; socket.checksum = 2; socket.recv\_buf\_size = 212992; socket.ssl\_ca = ca.pem; socket.ssl\_cert = server-cert.pem; socket.ssl\_cipher = ; socket.ssl\_compression = YES; socket.ssl\_key = server-key.pem;

2020-11-11T08:55:01.928344Z 0 [Note] [MY-000000] [WSREP] Starting replication

2020-11-11T08:55:01.928686Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1

2020-11-11T08:55:01.929245Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1

2020-11-11T08:55:01.929642Z 0 [Note] [MY-000000] [Galera] protonet asio version 0

2020-11-11T08:55:01.930507Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.

2020-11-11T08:55:01.931623Z 0 [Note] [MY-000000] [Galera] initializing ssl context

2020-11-11T08:55:01.933007Z 0 [Note] [MY-000000] [Galera] backend: asio

2020-11-11T08:55:01.933399Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0

2020-11-11T08:55:01.934128Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

2020-11-11T08:55:01.934289Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

2020-11-11T08:55:01.935964Z 0 [Note] [MY-000000] [Galera] GMCast version 0

2020-11-11T08:55:01.936419Z 0 [Note] [MY-000000] [Galera] (95a57b03, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567

2020-11-11T08:55:01.936777Z 0 [Note] [MY-000000] [Galera] (95a57b03, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1

2020-11-11T08:55:01.937883Z 0 [Note] [MY-000000] [Galera] EVS version 1

2020-11-11T08:55:01.938344Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group ‘percona-xtradb-cluster-pxc’

2020-11-11T08:55:01.938707Z 0 [Note] [MY-000000] [Galera] start\_prim is enabled, turn off pc\_recovery

2020-11-11T08:55:01.940309Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 → 1

2020-11-11T08:55:01.940842Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 → 1

2020-11-11T08:55:01.941347Z 0 [Note] [MY-000000] [Galera] Node 95a57b03 state primary

2020-11-11T08:55:01.941834Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view (view\_id(PRIM,95a57b03,1)

memb {

```
95a57b03,0

}

```

joined {

```
}

```

left {

```
}

```

partitioned {

```
}

```

)

2020-11-11T08:55:01.942081Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk

2020-11-11T08:55:01.951665Z 0 [Note] [MY-000000] [Galera] gcomm: connected

2020-11-11T08:55:01.954121Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636

2020-11-11T08:55:01.955325Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)

2020-11-11T08:55:01.955859Z 0 [Note] [MY-000000] [Galera] Opened channel ‘percona-xtradb-cluster-pxc’

2020-11-11T08:55:01.960517Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my\_idx = 0, memb\_num = 1

2020-11-11T08:55:01.966240Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 95a9c4c2-23fb-11eb-81e9-d32431edcdd2

2020-11-11T08:55:01.969461Z 0 [Note] [MY-000000] [Galera] STATE\_EXCHANGE: sent state UUID: 95aaaa86-23fb-11eb-ab85-6adf96a771ab

2020-11-11T08:55:01.969909Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 95aaaa86-23fb-11eb-ab85-6adf96a771ab

2020-11-11T08:55:01.970784Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 95aaaa86-23fb-11eb-ab85-6adf96a771ab from 0 (percona-xtradb-cluster-pxc-0)

2020-11-11T08:55:01.971242Z 0 [Note] [MY-000000] [Galera] Quorum results:

```
version = 6,

component = PRIMARY,

conf_id = 0,

members = 1/1 (primary/total),

act_id = 0,

last_appl. = 0,

protocols = 2/10/4 (gcs/repl/appl),

vote policy= 0,

group UUID = 95a9c4c2-23fb-11eb-81e9-d32431edcdd2

```

2020-11-11T08:55:01.971756Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]

2020-11-11T08:55:01.972103Z 0 [Note] [MY-000000] [Galera] Restored state OPEN → JOINED (1)

2020-11-11T08:55:01.972518Z 0 [Note] [MY-000000] [Galera] Member 0.0 (percona-xtradb-cluster-pxc-0) synced with group.

2020-11-11T08:55:01.973065Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 1)

2020-11-11T08:55:01.973406Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1

2020-11-11T08:55:01.973968Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2

2020-11-11T08:55:01.978752Z 2 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered

2020-11-11T08:55:01.993141Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1

2020-11-11T08:55:01.994465Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)

2020-11-11T08:55:01.995337Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 95a57b03-23fb-11eb-ad58-c242f987fcfd

2020-11-11T08:55:01.996329Z 2 [Note] [MY-000000] [Galera] Server percona-xtradb-cluster-pxc-0 connected to cluster at position 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:1 with ID 95a57b03-23fb-11eb-ad58-c242f987fcfd

2020-11-11T08:55:01.997288Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected → connected

2020-11-11T08:55:01.998214Z 2 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.

2020-11-11T08:55:01.999240Z 2 [Note] [MY-000000] [Galera] ####### ST not required

2020-11-11T08:55:02.000427Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no

2020-11-11T08:55:02.001559Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-11T08:55:02.002496Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5

2020-11-11T08:55:02.003646Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 → 1

2020-11-11T08:55:02.004846Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-11T08:55:02.005996Z 2 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1

2020-11-11T08:55:02.008501Z 2 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped

2020-11-11T08:55:02.010571Z 2 [Note] [MY-000000] [Galera] GCache history reset: old(2fa7d286-23fb-11eb-b83b-231d9f8d2cbe:1 → 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:0

2020-11-11T08:55:02.221936Z 2 [Note] [MY-000000] [Galera] Flushing memory map to disk…

2020-11-11T08:55:02.438824Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno\_reset(): discarded 184 bytes

2020-11-11T08:55:02.440566Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno\_reset(): found 0/1 locked buffers

2020-11-11T08:55:02.442264Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 1

2020-11-11T08:55:02.445908Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1

2020-11-11T08:55:02.448208Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1

2020-11-11T08:55:02.452316Z 2 [Note] [MY-000000] [Galera] ================================================

View:

id: 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:1

status: primary

protocol\_version: 4

capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL\_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL\_READ, INCREMENTAL\_WS, UNORDERED, PREORDERED, STREAMING, NBO

final: no

own\_index: 0

members(1):

```
0: 95a57b03-23fb-11eb-ad58-c242f987fcfd, percona-xtradb-cluster-pxc-0

```

=================================================

2020-11-11T08:55:02.455216Z 2 [Note] [MY-000000] [WSREP] Server status change connected → joiner

2020-11-11T08:55:02.455775Z 2 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.

2020-11-11T08:55:02.456242Z 2 [Note] [MY-000000] [WSREP] Server status change joiner → initializing

2020-11-11T08:55:02.456734Z 2 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.

2020-11-11T08:55:02.463972Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO\_set\_mem\_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.

2020-11-11T08:55:02.473774Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2020-11-11T08:55:03.131444Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2020-11-11T08:55:03.586137Z 3 [Note] [MY-000000] [WSREP] wsrep\_init\_schema\_and\_SR (nil)

2020-11-11T08:55:03.626812Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully

2020-11-11T08:55:03.648069Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: ‘/var/lib/mysql/mysqlx.sock’ bind-address: ‘::’ port: 33060

2020-11-11T08:55:03.719538Z 0 [Note] [MY-000000] [WSREP] Before binlog recovery (wsrep position: a2555643-2338-11eb-9e47-97282c20ffa4:21)

2020-11-11T08:55:03.721102Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…

2020-11-11T08:55:03.738572Z 0 [System] [MY-010232] [Server] XA crash recovery finished.

2020-11-11T08:55:03.740355Z 0 [Note] [MY-000000] [WSREP] After binlog recovery (wsrep position: a2555643-2338-11eb-9e47-97282c20ffa4:21)

2020-11-11T08:55:03.793289Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

2020-11-11T08:55:03.802819Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location ‘/var/lib/mysql’ in the path is accessible to all OS users. Consider choosing a different directory.

2020-11-11T08:55:03.813542Z 0 [ERROR] [MY-000067] [Server] unknown variable ‘wsrep\_sst\_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar’.

2020-11-11T08:55:03.816571Z 0 [ERROR] [MY-010119] [Server] Aborting

2020-11-11T08:55:03.816745Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation

2020-11-11T08:55:05.817227Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → disconnecting

2020-11-11T08:55:05.817356Z 0 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.

2020-11-11T08:55:05.817445Z 0 [Note] [MY-000000] [Galera] Closing send monitor…

2020-11-11T08:55:05.817506Z 0 [Note] [MY-000000] [Galera] Closed send monitor.

2020-11-11T08:55:05.817570Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread

2020-11-11T08:55:05.817645Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread

2020-11-11T08:55:05.817740Z 1 [Note] [MY-000000] [WSREP] rollbacker thread exiting 1

2020-11-11T08:55:05.818348Z 2 [ERROR] [MY-000000] [Galera] Exception: State wait was interrupted

2020-11-11T08:55:05.818456Z 2 [ERROR] [MY-000000] [Galera] View callback failed. This is unrecoverable, restart required. (FATAL)

```
 at galera/src/replicator_smm.cpp:submit_view_info():2588

```

2020-11-11T08:55:05.818488Z 2 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()

2020-11-11T08:55:05.818523Z 2 [Note] [MY-000000] [Galera] mysqld: Terminated.

08:55:05 UTC - mysqld got signal 11 ;

Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Thread pointer: 0x7f91700008c0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong…

stack\_bottom = 7f918585bc60 thread\_stack 0x46000

2020-11-11T08:55:05.820295Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend

2020-11-11T08:55:05.820872Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 → 0

2020-11-11T08:55:05.820945Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view ((empty))

2020-11-11T08:55:05.821271Z 0 [Note] [MY-000000] [Galera] gcomm: closed

2020-11-11T08:55:05.821365Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.

2020-11-11T08:55:05.821495Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]

2020-11-11T08:55:05.821543Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.

2020-11-11T08:55:05.821584Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED → CLOSED (TO: 1)

2020-11-11T08:55:05.821630Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success

2020-11-11T08:55:05.822808Z 0 [Note] [MY-000000] [Galera] recv\_thread() joined.

2020-11-11T08:55:05.823081Z 0 [Note] [MY-000000] [Galera] Closing replication queue.

2020-11-11T08:55:05.823274Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

/usr/sbin/mysqld(my\_print\_stacktrace(unsigned char const\*, unsigned long)+0x3d) [0x1fd3a0d]

/usr/sbin/mysqld(handle\_fatal\_signal+0x303) [0x1166583]

/lib64/libpthread.so.0(+0xf630) [0x7f91a0042630]

/lib64/libc.so.6(abort+0x297) [0x7f919dee2bc7]

/usr/lib64/libgalera\_smm.so(+0x1e6c7) [0x7f91909ec6c7]

/usr/lib64/libgalera\_smm.so(+0x1ea54c) [0x7f9190bb854c]

/usr/lib64/libgalera\_smm.so(+0x1f8c51) [0x7f9190bc6c51]

/usr/lib64/libgalera\_smm.so(+0x1c8918) [0x7f9190b96918]

/usr/lib64/libgalera\_smm.so(+0x1c8b98) [0x7f9190b96b98]

/usr/lib64/libgalera\_smm.so(+0x1f0f60) [0x7f9190bbef60]

/usr/lib64/libgalera\_smm.so(+0x2103cc) [0x7f9190bde3cc]

/usr/sbin/mysqld(wsrep::wsrep\_provider\_v26::run\_applier(wsrep::high\_priority\_service\*)+0xe) [0x272c13e]

/usr/sbin/mysqld() [0x11a9b54]

/usr/sbin/mysqld(start\_wsrep\_THD+0x331) [0xebe6e1]

/usr/sbin/mysqld() [0x24e37e9]

/lib64/libpthread.so.0(+0x7ea5) [0x7f91a003aea5]

/lib64/libc.so.6(clone+0x6d) [0x7f919dfa98dd]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0): is an invalid pointer

Connection ID (thread ID): 2

Status: KILL\_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting

[http://www.percona.com/software/percona-xtradb-cluster/.](http://www.percona.com/software/percona-xtradb-cluster/.) You may find information

in the manual which will help you identify the cause of the crash.

Writing a core file

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [November 11, 2020, 11:57am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/5 "2020-11-11T11:57:14Z")

</div>

@LaxminarayanaPlease use the forum tools to format your output. Hard to read when you just copy/paste.

I can see the error right here in your logs;

```auto
2020-11-11T08:55:03.813542Z 0 [ERROR] [MY-000067] [Server] unknown variable 'wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar'.

2020-11-11T08:55:03.816571Z 0 [ERROR] [MY-010119] [Server] Aborting

```

Are you sure you are using the latest version of the operator? Looks like v1.6.0 is the current GA version. Please upgrade to that version of the operator before attempting again. If you are indeed on 1.6.0, then this is a serious bug in the operator that needs to be addressed. The parameter wsrep\_sst\_auth was removed in PXC8 and should not be present in the scripts.

---

<div class="post-metadata">

**Author:** ![Laxminarayana](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@Laxminarayana](https://forums.percona.com/u/Laxminarayana)\
**Post date:** [November 17, 2020, 8:05am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/6 "2020-11-17T08:05:39Z")

</div>

Thanks for your response. I have removed parameter wsrep\_sst\_auth but it wasn’t worked. We are using helm for deployment but I don’t see updated chart version for Percona Xtradb Cluster and also we are doing continues deployment using argo cd. Here I have attached generated manifest file. Please look into manifest file and help me to resolve this issue.

[manifest.zip](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/16e7b7871dc6ae944a4f71bd93a3720c772c295f.zip)

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [November 17, 2020, 10:06am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/7 "2020-11-17T10:06:37Z")

</div>

@Laxminarayana After removing that parameter, what do the mysql logs say?

---

<div class="post-metadata">

**Author:** ![Laxminarayana](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@Laxminarayana](https://forums.percona.com/u/Laxminarayana)\
**Post date:** [November 17, 2020, 10:12am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/8 "2020-11-17T10:12:16Z")

</div>

@matthewb Thanks for quick response. Here is the log details.

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$ kubectl logs percona-xtradb-cluster-pxc-0 -c database -n default

- . /startup-scripts/functions.sh

++ awk ’ { print $1 } ’

++ hostname -i

- ipaddr=172.17.0.10

++ hostname

- hostname=percona-xtradb-cluster-pxc-0

- echo ‘I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10’

I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10

- ‘[’ ‘’ = - ‘]’

++ resolveip -s percona-xtradb-cluster-pxc-repl

/startup-scripts/entrypoint.sh: line 19: resolveip: command not found

++ echo ‘’

- cluster\_join=

- [[-z ‘’]]

- echo ‘I am the Primary Node’

- init\_mysql

- DATADIR=/var/lib/mysql

- ‘[’ ‘!’ -e /var/lib/mysql/mysql ‘]’

I am the Primary Node

- write\_password\_file

- [[-n ohchohtaigheecaemeeDaivo3wai3Dae]]

- cat

- exec mysqld --user=mysql --wsrep\_cluster\_name=percona-xtradb-cluster-pxc --wsrep\_node\_name=percona-xtradb-cluster-pxc-0 --wsrep\_cluster\_address=gcomm:// --wsrep\_sst\_method=xtrabackup-v2

2020-11-17T13:31:45.018951Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes and --log-slow-slave-statements have no effect if --slow-query-log is not set

2020-11-17T13:31:45.020825Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11.3) starting as process 1

2020-11-17T13:31:45.032709Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/libgalera\_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1

2020-11-17T13:31:45.032923Z 0 [Note] [MY-000000] [Galera] wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’

2020-11-17T13:31:45.040668Z 0 [Note] [MY-000000] [Galera] wsrep\_load(): Galera 4.3(r752664d) by Codership Oy [info@codership.com](mailto:info@codership.com) loaded successfully.

2020-11-17T13:31:45.040843Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.

2020-11-17T13:31:45.042434Z 0 [Note] [MY-000000] [Galera] Found saved state: 801d098a-28d8-11eb-9842-5a65a898fbb3:-1, safe\_to\_bootstrap: 1

2020-11-17T13:31:45.043839Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:

Version: 2

UUID: 801d098a-28d8-11eb-9842-5a65a898fbb3

Seqno: -1 - -1

Offset: -1

Synced: 0

2020-11-17T13:31:45.044021Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 801d098a-28d8-11eb-9842-5a65a898fbb3, offset: -1

2020-11-17T13:31:45.044243Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% ( 0/134217752 bytes) complete.

2020-11-17T13:31:45.044485Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan…100.0% (134217752/134217752 bytes) complete.

2020-11-17T13:31:45.044624Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1

2020-11-17T13:31:45.044820Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% ( 0/184 bytes) complete.

2020-11-17T13:31:45.044967Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan…100.0% (184/184 bytes) complete.

2020-11-17T13:31:45.045119Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers

2020-11-17T13:31:45.045301Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 0/134217728

2020-11-17T13:31:45.046821Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base\_dir = /var/lib/mysql/; base\_host = 172.17.0.10; base\_port = 4567; cert.log\_conflicts = no; cert.optimistic\_pa = no; debug = no; evs.auto\_evict = 0; evs.delay\_margin = PT1S; evs.delayed\_keep\_period = PT30S; evs.inactive\_check\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 10; evs.stats\_report\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 4; evs.view\_forget\_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze\_purge\_at\_seqno = -1; gcache.keep\_pages\_count = 0; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = galera.cache; gcache.page\_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread\_prio = ; gcs.fc\_debug = 0; gcs.fc\_factor = 1.0; gcs.fc\_limit = 100; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignore\_quorum = false; pc.ignore\_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait\_prim = true; pc.wait\_prim\_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal\_read\_timeout = PT30S; repl.commit\_order = 3; repl.key\_format = FLAT8; repl.max\_ws\_size = 2147483647; repl.proto\_max = 10; socket.checksum = 2; socket.recv\_buf\_size = 212992; socket.ssl\_ca = ca.pem; socket.ssl\_cert = server-cert.pem; socket.ssl\_cipher = ; socket.ssl\_compression = YES; socket.ssl\_key = server-key.pem;

2020-11-17T13:31:45.063441Z 0 [Note] [MY-000000] [WSREP] Starting replication

2020-11-17T13:31:45.063687Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1

2020-11-17T13:31:45.063901Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1

2020-11-17T13:31:45.064162Z 0 [Note] [MY-000000] [Galera] protonet asio version 0

2020-11-17T13:31:45.064464Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.

2020-11-17T13:31:45.064696Z 0 [Note] [MY-000000] [Galera] initializing ssl context

2020-11-17T13:31:45.065167Z 0 [Note] [MY-000000] [Galera] backend: asio

2020-11-17T13:31:45.065414Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0

2020-11-17T13:31:45.065704Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

2020-11-17T13:31:45.065969Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

2020-11-17T13:31:45.066904Z 0 [Note] [MY-000000] [Galera] GMCast version 0

2020-11-17T13:31:45.067224Z 0 [Note] [MY-000000] [Galera] (3c5cd0f9, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567

2020-11-17T13:31:45.067466Z 0 [Note] [MY-000000] [Galera] (3c5cd0f9, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1

2020-11-17T13:31:45.068063Z 0 [Note] [MY-000000] [Galera] EVS version 1

2020-11-17T13:31:45.068586Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group ‘percona-xtradb-cluster-pxc’

2020-11-17T13:31:45.069217Z 0 [Note] [MY-000000] [Galera] start\_prim is enabled, turn off pc\_recovery

2020-11-17T13:31:45.071242Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 → 1

2020-11-17T13:31:45.072234Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 → 1

2020-11-17T13:31:45.072804Z 0 [Note] [MY-000000] [Galera] Node 3c5cd0f9 state primary

2020-11-17T13:31:45.073074Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view (view\_id(PRIM,3c5cd0f9,1)

memb {

```
3c5cd0f9,0

}

```

joined {

```
}

```

left {

```
}

```

partitioned {

```
}

```

)

2020-11-17T13:31:45.073330Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk

2020-11-17T13:31:45.077210Z 0 [Note] [MY-000000] [Galera] gcomm: connected

2020-11-17T13:31:45.077632Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636

2020-11-17T13:31:45.077963Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)

2020-11-17T13:31:45.078203Z 0 [Note] [MY-000000] [Galera] Opened channel ‘percona-xtradb-cluster-pxc’

2020-11-17T13:31:45.078930Z 1 [Note] [MY-000000] [WSREP] Starting applier thread 1

2020-11-17T13:31:45.079433Z 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2

2020-11-17T13:31:45.080111Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my\_idx = 0, memb\_num = 1

2020-11-17T13:31:45.081267Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 3c5f0f8f-28d9-11eb-aa41-3766c17ee473

2020-11-17T13:31:45.081854Z 0 [Note] [MY-000000] [Galera] STATE\_EXCHANGE: sent state UUID: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986

2020-11-17T13:31:45.082047Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986

2020-11-17T13:31:45.082374Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986 from 0 (percona-xtradb-cluster-pxc-0)

2020-11-17T13:31:45.082659Z 0 [Note] [MY-000000] [Galera] Quorum results:

```
version = 6,

component = PRIMARY,

conf_id = 0,

members = 1/1 (primary/total),

act_id = 0,

last_appl. = 0,

protocols = 2/10/4 (gcs/repl/appl),

vote policy= 0,

group UUID = 3c5f0f8f-28d9-11eb-aa41-3766c17ee473

```

2020-11-17T13:31:45.082923Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]

2020-11-17T13:31:45.083134Z 0 [Note] [MY-000000] [Galera] Restored state OPEN → JOINED (1)

2020-11-17T13:31:45.083519Z 1 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered

2020-11-17T13:31:45.083750Z 1 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1

2020-11-17T13:31:45.083948Z 1 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)

2020-11-17T13:31:45.084216Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d

2020-11-17T13:31:45.084593Z 1 [Note] [MY-000000] [Galera] Server percona-xtradb-cluster-pxc-0 connected to cluster at position 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:1 with ID 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d

2020-11-17T13:31:45.084888Z 1 [Note] [MY-000000] [WSREP] Server status change disconnected → connected

2020-11-17T13:31:45.085347Z 1 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.

2020-11-17T13:31:45.085832Z 1 [Note] [MY-000000] [Galera] ####### ST not required

2020-11-17T13:31:45.086230Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no

2020-11-17T13:31:45.086559Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-17T13:31:45.086869Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5

2020-11-17T13:31:45.087107Z 1 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 → 1

2020-11-17T13:31:45.087421Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-17T13:31:45.087697Z 1 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1

2020-11-17T13:31:45.088032Z 0 [Note] [MY-000000] [Galera] Member 0.0 (percona-xtradb-cluster-pxc-0) synced with group.

2020-11-17T13:31:45.088360Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 1)

2020-11-17T13:31:45.088710Z 1 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped

2020-11-17T13:31:45.088995Z 1 [Note] [MY-000000] [Galera] GCache history reset: old(801d098a-28d8-11eb-9842-5a65a898fbb3:1 → 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:0

2020-11-17T13:31:45.184535Z 1 [Note] [MY-000000] [Galera] Flushing memory map to disk…

2020-11-17T13:31:45.437682Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno\_reset(): discarded 184 bytes

2020-11-17T13:31:45.438403Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno\_reset(): found 0/1 locked buffers

2020-11-17T13:31:45.440525Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 1

2020-11-17T13:31:45.440882Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1

2020-11-17T13:31:45.441627Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1

2020-11-17T13:31:45.442430Z 1 [Note] [MY-000000] [Galera] ================================================

View:

id: 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:1

status: primary

protocol\_version: 4

capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL\_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL\_READ, INCREMENTAL\_WS, UNORDERED, PREORDERED, STREAMING, NBO

final: no

own\_index: 0

members(1):

```
0: 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d, percona-xtradb-cluster-pxc-0

```

=================================================

2020-11-17T13:31:45.442791Z 1 [Note] [MY-000000] [WSREP] Server status change connected → joiner

2020-11-17T13:31:45.443006Z 1 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.

2020-11-17T13:31:45.443491Z 1 [Note] [MY-000000] [WSREP] Server status change joiner → initializing

2020-11-17T13:31:45.443789Z 1 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.

2020-11-17T13:31:45.444826Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO\_set\_mem\_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.

2020-11-17T13:31:45.452781Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2020-11-17T13:31:46.054042Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2020-11-17T13:31:46.240561Z 3 [Note] [MY-000000] [WSREP] wsrep\_init\_schema\_and\_SR (nil)

2020-11-17T13:31:46.246726Z 3 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement '#! PXC\_SECTION::START

# 

# All the commands between PXC\_SECTION::START and PXC\_SECTION::END will be

# run whenever PXC starts up. This is to ensure that the PXC internal session

# user always exists (this user is also created on upgrade, but in certain

# upgrade scenarios, such as upgrading from a PS 8.0 node, it will not

# run the upgrade code).

# 

# Note: The PXC\_SECTION markers must come after an SQL statement. Due to the

# way the code is generated, do not place any comments before the markers.

# 

# SQL commands to create the PXC internal session user and role that is used

# by the SST process

# 

# mysql.pxc.internal.session

# See the comments in mysql\_system\_tables.sql

# 

# These are the values for

# GRANT CREATE USER ON _._ TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

# GRANT SUPER ON _._ TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

# GRANT RELOAD ON _._ TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

# 

#INSERT IGNORE INTO mysql.user VALUES (‘localhost’,‘mysql.pxc.internal.session’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘Y’,‘N’,‘N’,‘N’,‘Y’,‘N’,‘N’,‘N’,‘N’,‘Y’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘Y’,‘N’,‘N’,‘N’,‘’,‘’,‘’,‘’,0,0,0,0,‘caching\_sha2\_password’,‘$A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED’,‘N’,CURRENT\_TIMESTAMP, NULL,‘Y’,‘N’,‘N’,NULL,NULL,NULL,NULL);

# (Use this since the CREATE DATABASE doesn’t work when the grant is in a role)

# So we assign the mysql.pxc.internal.session root privileges with grant option

# These are the values for

# GRANT ALL PRIVILEGES ON _._ TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

# GRANT BACKUP\_ADMIN, LOCK TABLES, PROCESS, RELOAD, REPLICATION CLIENT, SUPER ON _._ TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

# 

INSERT IGNORE INTO mysql.user VALUES (‘localhost’,‘mysql.pxc.internal.session’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘’,‘’,‘’,‘’,0,0,0,0,‘caching\_sha2\_password’,‘$A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED’,‘N’,CURRENT\_TIMESTAMP,NULL,‘Y’,‘Y’,‘Y’,NULL,NULL,NULL,NULL);

’ failed with error code = 1146, error message = ‘Table ‘mysql.user’ doesn’t exist’.

2020-11-17T13:31:46.250118Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.

2020-11-17T13:31:46.251086Z 0 [ERROR] [MY-010119] [Server] Aborting

2020-11-17T13:31:46.251134Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation

2020-11-17T13:31:48.252275Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → disconnecting

2020-11-17T13:31:48.252969Z 2 [Note] [MY-000000] [WSREP] rollbacker thread exiting 2

2020-11-17T13:31:48.254028Z 0 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification.

2020-11-17T13:31:48.254432Z 0 [Note] [MY-000000] [Galera] Closing send monitor…

2020-11-17T13:31:48.255061Z 0 [Note] [MY-000000] [Galera] Closed send monitor.

2020-11-17T13:31:48.255603Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread

2020-11-17T13:31:48.256179Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread

2020-11-17T13:31:48.257151Z 1 [ERROR] [MY-000000] [Galera] Exception: State wait was interrupted

2020-11-17T13:31:48.257370Z 1 [ERROR] [MY-000000] [Galera] View callback failed. This is unrecoverable, restart required. (FATAL)

```
 at galera/src/replicator_smm.cpp:submit_view_info():2588

```

2020-11-17T13:31:48.257560Z 1 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()

2020-11-17T13:31:48.257713Z 1 [Note] [MY-000000] [Galera] mysqld: Terminated.

13:31:48 UTC - mysqld got signal 11 ;

Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Thread pointer: 0x7f633c0008c0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong…

stack\_bottom = 7f634810cc60 thread\_stack 0x46000

2020-11-17T13:31:48.259551Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend

2020-11-17T13:31:48.259776Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 → 0

2020-11-17T13:31:48.259861Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view ((empty))

2020-11-17T13:31:48.260343Z 0 [Note] [MY-000000] [Galera] gcomm: closed

2020-11-17T13:31:48.261449Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.

2020-11-17T13:31:48.262479Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]

2020-11-17T13:31:48.262875Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.

2020-11-17T13:31:48.264147Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED → CLOSED (TO: 1)

2020-11-17T13:31:48.264645Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success

2020-11-17T13:31:48.265169Z 0 [Note] [MY-000000] [Galera] recv\_thread() joined.

2020-11-17T13:31:48.265579Z 0 [Note] [MY-000000] [Galera] Closing replication queue.

2020-11-17T13:31:48.265827Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

/usr/sbin/mysqld(my\_print\_stacktrace(unsigned char const\*, unsigned long)+0x3d) [0x1fd3a0d]

/usr/sbin/mysqld(handle\_fatal\_signal+0x303) [0x1166583]

/lib64/libpthread.so.0(+0xf630) [0x7f63628f3630]

/lib64/libc.so.6(abort+0x297) [0x7f6360793bc7]

/usr/lib64/libgalera\_smm.so(+0x1e6c7) [0x7f635329d6c7]

/usr/lib64/libgalera\_smm.so(+0x1ea54c) [0x7f635346954c]

/usr/lib64/libgalera\_smm.so(+0x1f8c51) [0x7f6353477c51]

/usr/lib64/libgalera\_smm.so(+0x1c8918) [0x7f6353447918]

/usr/lib64/libgalera\_smm.so(+0x1c8b98) [0x7f6353447b98]

/usr/lib64/libgalera\_smm.so(+0x1f0f60) [0x7f635346ff60]

/usr/lib64/libgalera\_smm.so(+0x2103cc) [0x7f635348f3cc]

/usr/sbin/mysqld(wsrep::wsrep\_provider\_v26::run\_applier(wsrep::high\_priority\_service\*)+0xe) [0x272c13e]

/usr/sbin/mysqld() [0x11a9b54]

/usr/sbin/mysqld(start\_wsrep\_THD+0x331) [0xebe6e1]

/usr/sbin/mysqld() [0x24e37e9]

/lib64/libpthread.so.0(+0x7ea5) [0x7f63628ebea5]

/lib64/libc.so.6(clone+0x6d) [0x7f636085a8dd]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0): is an invalid pointer

Connection ID (thread ID): 1

Status: KILL\_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting

[http://www.percona.com/software/percona-xtradb-cluster/.](http://www.percona.com/software/percona-xtradb-cluster/.) You may find information

in the manual which will help you identify the cause of the crash.

Writing a core file

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [November 17, 2020, 10:49am UTC](https://forums.percona.com/t/i-am-getting-crashlookbackoff-error-for-percona-xtradb-cluster-8-0-20-in-kubernetes/8253/9 "2020-11-17T10:49:02Z")

</div>

@Laxminarayana Please edit your post and use the formatting tools to format your code as “code” so I can read it.