I am getting CrashLookBackOff error for Percona XtraDB Cluster 8.0.20 in Kubernetes

MySQL & MariaDB Percona Distribution for MySQL
#1

I am getting CrashLookBackOff error for Percona XtraDB Cluster 8.0.20 in Kubernetes. We are deploying through helm chart. Is it stable version for helm chart, Can someone help on this.

#2

Hi @Laxminarayana Can’t help much if you don’t provide any logs or more detailed information. Have you looked at the mysql error log on the pod with the issue? Have you looked at other K8S logs? Please provide.

#3

@matthewb Thanks for response. Please find below manifest file and generated logs.

apiVersion: v1

kind: Pod

metadata:

creationTimestamp: ‘2020-11-11T08:48:29Z’

generateName: percona-xtradb-cluster-pxc-

labels:

app: percona-xtradb-cluster-pxc

chart: percona-xtradb-cluster-1.0.3

controller-revision-hash: percona-xtradb-cluster-pxc-c4cb996c8

heritage: Tiller

release: percona-xtradb-cluster

statefulset.kubernetes.io/pod-name: percona-xtradb-cluster-pxc-0

name: percona-xtradb-cluster-pxc-0

namespace: default

ownerReferences:

  • apiVersion: apps/v1

blockOwnerDeletion: true

controller: true

kind: StatefulSet

name: percona-xtradb-cluster-pxc

uid: ab89b4bf-23fa-11eb-9a3d-0242376f5115

resourceVersion: ‘33575’

selfLink: /api/v1/namespaces/default/pods/percona-xtradb-cluster-pxc-0

uid: ab958682-23fa-11eb-9a3d-0242376f5115

spec:

containers:

  • command:

    • /bin/bash

    • /startup-scripts/entrypoint.sh

env:

- name: MYSQL_ROOT_PASSWORD

 valueFrom:

  secretKeyRef:

   key: mysql-root-password

   name: percona-xtradb-cluster-pxc

- name: MYSQL_PASSWORD

 valueFrom:

  secretKeyRef:

   key: mysql-password

   name: percona-xtradb-cluster-pxc

- name: XTRABACKUP_PASSWORD

 valueFrom:

  secretKeyRef:

   key: xtrabackup-password

   name: percona-xtradb-cluster-pxc

- name: MYSQL_USER

 valueFrom:

  secretKeyRef:

   key: mysql-user

   name: percona-xtradb-cluster-pxc

- name: MYSQL_DATABASE

 valueFrom:

  secretKeyRef:

   key: mysql-database

   name: percona-xtradb-cluster-pxc

- name: ALLOW_ROOT_FROM

 value: '%'

- name: CLUSTER_NAME

 value: percona-xtradb-cluster-pxc

- name: SHORT_CLUSTER_NAME

 value: percona-xtradb-cluster-pxc

- name: K8S_SERVICE_NAME

 value: percona-xtradb-cluster-pxc-repl

- name: PXC_STRICT_MODE

 value: MASTER

- name: DEBUG

 value: 'true'

image: ‘local/percona-xtradb-cluster-server:8.0.20-11.3’

imagePullPolicy: Never

livenessProbe:

exec:

 command:

  - /bin/bash

  - '-c'

  - mysqladmin ping || test -e /var/lib/mysql/sst_in_progress

failureThreshold: 3

initialDelaySeconds: 30

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 2

name: database

ports:

- containerPort: 3306

 name: mysql

 protocol: TCP

- containerPort: 4567

 name: galera-repl

 protocol: TCP

- containerPort: 4568

 name: state-transfer

 protocol: TCP

- containerPort: 4444

 name: state-snapshot

 protocol: TCP

readinessProbe:

exec:

 command:

  - mysql

  - '-h'

  - 127.0.0.1

  - '-e'

  - SELECT 1

failureThreshold: 3

initialDelaySeconds: 30

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 2

resources: {}

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

volumeMounts:

- mountPath: /var/lib/mysql

 name: mysql-data

- mountPath: /startup-scripts

 name: mysql-startup-scripts

- mountPath: /etc/mysql/conf.d

 name: mysql-config-files

- mountPath: /root

 name: slash-root

- mountPath: /var/log

 name: var-log

- mountPath: /backup-data

 name: backup-data

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

 name: default-token-hl4xt

 readOnly: true

  • command:

    • tail

    • ‘-f’

    • /var/log/audit.log

image: ‘local/debian:jessie’

imagePullPolicy: Never

name: audit-logs

resources: {}

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

volumeMounts:

- mountPath: /var/log

 name: var-log

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

 name: default-token-hl4xt

 readOnly: true

dnsPolicy: ClusterFirst

enableServiceLinks: true

hostname: percona-xtradb-cluster-pxc-0

initContainers:

  • command:

    • rm

    • ‘-fr’

    • /var/lib/mysql/lost+found

image: ‘local/debian:jessie’

imagePullPolicy: Never

name: init

resources: {}

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

volumeMounts:

- mountPath: /var/lib/mysql

 name: mysql-data

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

 name: default-token-hl4xt

 readOnly: true

nodeName: minikube

priority: 0

restartPolicy: Always

schedulerName: default-scheduler

securityContext:

fsGroup: 1001

serviceAccount: default

serviceAccountName: default

subdomain: percona-xtradb-cluster-pxc

terminationGracePeriodSeconds: 30

tolerations:

  • effect: NoExecute

key: node.kubernetes.io/not-ready

operator: Exists

tolerationSeconds: 300

  • effect: NoExecute

key: node.kubernetes.io/unreachable

operator: Exists

tolerationSeconds: 300

volumes:

  • name: mysql-data

persistentVolumeClaim:

claimName: mysql-data-percona-xtradb-cluster-pxc-0
  • emptyDir: {}

name: slash-root

  • emptyDir: {}

name: var-log

  • configMap:

    defaultMode: 420

    name: percona-xtradb-cluster-pxc-config-files

name: mysql-config-files

  • configMap:

    defaultMode: 420

    name: percona-xtradb-cluster-pxc-startup-scripts

name: mysql-startup-scripts

  • emptyDir: {}

name: backup-data

  • emptyDir: {}

name: rsync-disk

  • name: default-token-hl4xt

secret:

defaultMode: 420

secretName: default-token-hl4xt

status:

conditions:

  • lastProbeTime: null

lastTransitionTime: ‘2020-11-11T08:48:31Z’

status: ‘True’

type: Initialized

  • lastProbeTime: null

lastTransitionTime: ‘2020-11-11T08:48:29Z’

message: ‘containers with unready status: [database]’

reason: ContainersNotReady

status: ‘False’

type: Ready

  • lastProbeTime: null

lastTransitionTime: ‘2020-11-11T08:48:29Z’

message: ‘containers with unready status: [database]’

reason: ContainersNotReady

status: ‘False’

type: ContainersReady

  • lastProbeTime: null

lastTransitionTime: ‘2020-11-11T08:48:29Z’

status: ‘True’

type: PodScheduled

containerStatuses:

  • containerID: >-

    docker://c68c8aec500be16b399b8752b00bfd8c0fbe0fe25ec8ac2a2e21e4a8a112c50b

image: ‘debian:jessie’

imageID: >-

docker-pullable://debian@sha256:62459e0ef769f129957fed32a1a4512b1415f09b794454acf012c82412ddf834

lastState:

terminated:

 containerID: >-

  docker://60621f5d5c1d56ac759c36a2cb7c7c7b72e0989e443510ba62f02ab98023efa5

 exitCode: 1

 finishedAt: '2020-11-11T08:48:48Z'

 reason: Error

 startedAt: '2020-11-11T08:48:48Z'

name: audit-logs

ready: true

restartCount: 3

state:

running:

 startedAt: '2020-11-11T08:49:15Z'

  • containerID: >-

    docker://eb49f2b69500dc3159bce2cc7bc32a94e65a35e560bf0191b0f44180918944b8

image: ‘local/percona-xtradb-cluster-server:8.0.20-11.3’

imageID: >-

docker-pullable://percona/percona-xtradb-cluster@sha256:08ef5b75e12dbbf7fba35c9b6598908e139d9d18227b05eaa06b55b18f574754

lastState:

terminated:

 containerID: >-

  docker://eb49f2b69500dc3159bce2cc7bc32a94e65a35e560bf0191b0f44180918944b8

 exitCode: 2

 finishedAt: '2020-11-11T08:55:05Z'

 reason: Error

 startedAt: '2020-11-11T08:55:01Z'

name: database

ready: false

restartCount: 6

state:

waiting:

 message: >-

  Back-off 5m0s restarting failed container=database

  pod=percona-xtradb-cluster-pxc-0_default(ab958682-23fa-11eb-9a3d-0242376f5115)

 reason: CrashLoopBackOff

hostIP: 192.168.49.2

initContainerStatuses:

  • containerID: >-

    docker://97caad90ea2f251639d4eabc23a44a91a6a328ae2baa2d90e776bf1f142a1948

image: ‘debian:jessie’

imageID: >-

docker-pullable://debian@sha256:62459e0ef769f129957fed32a1a4512b1415f09b794454acf012c82412ddf834

lastState: {}

name: init

ready: true

restartCount: 0

state:

terminated:

 containerID: >-

  docker://97caad90ea2f251639d4eabc23a44a91a6a328ae2baa2d90e776bf1f142a1948

 exitCode: 0

 finishedAt: '2020-11-11T08:48:30Z'

 reason: Completed

 startedAt: '2020-11-11T08:48:30Z'

phase: Running

podIP: 172.17.0.10

qosClass: BestEffort

startTime: ‘2020-11-11T08:48:29Z’

#4

Logs:

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$ kubectl logs percona-xtradb-cluster-pxc-0 -c database -n default

  • . /startup-scripts/functions.sh

++ hostname -i

++ awk ’ { print $1 } ’

  • ipaddr=172.17.0.10

++ hostname

  • hostname=percona-xtradb-cluster-pxc-0

  • echo ‘I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10’

I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10

  • ‘[’ ‘’ = - ‘]’

++ resolveip -s percona-xtradb-cluster-pxc-repl

/startup-scripts/entrypoint.sh: line 19: resolveip: command not found

++ echo ‘’

  • cluster_join=

  • [[ -z ‘’ ]]

  • echo ‘I am the Primary Node’

I am the Primary Node

  • init_mysql

  • DATADIR=/var/lib/mysql

  • ‘[’ ‘!’ -e /var/lib/mysql/mysql ‘]’

  • write_password_file

  • [[ -n ohchohtaigheecaemeeDaivo3wai3Dae ]]

  • cat

  • exec mysqld --user=mysql --wsrep_cluster_name=percona-xtradb-cluster-pxc --wsrep_node_name=percona-xtradb-cluster-pxc-0 --wsrep_cluster_address=gcomm:// --wsrep_sst_method=xtrabackup-v2 --wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar --wsrep_node_address=172.17.0.10 --pxc_strict_mode=MASTER

2020-11-11T08:55:01.770215Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes and --log-slow-slave-statements have no effect if --slow-query-log is not set

2020-11-11T08:55:01.777677Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11.3) starting as process 1

2020-11-11T08:55:01.786110Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1

2020-11-11T08:55:01.786220Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’

2020-11-11T08:55:01.791275Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy info@codership.com loaded successfully.

2020-11-11T08:55:01.791490Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.

2020-11-11T08:55:01.794837Z 0 [Note] [MY-000000] [Galera] Found saved state: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe:-1, safe_to_bootstrap: 1

2020-11-11T08:55:01.796754Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:

Version: 2

UUID: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe

Seqno: -1 - -1

Offset: -1

Synced: 0

2020-11-11T08:55:01.797015Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe, offset: -1

2020-11-11T08:55:01.797560Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% ( 0/134217752 bytes) complete.

2020-11-11T08:55:01.902603Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan…100.0% (134217752/134217752 bytes) complete.

2020-11-11T08:55:01.902810Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1

2020-11-11T08:55:01.902987Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% ( 0/184 bytes) complete.

2020-11-11T08:55:01.903113Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan…100.0% (184/184 bytes) complete.

2020-11-11T08:55:01.903239Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers

2020-11-11T08:55:01.903357Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 0/134217728

2020-11-11T08:55:01.904638Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.17.0.10; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem;

2020-11-11T08:55:01.928344Z 0 [Note] [MY-000000] [WSREP] Starting replication

2020-11-11T08:55:01.928686Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1

2020-11-11T08:55:01.929245Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1

2020-11-11T08:55:01.929642Z 0 [Note] [MY-000000] [Galera] protonet asio version 0

2020-11-11T08:55:01.930507Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.

2020-11-11T08:55:01.931623Z 0 [Note] [MY-000000] [Galera] initializing ssl context

2020-11-11T08:55:01.933007Z 0 [Note] [MY-000000] [Galera] backend: asio

2020-11-11T08:55:01.933399Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0

2020-11-11T08:55:01.934128Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

2020-11-11T08:55:01.934289Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

2020-11-11T08:55:01.935964Z 0 [Note] [MY-000000] [Galera] GMCast version 0

2020-11-11T08:55:01.936419Z 0 [Note] [MY-000000] [Galera] (95a57b03, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567

2020-11-11T08:55:01.936777Z 0 [Note] [MY-000000] [Galera] (95a57b03, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1

2020-11-11T08:55:01.937883Z 0 [Note] [MY-000000] [Galera] EVS version 1

2020-11-11T08:55:01.938344Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group ‘percona-xtradb-cluster-pxc’

2020-11-11T08:55:01.938707Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery

2020-11-11T08:55:01.940309Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1

2020-11-11T08:55:01.940842Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1

2020-11-11T08:55:01.941347Z 0 [Note] [MY-000000] [Galera] Node 95a57b03 state primary

2020-11-11T08:55:01.941834Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view (view_id(PRIM,95a57b03,1)

memb {

95a57b03,0

}

joined {

}

left {

}

partitioned {

}

)

2020-11-11T08:55:01.942081Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk

2020-11-11T08:55:01.951665Z 0 [Note] [MY-000000] [Galera] gcomm: connected

2020-11-11T08:55:01.954121Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636

2020-11-11T08:55:01.955325Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)

2020-11-11T08:55:01.955859Z 0 [Note] [MY-000000] [Galera] Opened channel ‘percona-xtradb-cluster-pxc’

2020-11-11T08:55:01.960517Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1

2020-11-11T08:55:01.966240Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 95a9c4c2-23fb-11eb-81e9-d32431edcdd2

2020-11-11T08:55:01.969461Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 95aaaa86-23fb-11eb-ab85-6adf96a771ab

2020-11-11T08:55:01.969909Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 95aaaa86-23fb-11eb-ab85-6adf96a771ab

2020-11-11T08:55:01.970784Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 95aaaa86-23fb-11eb-ab85-6adf96a771ab from 0 (percona-xtradb-cluster-pxc-0)

2020-11-11T08:55:01.971242Z 0 [Note] [MY-000000] [Galera] Quorum results:

version  = 6,

component = PRIMARY,

conf_id  = 0,

members  = 1/1 (primary/total),

act_id   = 0,

last_appl. = 0,

protocols = 2/10/4 (gcs/repl/appl),

vote policy= 0,

group UUID = 95a9c4c2-23fb-11eb-81e9-d32431edcdd2

2020-11-11T08:55:01.971756Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]

2020-11-11T08:55:01.972103Z 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (1)

2020-11-11T08:55:01.972518Z 0 [Note] [MY-000000] [Galera] Member 0.0 (percona-xtradb-cluster-pxc-0) synced with group.

2020-11-11T08:55:01.973065Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 1)

2020-11-11T08:55:01.973406Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1

2020-11-11T08:55:01.973968Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2

2020-11-11T08:55:01.978752Z 2 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered

2020-11-11T08:55:01.993141Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1

2020-11-11T08:55:01.994465Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)

2020-11-11T08:55:01.995337Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 95a57b03-23fb-11eb-ad58-c242f987fcfd

2020-11-11T08:55:01.996329Z 2 [Note] [MY-000000] [Galera] Server percona-xtradb-cluster-pxc-0 connected to cluster at position 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:1 with ID 95a57b03-23fb-11eb-ad58-c242f987fcfd

2020-11-11T08:55:01.997288Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected

2020-11-11T08:55:01.998214Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-11T08:55:01.999240Z 2 [Note] [MY-000000] [Galera] ####### ST not required

2020-11-11T08:55:02.000427Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no

2020-11-11T08:55:02.001559Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-11T08:55:02.002496Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5

2020-11-11T08:55:02.003646Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 1

2020-11-11T08:55:02.004846Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-11T08:55:02.005996Z 2 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1

2020-11-11T08:55:02.008501Z 2 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped

2020-11-11T08:55:02.010571Z 2 [Note] [MY-000000] [Galera] GCache history reset: old(2fa7d286-23fb-11eb-b83b-231d9f8d2cbe:1 -> 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:0

2020-11-11T08:55:02.221936Z 2 [Note] [MY-000000] [Galera] Flushing memory map to disk…

2020-11-11T08:55:02.438824Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 184 bytes

2020-11-11T08:55:02.440566Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 0/1 locked buffers

2020-11-11T08:55:02.442264Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 1

2020-11-11T08:55:02.445908Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1

2020-11-11T08:55:02.448208Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1

2020-11-11T08:55:02.452316Z 2 [Note] [MY-000000] [Galera] ================================================

View:

id: 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:1

status: primary

protocol_version: 4

capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO

final: no

own_index: 0

members(1):

0: 95a57b03-23fb-11eb-ad58-c242f987fcfd, percona-xtradb-cluster-pxc-0

=================================================

2020-11-11T08:55:02.455216Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner

2020-11-11T08:55:02.455775Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-11T08:55:02.456242Z 2 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing

2020-11-11T08:55:02.456734Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-11T08:55:02.463972Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.

2020-11-11T08:55:02.473774Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2020-11-11T08:55:03.131444Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2020-11-11T08:55:03.586137Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)

2020-11-11T08:55:03.626812Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully

2020-11-11T08:55:03.648069Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: ‘/var/lib/mysql/mysqlx.sock’ bind-address: ‘::’ port: 33060

2020-11-11T08:55:03.719538Z 0 [Note] [MY-000000] [WSREP] Before binlog recovery (wsrep position: a2555643-2338-11eb-9e47-97282c20ffa4:21)

2020-11-11T08:55:03.721102Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…

2020-11-11T08:55:03.738572Z 0 [System] [MY-010232] [Server] XA crash recovery finished.

2020-11-11T08:55:03.740355Z 0 [Note] [MY-000000] [WSREP] After binlog recovery (wsrep position: a2555643-2338-11eb-9e47-97282c20ffa4:21)

2020-11-11T08:55:03.793289Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

2020-11-11T08:55:03.802819Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location ‘/var/lib/mysql’ in the path is accessible to all OS users. Consider choosing a different directory.

2020-11-11T08:55:03.813542Z 0 [ERROR] [MY-000067] [Server] unknown variable ‘wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar’.

2020-11-11T08:55:03.816571Z 0 [ERROR] [MY-010119] [Server] Aborting

2020-11-11T08:55:03.816745Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation

2020-11-11T08:55:05.817227Z 0 [Note] [MY-000000] [WSREP] Server status change initializing -> disconnecting

2020-11-11T08:55:05.817356Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-11T08:55:05.817445Z 0 [Note] [MY-000000] [Galera] Closing send monitor…

2020-11-11T08:55:05.817506Z 0 [Note] [MY-000000] [Galera] Closed send monitor.

2020-11-11T08:55:05.817570Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread

2020-11-11T08:55:05.817645Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread

2020-11-11T08:55:05.817740Z 1 [Note] [MY-000000] [WSREP] rollbacker thread exiting 1

2020-11-11T08:55:05.818348Z 2 [ERROR] [MY-000000] [Galera] Exception: State wait was interrupted

2020-11-11T08:55:05.818456Z 2 [ERROR] [MY-000000] [Galera] View callback failed. This is unrecoverable, restart required. (FATAL)

 at galera/src/replicator_smm.cpp:submit_view_info():2588

2020-11-11T08:55:05.818488Z 2 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()

2020-11-11T08:55:05.818523Z 2 [Note] [MY-000000] [Galera] mysqld: Terminated.

08:55:05 UTC - mysqld got signal 11 ;

Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Thread pointer: 0x7f91700008c0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong…

stack_bottom = 7f918585bc60 thread_stack 0x46000

2020-11-11T08:55:05.820295Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend

2020-11-11T08:55:05.820872Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0

2020-11-11T08:55:05.820945Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view ((empty))

2020-11-11T08:55:05.821271Z 0 [Note] [MY-000000] [Galera] gcomm: closed

2020-11-11T08:55:05.821365Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.

2020-11-11T08:55:05.821495Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]

2020-11-11T08:55:05.821543Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.

2020-11-11T08:55:05.821584Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> CLOSED (TO: 1)

2020-11-11T08:55:05.821630Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success

2020-11-11T08:55:05.822808Z 0 [Note] [MY-000000] [Galera] recv_thread() joined.

2020-11-11T08:55:05.823081Z 0 [Note] [MY-000000] [Galera] Closing replication queue.

2020-11-11T08:55:05.823274Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x1fd3a0d]

/usr/sbin/mysqld(handle_fatal_signal+0x303) [0x1166583]

/lib64/libpthread.so.0(+0xf630) [0x7f91a0042630]

/lib64/libc.so.6(abort+0x297) [0x7f919dee2bc7]

/usr/lib64/libgalera_smm.so(+0x1e6c7) [0x7f91909ec6c7]

/usr/lib64/libgalera_smm.so(+0x1ea54c) [0x7f9190bb854c]

/usr/lib64/libgalera_smm.so(+0x1f8c51) [0x7f9190bc6c51]

/usr/lib64/libgalera_smm.so(+0x1c8918) [0x7f9190b96918]

/usr/lib64/libgalera_smm.so(+0x1c8b98) [0x7f9190b96b98]

/usr/lib64/libgalera_smm.so(+0x1f0f60) [0x7f9190bbef60]

/usr/lib64/libgalera_smm.so(+0x2103cc) [0x7f9190bde3cc]

/usr/sbin/mysqld(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*)+0xe) [0x272c13e]

/usr/sbin/mysqld() [0x11a9b54]

/usr/sbin/mysqld(start_wsrep_THD+0x331) [0xebe6e1]

/usr/sbin/mysqld() [0x24e37e9]

/lib64/libpthread.so.0(+0x7ea5) [0x7f91a003aea5]

/lib64/libc.so.6(clone+0x6d) [0x7f919dfa98dd]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0): is an invalid pointer

Connection ID (thread ID): 2

Status: KILL_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting

http://www.percona.com/software/percona-xtradb-cluster/. You may find information

in the manual which will help you identify the cause of the crash.

Writing a core file

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$

#5

@LaxminarayanaPlease use the forum tools to format your output. Hard to read when you just copy/paste.

I can see the error right here in your logs;

2020-11-11T08:55:03.813542Z 0 [ERROR] [MY-000067] [Server] unknown variable 'wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar'.

2020-11-11T08:55:03.816571Z 0 [ERROR] [MY-010119] [Server] Aborting

Are you sure you are using the latest version of the operator? Looks like v1.6.0 is the current GA version. Please upgrade to that version of the operator before attempting again. If you are indeed on 1.6.0, then this is a serious bug in the operator that needs to be addressed. The parameter wsrep_sst_auth was removed in PXC8 and should not be present in the scripts.

#6

Thanks for your response. I have removed parameter wsrep_sst_auth but it wasn’t worked. We are using helm for deployment but I don’t see updated chart version for Percona Xtradb Cluster and also we are doing continues deployment using argo cd. Here I have attached generated manifest file. Please look into manifest file and help me to resolve this issue.

manifest.zip

#7

@Laxminarayana After removing that parameter, what do the mysql logs say?

#8

@matthewb Thanks for quick response. Here is the log details.

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$ kubectl logs percona-xtradb-cluster-pxc-0 -c database -n default

  • . /startup-scripts/functions.sh

++ awk ’ { print $1 } ’

++ hostname -i

  • ipaddr=172.17.0.10

++ hostname

  • hostname=percona-xtradb-cluster-pxc-0

  • echo ‘I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10’

I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10

  • ‘[’ ‘’ = - ‘]’

++ resolveip -s percona-xtradb-cluster-pxc-repl

/startup-scripts/entrypoint.sh: line 19: resolveip: command not found

++ echo ‘’

  • cluster_join=

  • [[ -z ‘’ ]]

  • echo ‘I am the Primary Node’

  • init_mysql

  • DATADIR=/var/lib/mysql

  • ‘[’ ‘!’ -e /var/lib/mysql/mysql ‘]’

I am the Primary Node

  • write_password_file

  • [[ -n ohchohtaigheecaemeeDaivo3wai3Dae ]]

  • cat

  • exec mysqld --user=mysql --wsrep_cluster_name=percona-xtradb-cluster-pxc --wsrep_node_name=percona-xtradb-cluster-pxc-0 --wsrep_cluster_address=gcomm:// --wsrep_sst_method=xtrabackup-v2

2020-11-17T13:31:45.018951Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes and --log-slow-slave-statements have no effect if --slow-query-log is not set

2020-11-17T13:31:45.020825Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11.3) starting as process 1

2020-11-17T13:31:45.032709Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1

2020-11-17T13:31:45.032923Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’

2020-11-17T13:31:45.040668Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy info@codership.com loaded successfully.

2020-11-17T13:31:45.040843Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.

2020-11-17T13:31:45.042434Z 0 [Note] [MY-000000] [Galera] Found saved state: 801d098a-28d8-11eb-9842-5a65a898fbb3:-1, safe_to_bootstrap: 1

2020-11-17T13:31:45.043839Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:

Version: 2

UUID: 801d098a-28d8-11eb-9842-5a65a898fbb3

Seqno: -1 - -1

Offset: -1

Synced: 0

2020-11-17T13:31:45.044021Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 801d098a-28d8-11eb-9842-5a65a898fbb3, offset: -1

2020-11-17T13:31:45.044243Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% ( 0/134217752 bytes) complete.

2020-11-17T13:31:45.044485Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan…100.0% (134217752/134217752 bytes) complete.

2020-11-17T13:31:45.044624Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1

2020-11-17T13:31:45.044820Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% ( 0/184 bytes) complete.

2020-11-17T13:31:45.044967Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan…100.0% (184/184 bytes) complete.

2020-11-17T13:31:45.045119Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers

2020-11-17T13:31:45.045301Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 0/134217728

2020-11-17T13:31:45.046821Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.17.0.10; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem;

2020-11-17T13:31:45.063441Z 0 [Note] [MY-000000] [WSREP] Starting replication

2020-11-17T13:31:45.063687Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1

2020-11-17T13:31:45.063901Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1

2020-11-17T13:31:45.064162Z 0 [Note] [MY-000000] [Galera] protonet asio version 0

2020-11-17T13:31:45.064464Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.

2020-11-17T13:31:45.064696Z 0 [Note] [MY-000000] [Galera] initializing ssl context

2020-11-17T13:31:45.065167Z 0 [Note] [MY-000000] [Galera] backend: asio

2020-11-17T13:31:45.065414Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0

2020-11-17T13:31:45.065704Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

2020-11-17T13:31:45.065969Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

2020-11-17T13:31:45.066904Z 0 [Note] [MY-000000] [Galera] GMCast version 0

2020-11-17T13:31:45.067224Z 0 [Note] [MY-000000] [Galera] (3c5cd0f9, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567

2020-11-17T13:31:45.067466Z 0 [Note] [MY-000000] [Galera] (3c5cd0f9, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1

2020-11-17T13:31:45.068063Z 0 [Note] [MY-000000] [Galera] EVS version 1

2020-11-17T13:31:45.068586Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group ‘percona-xtradb-cluster-pxc’

2020-11-17T13:31:45.069217Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery

2020-11-17T13:31:45.071242Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1

2020-11-17T13:31:45.072234Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1

2020-11-17T13:31:45.072804Z 0 [Note] [MY-000000] [Galera] Node 3c5cd0f9 state primary

2020-11-17T13:31:45.073074Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view (view_id(PRIM,3c5cd0f9,1)

memb {

3c5cd0f9,0

}

joined {

}

left {

}

partitioned {

}

)

2020-11-17T13:31:45.073330Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk

2020-11-17T13:31:45.077210Z 0 [Note] [MY-000000] [Galera] gcomm: connected

2020-11-17T13:31:45.077632Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636

2020-11-17T13:31:45.077963Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)

2020-11-17T13:31:45.078203Z 0 [Note] [MY-000000] [Galera] Opened channel ‘percona-xtradb-cluster-pxc’

2020-11-17T13:31:45.078930Z 1 [Note] [MY-000000] [WSREP] Starting applier thread 1

2020-11-17T13:31:45.079433Z 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2

2020-11-17T13:31:45.080111Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1

2020-11-17T13:31:45.081267Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 3c5f0f8f-28d9-11eb-aa41-3766c17ee473

2020-11-17T13:31:45.081854Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986

2020-11-17T13:31:45.082047Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986

2020-11-17T13:31:45.082374Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986 from 0 (percona-xtradb-cluster-pxc-0)

2020-11-17T13:31:45.082659Z 0 [Note] [MY-000000] [Galera] Quorum results:

version  = 6,

component = PRIMARY,

conf_id  = 0,

members  = 1/1 (primary/total),

act_id   = 0,

last_appl. = 0,

protocols = 2/10/4 (gcs/repl/appl),

vote policy= 0,

group UUID = 3c5f0f8f-28d9-11eb-aa41-3766c17ee473

2020-11-17T13:31:45.082923Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]

2020-11-17T13:31:45.083134Z 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (1)

2020-11-17T13:31:45.083519Z 1 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered

2020-11-17T13:31:45.083750Z 1 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1

2020-11-17T13:31:45.083948Z 1 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)

2020-11-17T13:31:45.084216Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d

2020-11-17T13:31:45.084593Z 1 [Note] [MY-000000] [Galera] Server percona-xtradb-cluster-pxc-0 connected to cluster at position 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:1 with ID 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d

2020-11-17T13:31:45.084888Z 1 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected

2020-11-17T13:31:45.085347Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-17T13:31:45.085832Z 1 [Note] [MY-000000] [Galera] ####### ST not required

2020-11-17T13:31:45.086230Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no

2020-11-17T13:31:45.086559Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-17T13:31:45.086869Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5

2020-11-17T13:31:45.087107Z 1 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 1

2020-11-17T13:31:45.087421Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-17T13:31:45.087697Z 1 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1

2020-11-17T13:31:45.088032Z 0 [Note] [MY-000000] [Galera] Member 0.0 (percona-xtradb-cluster-pxc-0) synced with group.

2020-11-17T13:31:45.088360Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 1)

2020-11-17T13:31:45.088710Z 1 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped

2020-11-17T13:31:45.088995Z 1 [Note] [MY-000000] [Galera] GCache history reset: old(801d098a-28d8-11eb-9842-5a65a898fbb3:1 -> 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:0

2020-11-17T13:31:45.184535Z 1 [Note] [MY-000000] [Galera] Flushing memory map to disk…

2020-11-17T13:31:45.437682Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 184 bytes

2020-11-17T13:31:45.438403Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 0/1 locked buffers

2020-11-17T13:31:45.440525Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 1

2020-11-17T13:31:45.440882Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1

2020-11-17T13:31:45.441627Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1

2020-11-17T13:31:45.442430Z 1 [Note] [MY-000000] [Galera] ================================================

View:

id: 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:1

status: primary

protocol_version: 4

capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO

final: no

own_index: 0

members(1):

0: 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d, percona-xtradb-cluster-pxc-0

=================================================

2020-11-17T13:31:45.442791Z 1 [Note] [MY-000000] [WSREP] Server status change connected -> joiner

2020-11-17T13:31:45.443006Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-17T13:31:45.443491Z 1 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing

2020-11-17T13:31:45.443789Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-17T13:31:45.444826Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.

2020-11-17T13:31:45.452781Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2020-11-17T13:31:46.054042Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2020-11-17T13:31:46.240561Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)

2020-11-17T13:31:46.246726Z 3 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement '#! PXC_SECTION::START

All the commands between PXC_SECTION::START and PXC_SECTION::END will be

run whenever PXC starts up. This is to ensure that the PXC internal session

user always exists (this user is also created on upgrade, but in certain

upgrade scenarios, such as upgrading from a PS 8.0 node, it will not

run the upgrade code).

Note: The PXC_SECTION markers must come after an SQL statement. Due to the

way the code is generated, do not place any comments before the markers.

SQL commands to create the PXC internal session user and role that is used

by the SST process

mysql.pxc.internal.session

See the comments in mysql_system_tables.sql

These are the values for

GRANT CREATE USER ON . TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

GRANT SUPER ON . TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

GRANT RELOAD ON . TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

#INSERT IGNORE INTO mysql.user VALUES (‘localhost’,‘mysql.pxc.internal.session’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘Y’,‘N’,‘N’,‘N’,‘Y’,‘N’,‘N’,‘N’,‘N’,‘Y’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘N’,‘Y’,‘N’,‘N’,‘N’,’’,’’,’’,’’,0,0,0,0,‘caching_sha2_password’,’$A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED’,‘N’,CURRENT_TIMESTAMP, NULL,‘Y’,‘N’,‘N’,NULL,NULL,NULL,NULL);

(Use this since the CREATE DATABASE doesn’t work when the grant is in a role)

So we assign the mysql.pxc.internal.session root privileges with grant option

These are the values for

GRANT ALL PRIVILEGES ON . TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

GRANT BACKUP_ADMIN, LOCK TABLES, PROCESS, RELOAD, REPLICATION CLIENT, SUPER ON . TO ‘mysql.pxc.internal.session’@localhost WITH GRANT OPTION;

INSERT IGNORE INTO mysql.user VALUES (‘localhost’,‘mysql.pxc.internal.session’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,‘Y’,’’,’’,’’,’’,0,0,0,0,‘caching_sha2_password’,’$A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED’,‘N’,CURRENT_TIMESTAMP,NULL,‘Y’,‘Y’,‘Y’,NULL,NULL,NULL,NULL);

’ failed with error code = 1146, error message = ‘Table ‘mysql.user’ doesn’t exist’.

2020-11-17T13:31:46.250118Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.

2020-11-17T13:31:46.251086Z 0 [ERROR] [MY-010119] [Server] Aborting

2020-11-17T13:31:46.251134Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation

2020-11-17T13:31:48.252275Z 0 [Note] [MY-000000] [WSREP] Server status change initializing -> disconnecting

2020-11-17T13:31:48.252969Z 2 [Note] [MY-000000] [WSREP] rollbacker thread exiting 2

2020-11-17T13:31:48.254028Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-17T13:31:48.254432Z 0 [Note] [MY-000000] [Galera] Closing send monitor…

2020-11-17T13:31:48.255061Z 0 [Note] [MY-000000] [Galera] Closed send monitor.

2020-11-17T13:31:48.255603Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread

2020-11-17T13:31:48.256179Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread

2020-11-17T13:31:48.257151Z 1 [ERROR] [MY-000000] [Galera] Exception: State wait was interrupted

2020-11-17T13:31:48.257370Z 1 [ERROR] [MY-000000] [Galera] View callback failed. This is unrecoverable, restart required. (FATAL)

 at galera/src/replicator_smm.cpp:submit_view_info():2588

2020-11-17T13:31:48.257560Z 1 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()

2020-11-17T13:31:48.257713Z 1 [Note] [MY-000000] [Galera] mysqld: Terminated.

13:31:48 UTC - mysqld got signal 11 ;

Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Thread pointer: 0x7f633c0008c0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong…

stack_bottom = 7f634810cc60 thread_stack 0x46000

2020-11-17T13:31:48.259551Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend

2020-11-17T13:31:48.259776Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0

2020-11-17T13:31:48.259861Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view ((empty))

2020-11-17T13:31:48.260343Z 0 [Note] [MY-000000] [Galera] gcomm: closed

2020-11-17T13:31:48.261449Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.

2020-11-17T13:31:48.262479Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]

2020-11-17T13:31:48.262875Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.

2020-11-17T13:31:48.264147Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> CLOSED (TO: 1)

2020-11-17T13:31:48.264645Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success

2020-11-17T13:31:48.265169Z 0 [Note] [MY-000000] [Galera] recv_thread() joined.

2020-11-17T13:31:48.265579Z 0 [Note] [MY-000000] [Galera] Closing replication queue.

2020-11-17T13:31:48.265827Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x1fd3a0d]

/usr/sbin/mysqld(handle_fatal_signal+0x303) [0x1166583]

/lib64/libpthread.so.0(+0xf630) [0x7f63628f3630]

/lib64/libc.so.6(abort+0x297) [0x7f6360793bc7]

/usr/lib64/libgalera_smm.so(+0x1e6c7) [0x7f635329d6c7]

/usr/lib64/libgalera_smm.so(+0x1ea54c) [0x7f635346954c]

/usr/lib64/libgalera_smm.so(+0x1f8c51) [0x7f6353477c51]

/usr/lib64/libgalera_smm.so(+0x1c8918) [0x7f6353447918]

/usr/lib64/libgalera_smm.so(+0x1c8b98) [0x7f6353447b98]

/usr/lib64/libgalera_smm.so(+0x1f0f60) [0x7f635346ff60]

/usr/lib64/libgalera_smm.so(+0x2103cc) [0x7f635348f3cc]

/usr/sbin/mysqld(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*)+0xe) [0x272c13e]

/usr/sbin/mysqld() [0x11a9b54]

/usr/sbin/mysqld(start_wsrep_THD+0x331) [0xebe6e1]

/usr/sbin/mysqld() [0x24e37e9]

/lib64/libpthread.so.0(+0x7ea5) [0x7f63628ebea5]

/lib64/libc.so.6(clone+0x6d) [0x7f636085a8dd]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0): is an invalid pointer

Connection ID (thread ID): 1

Status: KILL_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting

http://www.percona.com/software/percona-xtradb-cluster/. You may find information

in the manual which will help you identify the cause of the crash.

Writing a core file

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$

#9

@Laxminarayana Please edit your post and use the formatting tools to format your code as “code” so I can read it.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2021 Percona LLC. All rights reserved.