Not the answer you need?
Register and ask your own question!

I am getting CrashLookBackOff error for Percona XtraDB Cluster 8.0.20 in Kubernetes

LaxminarayanaLaxminarayana Current User Role Contributor

I am getting CrashLookBackOff error for Percona XtraDB Cluster 8.0.20 in Kubernetes. We are deploying through helm chart. Is it stable version for helm chart, Can someone help on this.

Tagged:

Answers

  • matthewbmatthewb Senior [email protected] Percona Staff Role

    Hi @Laxminarayana Can't help much if you don't provide any logs or more detailed information. Have you looked at the mysql error log on the pod with the issue? Have you looked at other K8S logs? Please provide.

  • LaxminarayanaLaxminarayana Current User Role Contributor

    @matthewb Thanks for response. Please find below manifest file and generated logs.

    apiVersion: v1

    kind: Pod

    metadata:

     creationTimestamp: '2020-11-11T08:48:29Z'

     generateName: percona-xtradb-cluster-pxc-

     labels:

      app: percona-xtradb-cluster-pxc

      chart: percona-xtradb-cluster-1.0.3

      controller-revision-hash: percona-xtradb-cluster-pxc-c4cb996c8

      heritage: Tiller

      release: percona-xtradb-cluster

      statefulset.kubernetes.io/pod-name: percona-xtradb-cluster-pxc-0

     name: percona-xtradb-cluster-pxc-0

     namespace: default

     ownerReferences:

      - apiVersion: apps/v1

       blockOwnerDeletion: true

       controller: true

       kind: StatefulSet

       name: percona-xtradb-cluster-pxc

       uid: ab89b4bf-23fa-11eb-9a3d-0242376f5115

     resourceVersion: '33575'

     selfLink: /api/v1/namespaces/default/pods/percona-xtradb-cluster-pxc-0

     uid: ab958682-23fa-11eb-9a3d-0242376f5115

    spec:

     containers:

      - command:

        - /bin/bash

        - /startup-scripts/entrypoint.sh

       env:

        - name: MYSQL_ROOT_PASSWORD

         valueFrom:

          secretKeyRef:

           key: mysql-root-password

           name: percona-xtradb-cluster-pxc

        - name: MYSQL_PASSWORD

         valueFrom:

          secretKeyRef:

           key: mysql-password

           name: percona-xtradb-cluster-pxc

        - name: XTRABACKUP_PASSWORD

         valueFrom:

          secretKeyRef:

           key: xtrabackup-password

           name: percona-xtradb-cluster-pxc

        - name: MYSQL_USER

         valueFrom:

          secretKeyRef:

           key: mysql-user

           name: percona-xtradb-cluster-pxc

        - name: MYSQL_DATABASE

         valueFrom:

          secretKeyRef:

           key: mysql-database

           name: percona-xtradb-cluster-pxc

        - name: ALLOW_ROOT_FROM

         value: '%'

        - name: CLUSTER_NAME

         value: percona-xtradb-cluster-pxc

        - name: SHORT_CLUSTER_NAME

         value: percona-xtradb-cluster-pxc

        - name: K8S_SERVICE_NAME

         value: percona-xtradb-cluster-pxc-repl

        - name: PXC_STRICT_MODE

         value: MASTER

        - name: DEBUG

         value: 'true'

       image: 'local/percona-xtradb-cluster-server:8.0.20-11.3'

       imagePullPolicy: Never

       livenessProbe:

        exec:

         command:

          - /bin/bash

          - '-c'

          - mysqladmin ping || test -e /var/lib/mysql/sst_in_progress

        failureThreshold: 3

        initialDelaySeconds: 30

        periodSeconds: 10

        successThreshold: 1

        timeoutSeconds: 2

       name: database

       ports:

        - containerPort: 3306

         name: mysql

         protocol: TCP

        - containerPort: 4567

         name: galera-repl

         protocol: TCP

        - containerPort: 4568

         name: state-transfer

         protocol: TCP

        - containerPort: 4444

         name: state-snapshot

         protocol: TCP

       readinessProbe:

        exec:

         command:

          - mysql

          - '-h'

          - 127.0.0.1

          - '-e'

          - SELECT 1

        failureThreshold: 3

        initialDelaySeconds: 30

        periodSeconds: 10

        successThreshold: 1

        timeoutSeconds: 2

       resources: {}

       terminationMessagePath: /dev/termination-log

       terminationMessagePolicy: File

       volumeMounts:

        - mountPath: /var/lib/mysql

         name: mysql-data

        - mountPath: /startup-scripts

         name: mysql-startup-scripts

        - mountPath: /etc/mysql/conf.d

         name: mysql-config-files

        - mountPath: /root

         name: slash-root

        - mountPath: /var/log

         name: var-log

        - mountPath: /backup-data

         name: backup-data

        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount

         name: default-token-hl4xt

         readOnly: true

      - command:

        - tail

        - '-f'

        - /var/log/audit.log

       image: 'local/debian:jessie'

       imagePullPolicy: Never

       name: audit-logs

       resources: {}

       terminationMessagePath: /dev/termination-log

       terminationMessagePolicy: File

       volumeMounts:

        - mountPath: /var/log

         name: var-log

        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount

         name: default-token-hl4xt

         readOnly: true

     dnsPolicy: ClusterFirst

     enableServiceLinks: true

     hostname: percona-xtradb-cluster-pxc-0

     initContainers:

      - command:

        - rm

        - '-fr'

        - /var/lib/mysql/lost+found

       image: 'local/debian:jessie'

       imagePullPolicy: Never

       name: init

       resources: {}

       terminationMessagePath: /dev/termination-log

       terminationMessagePolicy: File

       volumeMounts:

        - mountPath: /var/lib/mysql

         name: mysql-data

        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount

         name: default-token-hl4xt

         readOnly: true

     nodeName: minikube

     priority: 0

     restartPolicy: Always

     schedulerName: default-scheduler

     securityContext:

      fsGroup: 1001

     serviceAccount: default

     serviceAccountName: default

     subdomain: percona-xtradb-cluster-pxc

     terminationGracePeriodSeconds: 30

     tolerations:

      - effect: NoExecute

       key: node.kubernetes.io/not-ready

       operator: Exists

       tolerationSeconds: 300

      - effect: NoExecute

       key: node.kubernetes.io/unreachable

       operator: Exists

       tolerationSeconds: 300

     volumes:

      - name: mysql-data

       persistentVolumeClaim:

        claimName: mysql-data-percona-xtradb-cluster-pxc-0

      - emptyDir: {}

       name: slash-root

      - emptyDir: {}

       name: var-log

      - configMap:

        defaultMode: 420

        name: percona-xtradb-cluster-pxc-config-files

       name: mysql-config-files

      - configMap:

        defaultMode: 420

        name: percona-xtradb-cluster-pxc-startup-scripts

       name: mysql-startup-scripts

      - emptyDir: {}

       name: backup-data

      - emptyDir: {}

       name: rsync-disk

      - name: default-token-hl4xt

       secret:

        defaultMode: 420

        secretName: default-token-hl4xt

    status:

     conditions:

      - lastProbeTime: null

       lastTransitionTime: '2020-11-11T08:48:31Z'

       status: 'True'

       type: Initialized

      - lastProbeTime: null

       lastTransitionTime: '2020-11-11T08:48:29Z'

       message: 'containers with unready status: [database]'

       reason: ContainersNotReady

       status: 'False'

       type: Ready

      - lastProbeTime: null

       lastTransitionTime: '2020-11-11T08:48:29Z'

       message: 'containers with unready status: [database]'

       reason: ContainersNotReady

       status: 'False'

       type: ContainersReady

      - lastProbeTime: null

       lastTransitionTime: '2020-11-11T08:48:29Z'

       status: 'True'

       type: PodScheduled

     containerStatuses:

      - containerID: >-

        docker://c68c8aec500be16b399b8752b00bfd8c0fbe0fe25ec8ac2a2e21e4a8a112c50b

       image: 'debian:jessie'

       imageID: >-

        docker-pullable://[email protected]:62459e0ef769f129957fed32a1a4512b1415f09b794454acf012c82412ddf834

       lastState:

        terminated:

         containerID: >-

          docker://60621f5d5c1d56ac759c36a2cb7c7c7b72e0989e443510ba62f02ab98023efa5

         exitCode: 1

         finishedAt: '2020-11-11T08:48:48Z'

         reason: Error

         startedAt: '2020-11-11T08:48:48Z'

       name: audit-logs

       ready: true

       restartCount: 3

       state:

        running:

         startedAt: '2020-11-11T08:49:15Z'

      - containerID: >-

        docker://eb49f2b69500dc3159bce2cc7bc32a94e65a35e560bf0191b0f44180918944b8

       image: 'local/percona-xtradb-cluster-server:8.0.20-11.3'

       imageID: >-

        docker-pullable://percona/[email protected]:08ef5b75e12dbbf7fba35c9b6598908e139d9d18227b05eaa06b55b18f574754

       lastState:

        terminated:

         containerID: >-

          docker://eb49f2b69500dc3159bce2cc7bc32a94e65a35e560bf0191b0f44180918944b8

         exitCode: 2

         finishedAt: '2020-11-11T08:55:05Z'

         reason: Error

         startedAt: '2020-11-11T08:55:01Z'

       name: database

       ready: false

       restartCount: 6

       state:

        waiting:

         message: >-

          Back-off 5m0s restarting failed container=database

          pod=percona-xtradb-cluster-pxc-0_default(ab958682-23fa-11eb-9a3d-0242376f5115)

         reason: CrashLoopBackOff

     hostIP: 192.168.49.2

     initContainerStatuses:

      - containerID: >-

        docker://97caad90ea2f251639d4eabc23a44a91a6a328ae2baa2d90e776bf1f142a1948

       image: 'debian:jessie'

       imageID: >-

        docker-pullable://[email protected]:62459e0ef769f129957fed32a1a4512b1415f09b794454acf012c82412ddf834

       lastState: {}

       name: init

       ready: true

       restartCount: 0

       state:

        terminated:

         containerID: >-

          docker://97caad90ea2f251639d4eabc23a44a91a6a328ae2baa2d90e776bf1f142a1948

         exitCode: 0

         finishedAt: '2020-11-11T08:48:30Z'

         reason: Completed

         startedAt: '2020-11-11T08:48:30Z'

     phase: Running

     podIP: 172.17.0.10

     qosClass: BestEffort

     startTime: '2020-11-11T08:48:29Z'

  • LaxminarayanaLaxminarayana Current User Role Contributor

    Logs:

    [email protected]:~/sp-composer/kub/helm$ kubectl logs percona-xtradb-cluster-pxc-0 -c database -n default

    + . /startup-scripts/functions.sh

    ++ hostname -i

    ++ awk ' { print $1 } '

    + ipaddr=172.17.0.10

    ++ hostname

    + hostname=percona-xtradb-cluster-pxc-0

    + echo 'I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10'

    I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10

    + '[' '' = - ']'

    ++ resolveip -s percona-xtradb-cluster-pxc-repl

    /startup-scripts/entrypoint.sh: line 19: resolveip: command not found

    ++ echo ''

    + cluster_join=

    + [[ -z '' ]]

    + echo 'I am the Primary Node'

    I am the Primary Node

    + init_mysql

    + DATADIR=/var/lib/mysql

    + '[' '!' -e /var/lib/mysql/mysql ']'

    + write_password_file

    + [[ -n ohchohtaigheecaemeeDaivo3wai3Dae ]]

    + cat

    + exec mysqld --user=mysql --wsrep_cluster_name=percona-xtradb-cluster-pxc --wsrep_node_name=percona-xtradb-cluster-pxc-0 --wsrep_cluster_address=gcomm:// --wsrep_sst_method=xtrabackup-v2 --wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar --wsrep_node_address=172.17.0.10 --pxc_strict_mode=MASTER

    2020-11-11T08:55:01.770215Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes and --log-slow-slave-statements have no effect if --slow-query-log is not set

    2020-11-11T08:55:01.777677Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11.3) starting as process 1

    2020-11-11T08:55:01.786110Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1

    2020-11-11T08:55:01.786220Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'

    2020-11-11T08:55:01.791275Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy <[email protected]> loaded successfully.

    2020-11-11T08:55:01.791490Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.

    2020-11-11T08:55:01.794837Z 0 [Note] [MY-000000] [Galera] Found saved state: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe:-1, safe_to_bootstrap: 1

    2020-11-11T08:55:01.796754Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:

    Version: 2

    UUID: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe

    Seqno: -1 - -1

    Offset: -1

    Synced: 0

    2020-11-11T08:55:01.797015Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe, offset: -1

    2020-11-11T08:55:01.797560Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% (    0/134217752 bytes) complete.

    2020-11-11T08:55:01.902603Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.

    2020-11-11T08:55:01.902810Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1

    2020-11-11T08:55:01.902987Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/184 bytes) complete.

    2020-11-11T08:55:01.903113Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (184/184 bytes) complete.

    2020-11-11T08:55:01.903239Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers

    2020-11-11T08:55:01.903357Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 0/134217728

    2020-11-11T08:55:01.904638Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.17.0.10; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem; 

    2020-11-11T08:55:01.928344Z 0 [Note] [MY-000000] [WSREP] Starting replication

    2020-11-11T08:55:01.928686Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1

    2020-11-11T08:55:01.929245Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1

    2020-11-11T08:55:01.929642Z 0 [Note] [MY-000000] [Galera] protonet asio version 0

    2020-11-11T08:55:01.930507Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.

    2020-11-11T08:55:01.931623Z 0 [Note] [MY-000000] [Galera] initializing ssl context

    2020-11-11T08:55:01.933007Z 0 [Note] [MY-000000] [Galera] backend: asio

    2020-11-11T08:55:01.933399Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 

    2020-11-11T08:55:01.934128Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

    2020-11-11T08:55:01.934289Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

    2020-11-11T08:55:01.935964Z 0 [Note] [MY-000000] [Galera] GMCast version 0

    2020-11-11T08:55:01.936419Z 0 [Note] [MY-000000] [Galera] (95a57b03, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567

    2020-11-11T08:55:01.936777Z 0 [Note] [MY-000000] [Galera] (95a57b03, 'ssl://0.0.0.0:4567') multicast: , ttl: 1

    2020-11-11T08:55:01.937883Z 0 [Note] [MY-000000] [Galera] EVS version 1

    2020-11-11T08:55:01.938344Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group 'percona-xtradb-cluster-pxc'

    2020-11-11T08:55:01.938707Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery

    2020-11-11T08:55:01.940309Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1

    2020-11-11T08:55:01.940842Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1

    2020-11-11T08:55:01.941347Z 0 [Note] [MY-000000] [Galera] Node 95a57b03 state primary

    2020-11-11T08:55:01.941834Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

    view (view_id(PRIM,95a57b03,1)

    memb {

        95a57b03,0

        }

    joined {

        }

    left {

        }

    partitioned {

        }

    )

    2020-11-11T08:55:01.942081Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk

    2020-11-11T08:55:01.951665Z 0 [Note] [MY-000000] [Galera] gcomm: connected

    2020-11-11T08:55:01.954121Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636

    2020-11-11T08:55:01.955325Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)

    2020-11-11T08:55:01.955859Z 0 [Note] [MY-000000] [Galera] Opened channel 'percona-xtradb-cluster-pxc'

    2020-11-11T08:55:01.960517Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1

    2020-11-11T08:55:01.966240Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 95a9c4c2-23fb-11eb-81e9-d32431edcdd2

    2020-11-11T08:55:01.969461Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 95aaaa86-23fb-11eb-ab85-6adf96a771ab

    2020-11-11T08:55:01.969909Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 95aaaa86-23fb-11eb-ab85-6adf96a771ab

    2020-11-11T08:55:01.970784Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 95aaaa86-23fb-11eb-ab85-6adf96a771ab from 0 (percona-xtradb-cluster-pxc-0)

    2020-11-11T08:55:01.971242Z 0 [Note] [MY-000000] [Galera] Quorum results:

        version  = 6,

        component = PRIMARY,

        conf_id  = 0,

        members  = 1/1 (primary/total),

        act_id   = 0,

        last_appl. = 0,

        protocols = 2/10/4 (gcs/repl/appl),

        vote policy= 0,

        group UUID = 95a9c4c2-23fb-11eb-81e9-d32431edcdd2

    2020-11-11T08:55:01.971756Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]

    2020-11-11T08:55:01.972103Z 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (1)

    2020-11-11T08:55:01.972518Z 0 [Note] [MY-000000] [Galera] Member 0.0 (percona-xtradb-cluster-pxc-0) synced with group.

    2020-11-11T08:55:01.973065Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 1)

    2020-11-11T08:55:01.973406Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1

    2020-11-11T08:55:01.973968Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2

    2020-11-11T08:55:01.978752Z 2 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered

    2020-11-11T08:55:01.993141Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1

    2020-11-11T08:55:01.994465Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)

    2020-11-11T08:55:01.995337Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 95a57b03-23fb-11eb-ad58-c242f987fcfd

    2020-11-11T08:55:01.996329Z 2 [Note] [MY-000000] [Galera] Server percona-xtradb-cluster-pxc-0 connected to cluster at position 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:1 with ID 95a57b03-23fb-11eb-ad58-c242f987fcfd

    2020-11-11T08:55:01.997288Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected

    2020-11-11T08:55:01.998214Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

    2020-11-11T08:55:01.999240Z 2 [Note] [MY-000000] [Galera] ####### ST not required

    2020-11-11T08:55:02.000427Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no

    2020-11-11T08:55:02.001559Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

    2020-11-11T08:55:02.002496Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5

    2020-11-11T08:55:02.003646Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 1

    2020-11-11T08:55:02.004846Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

    2020-11-11T08:55:02.005996Z 2 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1

    2020-11-11T08:55:02.008501Z 2 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped

    2020-11-11T08:55:02.010571Z 2 [Note] [MY-000000] [Galera] GCache history reset: old(2fa7d286-23fb-11eb-b83b-231d9f8d2cbe:1 -> 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:0

    2020-11-11T08:55:02.221936Z 2 [Note] [MY-000000] [Galera] Flushing memory map to disk...

    2020-11-11T08:55:02.438824Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 184 bytes

    2020-11-11T08:55:02.440566Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 0/1 locked buffers

    2020-11-11T08:55:02.442264Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 1

    2020-11-11T08:55:02.445908Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1

    2020-11-11T08:55:02.448208Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1

    2020-11-11T08:55:02.452316Z 2 [Note] [MY-000000] [Galera] ================================================

    View:

     id: 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:1

     status: primary

     protocol_version: 4

     capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO

     final: no

     own_index: 0

     members(1):

        0: 95a57b03-23fb-11eb-ad58-c242f987fcfd, percona-xtradb-cluster-pxc-0

    =================================================

    2020-11-11T08:55:02.455216Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner

    2020-11-11T08:55:02.455775Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

    2020-11-11T08:55:02.456242Z 2 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing

    2020-11-11T08:55:02.456734Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

    2020-11-11T08:55:02.463972Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.

    2020-11-11T08:55:02.473774Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

    2020-11-11T08:55:03.131444Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

    2020-11-11T08:55:03.586137Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)

    2020-11-11T08:55:03.626812Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully

    2020-11-11T08:55:03.648069Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/lib/mysql/mysqlx.sock' bind-address: '::' port: 33060

    2020-11-11T08:55:03.719538Z 0 [Note] [MY-000000] [WSREP] Before binlog recovery (wsrep position: a2555643-2338-11eb-9e47-97282c20ffa4:21)

    2020-11-11T08:55:03.721102Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...

    2020-11-11T08:55:03.738572Z 0 [System] [MY-010232] [Server] XA crash recovery finished.

    2020-11-11T08:55:03.740355Z 0 [Note] [MY-000000] [WSREP] After binlog recovery (wsrep position: a2555643-2338-11eb-9e47-97282c20ffa4:21)

    2020-11-11T08:55:03.793289Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

    2020-11-11T08:55:03.802819Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/lib/mysql' in the path is accessible to all OS users. Consider choosing a different directory.

    2020-11-11T08:55:03.813542Z 0 [ERROR] [MY-000067] [Server] unknown variable 'wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar'.

    2020-11-11T08:55:03.816571Z 0 [ERROR] [MY-010119] [Server] Aborting

    2020-11-11T08:55:03.816745Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation

    2020-11-11T08:55:05.817227Z 0 [Note] [MY-000000] [WSREP] Server status change initializing -> disconnecting

    2020-11-11T08:55:05.817356Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

    2020-11-11T08:55:05.817445Z 0 [Note] [MY-000000] [Galera] Closing send monitor...

    2020-11-11T08:55:05.817506Z 0 [Note] [MY-000000] [Galera] Closed send monitor.

    2020-11-11T08:55:05.817570Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread

    2020-11-11T08:55:05.817645Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread

    2020-11-11T08:55:05.817740Z 1 [Note] [MY-000000] [WSREP] rollbacker thread exiting 1

    2020-11-11T08:55:05.818348Z 2 [ERROR] [MY-000000] [Galera] Exception: State wait was interrupted

    2020-11-11T08:55:05.818456Z 2 [ERROR] [MY-000000] [Galera] View callback failed. This is unrecoverable, restart required. (FATAL)

         at galera/src/replicator_smm.cpp:submit_view_info():2588

    2020-11-11T08:55:05.818488Z 2 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()

    2020-11-11T08:55:05.818523Z 2 [Note] [MY-000000] [Galera] mysqld: Terminated.

    08:55:05 UTC - mysqld got signal 11 ;

    Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

    Thread pointer: 0x7f91700008c0

    Attempting backtrace. You can use the following information to find out

    where mysqld died. If you see no messages after this, something went

    terribly wrong...

    stack_bottom = 7f918585bc60 thread_stack 0x46000

    2020-11-11T08:55:05.820295Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend

    2020-11-11T08:55:05.820872Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0

    2020-11-11T08:55:05.820945Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

    view ((empty))

    2020-11-11T08:55:05.821271Z 0 [Note] [MY-000000] [Galera] gcomm: closed

    2020-11-11T08:55:05.821365Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.

    2020-11-11T08:55:05.821495Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]

    2020-11-11T08:55:05.821543Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.

    2020-11-11T08:55:05.821584Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> CLOSED (TO: 1)

    2020-11-11T08:55:05.821630Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success

    2020-11-11T08:55:05.822808Z 0 [Note] [MY-000000] [Galera] recv_thread() joined.

    2020-11-11T08:55:05.823081Z 0 [Note] [MY-000000] [Galera] Closing replication queue.

    2020-11-11T08:55:05.823274Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

    /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x1fd3a0d]

    /usr/sbin/mysqld(handle_fatal_signal+0x303) [0x1166583]

    /lib64/libpthread.so.0(+0xf630) [0x7f91a0042630]

    /lib64/libc.so.6(abort+0x297) [0x7f919dee2bc7]

    /usr/lib64/libgalera_smm.so(+0x1e6c7) [0x7f91909ec6c7]

    /usr/lib64/libgalera_smm.so(+0x1ea54c) [0x7f9190bb854c]

    /usr/lib64/libgalera_smm.so(+0x1f8c51) [0x7f9190bc6c51]

    /usr/lib64/libgalera_smm.so(+0x1c8918) [0x7f9190b96918]

    /usr/lib64/libgalera_smm.so(+0x1c8b98) [0x7f9190b96b98]

    /usr/lib64/libgalera_smm.so(+0x1f0f60) [0x7f9190bbef60]

    /usr/lib64/libgalera_smm.so(+0x2103cc) [0x7f9190bde3cc]

    /usr/sbin/mysqld(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*)+0xe) [0x272c13e]

    /usr/sbin/mysqld() [0x11a9b54]

    /usr/sbin/mysqld(start_wsrep_THD+0x331) [0xebe6e1]

    /usr/sbin/mysqld() [0x24e37e9]

    /lib64/libpthread.so.0(+0x7ea5) [0x7f91a003aea5]

    /lib64/libc.so.6(clone+0x6d) [0x7f919dfa98dd]


    Trying to get some variables.

    Some pointers may be invalid and cause the dump to abort.

    Query (0): is an invalid pointer

    Connection ID (thread ID): 2

    Status: KILL_CONNECTION


    You may download the Percona XtraDB Cluster operations manual by visiting

    http://www.percona.com/software/percona-xtradb-cluster/. You may find information

    in the manual which will help you identify the cause of the crash.

    Writing a core file

    [email protected]:~/sp-composer/kub/helm$ 

  • matthewbmatthewb Senior [email protected] Percona Staff Role

    @Laxminarayana Please use the forum tools to format your output. Hard to read when you just copy/paste.

    I can see the error right here in your logs;

    2020-11-11T08:55:03.813542Z 0 [ERROR] [MY-000067] [Server] unknown variable 'wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar'.
    
    2020-11-11T08:55:03.816571Z 0 [ERROR] [MY-010119] [Server] Aborting
    

    Are you sure you are using the latest version of the operator? Looks like v1.6.0 is the current GA version. Please upgrade to that version of the operator before attempting again. If you are indeed on 1.6.0, then this is a serious bug in the operator that needs to be addressed. The parameter wsrep_sst_auth was removed in PXC8 and should not be present in the scripts.

  • LaxminarayanaLaxminarayana Current User Role Contributor
    edited November 17

    Thanks for your response. I have removed parameter wsrep_sst_auth but it wasn't worked. We are using helm for deployment but I don't see updated chart version for Percona Xtradb Cluster and also we are doing continues deployment using argo cd. Here I have attached generated manifest file. Please look into manifest file and help me to resolve this issue.


  • matthewbmatthewb Senior [email protected] Percona Staff Role

    @Laxminarayana After removing that parameter, what do the mysql logs say?

  • LaxminarayanaLaxminarayana Current User Role Contributor

    @matthewb Thanks for quick response. Here is the log details.

    [email protected]:~/sp-composer/kub/helm$ kubectl logs percona-xtradb-cluster-pxc-0 -c database -n default

    + . /startup-scripts/functions.sh

    ++ awk ' { print $1 } '

    ++ hostname -i

    + ipaddr=172.17.0.10

    ++ hostname

    + hostname=percona-xtradb-cluster-pxc-0

    + echo 'I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10'

    I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10

    + '[' '' = - ']'

    ++ resolveip -s percona-xtradb-cluster-pxc-repl

    /startup-scripts/entrypoint.sh: line 19: resolveip: command not found

    ++ echo ''

    + cluster_join=

    + [[ -z '' ]]

    + echo 'I am the Primary Node'

    + init_mysql

    + DATADIR=/var/lib/mysql

    + '[' '!' -e /var/lib/mysql/mysql ']'

    I am the Primary Node

    + write_password_file

    + [[ -n ohchohtaigheecaemeeDaivo3wai3Dae ]]

    + cat

    + exec mysqld --user=mysql --wsrep_cluster_name=percona-xtradb-cluster-pxc --wsrep_node_name=percona-xtradb-cluster-pxc-0 --wsrep_cluster_address=gcomm:// --wsrep_sst_method=xtrabackup-v2

    2020-11-17T13:31:45.018951Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes and --log-slow-slave-statements have no effect if --slow-query-log is not set

    2020-11-17T13:31:45.020825Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11.3) starting as process 1

    2020-11-17T13:31:45.032709Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1

    2020-11-17T13:31:45.032923Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'

    2020-11-17T13:31:45.040668Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy <[email protected]> loaded successfully.

    2020-11-17T13:31:45.040843Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.

    2020-11-17T13:31:45.042434Z 0 [Note] [MY-000000] [Galera] Found saved state: 801d098a-28d8-11eb-9842-5a65a898fbb3:-1, safe_to_bootstrap: 1

    2020-11-17T13:31:45.043839Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:

    Version: 2

    UUID: 801d098a-28d8-11eb-9842-5a65a898fbb3

    Seqno: -1 - -1

    Offset: -1

    Synced: 0

    2020-11-17T13:31:45.044021Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 801d098a-28d8-11eb-9842-5a65a898fbb3, offset: -1

    2020-11-17T13:31:45.044243Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% (    0/134217752 bytes) complete.

    2020-11-17T13:31:45.044485Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.

    2020-11-17T13:31:45.044624Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1

    2020-11-17T13:31:45.044820Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/184 bytes) complete.

    2020-11-17T13:31:45.044967Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (184/184 bytes) complete.

    2020-11-17T13:31:45.045119Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers

    2020-11-17T13:31:45.045301Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 0/134217728

    2020-11-17T13:31:45.046821Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.17.0.10; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem; 

    2020-11-17T13:31:45.063441Z 0 [Note] [MY-000000] [WSREP] Starting replication

    2020-11-17T13:31:45.063687Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1

    2020-11-17T13:31:45.063901Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1

    2020-11-17T13:31:45.064162Z 0 [Note] [MY-000000] [Galera] protonet asio version 0

    2020-11-17T13:31:45.064464Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.

    2020-11-17T13:31:45.064696Z 0 [Note] [MY-000000] [Galera] initializing ssl context

    2020-11-17T13:31:45.065167Z 0 [Note] [MY-000000] [Galera] backend: asio

    2020-11-17T13:31:45.065414Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 

    2020-11-17T13:31:45.065704Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

    2020-11-17T13:31:45.065969Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

    2020-11-17T13:31:45.066904Z 0 [Note] [MY-000000] [Galera] GMCast version 0

    2020-11-17T13:31:45.067224Z 0 [Note] [MY-000000] [Galera] (3c5cd0f9, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567

    2020-11-17T13:31:45.067466Z 0 [Note] [MY-000000] [Galera] (3c5cd0f9, 'ssl://0.0.0.0:4567') multicast: , ttl: 1

    2020-11-17T13:31:45.068063Z 0 [Note] [MY-000000] [Galera] EVS version 1

    2020-11-17T13:31:45.068586Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group 'percona-xtradb-cluster-pxc'

    2020-11-17T13:31:45.069217Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery

    2020-11-17T13:31:45.071242Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1

    2020-11-17T13:31:45.072234Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1

    2020-11-17T13:31:45.072804Z 0 [Note] [MY-000000] [Galera] Node 3c5cd0f9 state primary

    2020-11-17T13:31:45.073074Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

    view (view_id(PRIM,3c5cd0f9,1)

    memb {

        3c5cd0f9,0

        }

    joined {

        }

    left {

        }

    partitioned {

        }

    )

    2020-11-17T13:31:45.073330Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk

    2020-11-17T13:31:45.077210Z 0 [Note] [MY-000000] [Galera] gcomm: connected

    2020-11-17T13:31:45.077632Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636

    2020-11-17T13:31:45.077963Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)

    2020-11-17T13:31:45.078203Z 0 [Note] [MY-000000] [Galera] Opened channel 'percona-xtradb-cluster-pxc'

    2020-11-17T13:31:45.078930Z 1 [Note] [MY-000000] [WSREP] Starting applier thread 1

    2020-11-17T13:31:45.079433Z 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2

    2020-11-17T13:31:45.080111Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1

    2020-11-17T13:31:45.081267Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 3c5f0f8f-28d9-11eb-aa41-3766c17ee473

    2020-11-17T13:31:45.081854Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986

    2020-11-17T13:31:45.082047Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986

    2020-11-17T13:31:45.082374Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 3c5f23f9-28d9-11eb-9ffa-62abfba0f986 from 0 (percona-xtradb-cluster-pxc-0)

    2020-11-17T13:31:45.082659Z 0 [Note] [MY-000000] [Galera] Quorum results:

        version  = 6,

        component = PRIMARY,

        conf_id  = 0,

        members  = 1/1 (primary/total),

        act_id   = 0,

        last_appl. = 0,

        protocols = 2/10/4 (gcs/repl/appl),

        vote policy= 0,

        group UUID = 3c5f0f8f-28d9-11eb-aa41-3766c17ee473

    2020-11-17T13:31:45.082923Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]

    2020-11-17T13:31:45.083134Z 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (1)

    2020-11-17T13:31:45.083519Z 1 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered

    2020-11-17T13:31:45.083750Z 1 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1

    2020-11-17T13:31:45.083948Z 1 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)

    2020-11-17T13:31:45.084216Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d

    2020-11-17T13:31:45.084593Z 1 [Note] [MY-000000] [Galera] Server percona-xtradb-cluster-pxc-0 connected to cluster at position 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:1 with ID 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d

    2020-11-17T13:31:45.084888Z 1 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected

    2020-11-17T13:31:45.085347Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

    2020-11-17T13:31:45.085832Z 1 [Note] [MY-000000] [Galera] ####### ST not required

    2020-11-17T13:31:45.086230Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no

    2020-11-17T13:31:45.086559Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

    2020-11-17T13:31:45.086869Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5

    2020-11-17T13:31:45.087107Z 1 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 1

    2020-11-17T13:31:45.087421Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

    2020-11-17T13:31:45.087697Z 1 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1

    2020-11-17T13:31:45.088032Z 0 [Note] [MY-000000] [Galera] Member 0.0 (percona-xtradb-cluster-pxc-0) synced with group.

    2020-11-17T13:31:45.088360Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 1)

    2020-11-17T13:31:45.088710Z 1 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped

    2020-11-17T13:31:45.088995Z 1 [Note] [MY-000000] [Galera] GCache history reset: old(801d098a-28d8-11eb-9842-5a65a898fbb3:1 -> 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:0

    2020-11-17T13:31:45.184535Z 1 [Note] [MY-000000] [Galera] Flushing memory map to disk...

    2020-11-17T13:31:45.437682Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 184 bytes

    2020-11-17T13:31:45.438403Z 1 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 0/1 locked buffers

    2020-11-17T13:31:45.440525Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 1

    2020-11-17T13:31:45.440882Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1

    2020-11-17T13:31:45.441627Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1

    2020-11-17T13:31:45.442430Z 1 [Note] [MY-000000] [Galera] ================================================

    View:

     id: 3c5f0f8f-28d9-11eb-aa41-3766c17ee473:1

     status: primary

     protocol_version: 4

     capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO

     final: no

     own_index: 0

     members(1):

        0: 3c5cd0f9-28d9-11eb-bc24-7f4878a4454d, percona-xtradb-cluster-pxc-0

    =================================================

    2020-11-17T13:31:45.442791Z 1 [Note] [MY-000000] [WSREP] Server status change connected -> joiner

    2020-11-17T13:31:45.443006Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

    2020-11-17T13:31:45.443491Z 1 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing

    2020-11-17T13:31:45.443789Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

    2020-11-17T13:31:45.444826Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.

    2020-11-17T13:31:45.452781Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

    2020-11-17T13:31:46.054042Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

    2020-11-17T13:31:46.240561Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)

    2020-11-17T13:31:46.246726Z 3 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement '#! PXC_SECTION::START

    #

    # All the commands between PXC_SECTION::START and PXC_SECTION::END will be

    # run whenever PXC starts up. This is to ensure that the PXC internal session

    # user always exists (this user is also created on upgrade, but in certain

    # upgrade scenarios, such as upgrading from a PS 8.0 node, it will not

    # run the upgrade code).

    #

    # Note: The PXC_SECTION markers must come after an SQL statement. Due to the

    # way the code is generated, do not place any comments before the markers.

    #

    # SQL commands to create the PXC internal session user and role that is used

    # by the SST process

    #

    # mysql.pxc.internal.session

    #  See the comments in mysql_system_tables.sql

    #

    # These are the values for

    # GRANT CREATE USER ON *.* TO 'mysql.pxc.internal.session'@localhost WITH GRANT OPTION;

    # GRANT SUPER ON *.* TO 'mysql.pxc.internal.session'@localhost WITH GRANT OPTION;

    # GRANT RELOAD ON *.* TO 'mysql.pxc.internal.session'@localhost WITH GRANT OPTION;

    #

    #INSERT IGNORE INTO mysql.user VALUES ('localhost','mysql.pxc.internal.session','N','N','N','N','N','N','Y','N','N','N','Y','N','N','N','N','Y','N','N','N','N','N','N','N','N','N','Y','N','N','N','','','','',0,0,0,0,'caching_sha2_password','$A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED','N',CURRENT_TIMESTAMP, NULL,'Y','N','N',NULL,NULL,NULL,NULL);

    # (Use this since the CREATE DATABASE doesn't work when the grant is in a role)

    # So we assign the mysql.pxc.internal.session root privileges with grant option

    # These are the values for

    # GRANT ALL PRIVILEGES ON *.* TO 'mysql.pxc.internal.session'@localhost WITH GRANT OPTION;

    # GRANT BACKUP_ADMIN, LOCK TABLES, PROCESS, RELOAD, REPLICATION CLIENT, SUPER ON *.* TO 'mysql.pxc.internal.session'@localhost WITH GRANT OPTION;

    #

    INSERT IGNORE INTO mysql.user VALUES ('localhost','mysql.pxc.internal.session','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'caching_sha2_password','$A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED','N',CURRENT_TIMESTAMP,NULL,'Y','Y','Y',NULL,NULL,NULL,NULL);

    ' failed with error code = 1146, error message = 'Table 'mysql.user' doesn't exist'.

    2020-11-17T13:31:46.250118Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.

    2020-11-17T13:31:46.251086Z 0 [ERROR] [MY-010119] [Server] Aborting

    2020-11-17T13:31:46.251134Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation

    2020-11-17T13:31:48.252275Z 0 [Note] [MY-000000] [WSREP] Server status change initializing -> disconnecting

    2020-11-17T13:31:48.252969Z 2 [Note] [MY-000000] [WSREP] rollbacker thread exiting 2

    2020-11-17T13:31:48.254028Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

    2020-11-17T13:31:48.254432Z 0 [Note] [MY-000000] [Galera] Closing send monitor...

    2020-11-17T13:31:48.255061Z 0 [Note] [MY-000000] [Galera] Closed send monitor.

    2020-11-17T13:31:48.255603Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread

    2020-11-17T13:31:48.256179Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread

    2020-11-17T13:31:48.257151Z 1 [ERROR] [MY-000000] [Galera] Exception: State wait was interrupted

    2020-11-17T13:31:48.257370Z 1 [ERROR] [MY-000000] [Galera] View callback failed. This is unrecoverable, restart required. (FATAL)

         at galera/src/replicator_smm.cpp:submit_view_info():2588

    2020-11-17T13:31:48.257560Z 1 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()

    2020-11-17T13:31:48.257713Z 1 [Note] [MY-000000] [Galera] mysqld: Terminated.

    13:31:48 UTC - mysqld got signal 11 ;

    Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

    Thread pointer: 0x7f633c0008c0

    Attempting backtrace. You can use the following information to find out

    where mysqld died. If you see no messages after this, something went

    terribly wrong...

    stack_bottom = 7f634810cc60 thread_stack 0x46000

    2020-11-17T13:31:48.259551Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend

    2020-11-17T13:31:48.259776Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0

    2020-11-17T13:31:48.259861Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

    view ((empty))

    2020-11-17T13:31:48.260343Z 0 [Note] [MY-000000] [Galera] gcomm: closed

    2020-11-17T13:31:48.261449Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.

    2020-11-17T13:31:48.262479Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]

    2020-11-17T13:31:48.262875Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.

    2020-11-17T13:31:48.264147Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> CLOSED (TO: 1)

    2020-11-17T13:31:48.264645Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success

    2020-11-17T13:31:48.265169Z 0 [Note] [MY-000000] [Galera] recv_thread() joined.

    2020-11-17T13:31:48.265579Z 0 [Note] [MY-000000] [Galera] Closing replication queue.

    2020-11-17T13:31:48.265827Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

    /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x1fd3a0d]

    /usr/sbin/mysqld(handle_fatal_signal+0x303) [0x1166583]

    /lib64/libpthread.so.0(+0xf630) [0x7f63628f3630]

    /lib64/libc.so.6(abort+0x297) [0x7f6360793bc7]

    /usr/lib64/libgalera_smm.so(+0x1e6c7) [0x7f635329d6c7]

    /usr/lib64/libgalera_smm.so(+0x1ea54c) [0x7f635346954c]

    /usr/lib64/libgalera_smm.so(+0x1f8c51) [0x7f6353477c51]

    /usr/lib64/libgalera_smm.so(+0x1c8918) [0x7f6353447918]

    /usr/lib64/libgalera_smm.so(+0x1c8b98) [0x7f6353447b98]

    /usr/lib64/libgalera_smm.so(+0x1f0f60) [0x7f635346ff60]

    /usr/lib64/libgalera_smm.so(+0x2103cc) [0x7f635348f3cc]

    /usr/sbin/mysqld(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*)+0xe) [0x272c13e]

    /usr/sbin/mysqld() [0x11a9b54]

    /usr/sbin/mysqld(start_wsrep_THD+0x331) [0xebe6e1]

    /usr/sbin/mysqld() [0x24e37e9]

    /lib64/libpthread.so.0(+0x7ea5) [0x7f63628ebea5]

    /lib64/libc.so.6(clone+0x6d) [0x7f636085a8dd]


    Trying to get some variables.

    Some pointers may be invalid and cause the dump to abort.

    Query (0): is an invalid pointer

    Connection ID (thread ID): 1

    Status: KILL_CONNECTION


    You may download the Percona XtraDB Cluster operations manual by visiting

    http://www.percona.com/software/percona-xtradb-cluster/. You may find information

    in the manual which will help you identify the cause of the crash.

    Writing a core file

    [email protected]:~/sp-composer/kub/helm$ 

  • matthewbmatthewb Senior [email protected] Percona Staff Role

    @Laxminarayana Please edit your post and use the formatting tools to format your code as "code" so I can read it.

Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.