I am getting CrashLookBackOff error for Percona XtraDB Cluster 8.0.20 in Kubernetes

Logs:

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$ kubectl logs percona-xtradb-cluster-pxc-0 -c database -n default

  • . /startup-scripts/functions.sh

++ hostname -i

++ awk ’ { print $1 } ’

  • ipaddr=172.17.0.10

++ hostname

  • hostname=percona-xtradb-cluster-pxc-0

  • echo ‘I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10’

I AM percona-xtradb-cluster-pxc-0 - 172.17.0.10

  • ‘[’ ‘’ = - ‘]’

++ resolveip -s percona-xtradb-cluster-pxc-repl

/startup-scripts/entrypoint.sh: line 19: resolveip: command not found

++ echo ‘’

  • cluster_join=

  • [[ -z ‘’ ]]

  • echo ‘I am the Primary Node’

I am the Primary Node

  • init_mysql

  • DATADIR=/var/lib/mysql

  • ‘[’ ‘!’ -e /var/lib/mysql/mysql ‘]’

  • write_password_file

  • [[ -n ohchohtaigheecaemeeDaivo3wai3Dae ]]

  • cat

  • exec mysqld --user=mysql --wsrep_cluster_name=percona-xtradb-cluster-pxc --wsrep_node_name=percona-xtradb-cluster-pxc-0 --wsrep_cluster_address=gcomm:// --wsrep_sst_method=xtrabackup-v2 --wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar --wsrep_node_address=172.17.0.10 --pxc_strict_mode=MASTER

2020-11-11T08:55:01.770215Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes and --log-slow-slave-statements have no effect if --slow-query-log is not set

2020-11-11T08:55:01.777677Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11.3) starting as process 1

2020-11-11T08:55:01.786110Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1

2020-11-11T08:55:01.786220Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’

2020-11-11T08:55:01.791275Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy info@codership.com loaded successfully.

2020-11-11T08:55:01.791490Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.

2020-11-11T08:55:01.794837Z 0 [Note] [MY-000000] [Galera] Found saved state: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe:-1, safe_to_bootstrap: 1

2020-11-11T08:55:01.796754Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:

Version: 2

UUID: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe

Seqno: -1 - -1

Offset: -1

Synced: 0

2020-11-11T08:55:01.797015Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 2fa7d286-23fb-11eb-b83b-231d9f8d2cbe, offset: -1

2020-11-11T08:55:01.797560Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% ( 0/134217752 bytes) complete.

2020-11-11T08:55:01.902603Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan…100.0% (134217752/134217752 bytes) complete.

2020-11-11T08:55:01.902810Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1

2020-11-11T08:55:01.902987Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% ( 0/184 bytes) complete.

2020-11-11T08:55:01.903113Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan…100.0% (184/184 bytes) complete.

2020-11-11T08:55:01.903239Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers

2020-11-11T08:55:01.903357Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 0/134217728

2020-11-11T08:55:01.904638Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.17.0.10; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem;

2020-11-11T08:55:01.928344Z 0 [Note] [MY-000000] [WSREP] Starting replication

2020-11-11T08:55:01.928686Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1

2020-11-11T08:55:01.929245Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1

2020-11-11T08:55:01.929642Z 0 [Note] [MY-000000] [Galera] protonet asio version 0

2020-11-11T08:55:01.930507Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.

2020-11-11T08:55:01.931623Z 0 [Note] [MY-000000] [Galera] initializing ssl context

2020-11-11T08:55:01.933007Z 0 [Note] [MY-000000] [Galera] backend: asio

2020-11-11T08:55:01.933399Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0

2020-11-11T08:55:01.934128Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

2020-11-11T08:55:01.934289Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

2020-11-11T08:55:01.935964Z 0 [Note] [MY-000000] [Galera] GMCast version 0

2020-11-11T08:55:01.936419Z 0 [Note] [MY-000000] [Galera] (95a57b03, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567

2020-11-11T08:55:01.936777Z 0 [Note] [MY-000000] [Galera] (95a57b03, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1

2020-11-11T08:55:01.937883Z 0 [Note] [MY-000000] [Galera] EVS version 1

2020-11-11T08:55:01.938344Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group ‘percona-xtradb-cluster-pxc’

2020-11-11T08:55:01.938707Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery

2020-11-11T08:55:01.940309Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 → 1

2020-11-11T08:55:01.940842Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 → 1

2020-11-11T08:55:01.941347Z 0 [Note] [MY-000000] [Galera] Node 95a57b03 state primary

2020-11-11T08:55:01.941834Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view (view_id(PRIM,95a57b03,1)

memb {

95a57b03,0

}

joined {

}

left {

}

partitioned {

}

)

2020-11-11T08:55:01.942081Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk

2020-11-11T08:55:01.951665Z 0 [Note] [MY-000000] [Galera] gcomm: connected

2020-11-11T08:55:01.954121Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636

2020-11-11T08:55:01.955325Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)

2020-11-11T08:55:01.955859Z 0 [Note] [MY-000000] [Galera] Opened channel ‘percona-xtradb-cluster-pxc’

2020-11-11T08:55:01.960517Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1

2020-11-11T08:55:01.966240Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 95a9c4c2-23fb-11eb-81e9-d32431edcdd2

2020-11-11T08:55:01.969461Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 95aaaa86-23fb-11eb-ab85-6adf96a771ab

2020-11-11T08:55:01.969909Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 95aaaa86-23fb-11eb-ab85-6adf96a771ab

2020-11-11T08:55:01.970784Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 95aaaa86-23fb-11eb-ab85-6adf96a771ab from 0 (percona-xtradb-cluster-pxc-0)

2020-11-11T08:55:01.971242Z 0 [Note] [MY-000000] [Galera] Quorum results:

version  = 6,

component = PRIMARY,

conf_id  = 0,

members  = 1/1 (primary/total),

act_id   = 0,

last_appl. = 0,

protocols = 2/10/4 (gcs/repl/appl),

vote policy= 0,

group UUID = 95a9c4c2-23fb-11eb-81e9-d32431edcdd2

2020-11-11T08:55:01.971756Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]

2020-11-11T08:55:01.972103Z 0 [Note] [MY-000000] [Galera] Restored state OPEN → JOINED (1)

2020-11-11T08:55:01.972518Z 0 [Note] [MY-000000] [Galera] Member 0.0 (percona-xtradb-cluster-pxc-0) synced with group.

2020-11-11T08:55:01.973065Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 1)

2020-11-11T08:55:01.973406Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1

2020-11-11T08:55:01.973968Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2

2020-11-11T08:55:01.978752Z 2 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered

2020-11-11T08:55:01.993141Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1

2020-11-11T08:55:01.994465Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)

2020-11-11T08:55:01.995337Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 95a57b03-23fb-11eb-ad58-c242f987fcfd

2020-11-11T08:55:01.996329Z 2 [Note] [MY-000000] [Galera] Server percona-xtradb-cluster-pxc-0 connected to cluster at position 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:1 with ID 95a57b03-23fb-11eb-ad58-c242f987fcfd

2020-11-11T08:55:01.997288Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected → connected

2020-11-11T08:55:01.998214Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-11T08:55:01.999240Z 2 [Note] [MY-000000] [Galera] ####### ST not required

2020-11-11T08:55:02.000427Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no

2020-11-11T08:55:02.001559Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-11T08:55:02.002496Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5

2020-11-11T08:55:02.003646Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 → 1

2020-11-11T08:55:02.004846Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.

2020-11-11T08:55:02.005996Z 2 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1

2020-11-11T08:55:02.008501Z 2 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped

2020-11-11T08:55:02.010571Z 2 [Note] [MY-000000] [Galera] GCache history reset: old(2fa7d286-23fb-11eb-b83b-231d9f8d2cbe:1 → 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:0

2020-11-11T08:55:02.221936Z 2 [Note] [MY-000000] [Galera] Flushing memory map to disk…

2020-11-11T08:55:02.438824Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 184 bytes

2020-11-11T08:55:02.440566Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 0/1 locked buffers

2020-11-11T08:55:02.442264Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 1

2020-11-11T08:55:02.445908Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1

2020-11-11T08:55:02.448208Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1

2020-11-11T08:55:02.452316Z 2 [Note] [MY-000000] [Galera] ================================================

View:

id: 95a9c4c2-23fb-11eb-81e9-d32431edcdd2:1

status: primary

protocol_version: 4

capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO

final: no

own_index: 0

members(1):

0: 95a57b03-23fb-11eb-ad58-c242f987fcfd, percona-xtradb-cluster-pxc-0

=================================================

2020-11-11T08:55:02.455216Z 2 [Note] [MY-000000] [WSREP] Server status change connected → joiner

2020-11-11T08:55:02.455775Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-11T08:55:02.456242Z 2 [Note] [MY-000000] [WSREP] Server status change joiner → initializing

2020-11-11T08:55:02.456734Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-11T08:55:02.463972Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.

2020-11-11T08:55:02.473774Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2020-11-11T08:55:03.131444Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2020-11-11T08:55:03.586137Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)

2020-11-11T08:55:03.626812Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully

2020-11-11T08:55:03.648069Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: ‘/var/lib/mysql/mysqlx.sock’ bind-address: ‘::’ port: 33060

2020-11-11T08:55:03.719538Z 0 [Note] [MY-000000] [WSREP] Before binlog recovery (wsrep position: a2555643-2338-11eb-9e47-97282c20ffa4:21)

2020-11-11T08:55:03.721102Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…

2020-11-11T08:55:03.738572Z 0 [System] [MY-010232] [Server] XA crash recovery finished.

2020-11-11T08:55:03.740355Z 0 [Note] [MY-000000] [WSREP] After binlog recovery (wsrep position: a2555643-2338-11eb-9e47-97282c20ffa4:21)

2020-11-11T08:55:03.793289Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

2020-11-11T08:55:03.802819Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location ‘/var/lib/mysql’ in the path is accessible to all OS users. Consider choosing a different directory.

2020-11-11T08:55:03.813542Z 0 [ERROR] [MY-000067] [Server] unknown variable ‘wsrep_sst_auth=xtrabackup:rahyeQu5wee4eiSaepheW1aweiroh5ar’.

2020-11-11T08:55:03.816571Z 0 [ERROR] [MY-010119] [Server] Aborting

2020-11-11T08:55:03.816745Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation

2020-11-11T08:55:05.817227Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → disconnecting

2020-11-11T08:55:05.817356Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.

2020-11-11T08:55:05.817445Z 0 [Note] [MY-000000] [Galera] Closing send monitor…

2020-11-11T08:55:05.817506Z 0 [Note] [MY-000000] [Galera] Closed send monitor.

2020-11-11T08:55:05.817570Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread

2020-11-11T08:55:05.817645Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread

2020-11-11T08:55:05.817740Z 1 [Note] [MY-000000] [WSREP] rollbacker thread exiting 1

2020-11-11T08:55:05.818348Z 2 [ERROR] [MY-000000] [Galera] Exception: State wait was interrupted

2020-11-11T08:55:05.818456Z 2 [ERROR] [MY-000000] [Galera] View callback failed. This is unrecoverable, restart required. (FATAL)

 at galera/src/replicator_smm.cpp:submit_view_info():2588

2020-11-11T08:55:05.818488Z 2 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()

2020-11-11T08:55:05.818523Z 2 [Note] [MY-000000] [Galera] mysqld: Terminated.

08:55:05 UTC - mysqld got signal 11 ;

Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Thread pointer: 0x7f91700008c0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong…

stack_bottom = 7f918585bc60 thread_stack 0x46000

2020-11-11T08:55:05.820295Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend

2020-11-11T08:55:05.820872Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 → 0

2020-11-11T08:55:05.820945Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view ((empty))

2020-11-11T08:55:05.821271Z 0 [Note] [MY-000000] [Galera] gcomm: closed

2020-11-11T08:55:05.821365Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.

2020-11-11T08:55:05.821495Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]

2020-11-11T08:55:05.821543Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.

2020-11-11T08:55:05.821584Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED → CLOSED (TO: 1)

2020-11-11T08:55:05.821630Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success

2020-11-11T08:55:05.822808Z 0 [Note] [MY-000000] [Galera] recv_thread() joined.

2020-11-11T08:55:05.823081Z 0 [Note] [MY-000000] [Galera] Closing replication queue.

2020-11-11T08:55:05.823274Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x1fd3a0d]

/usr/sbin/mysqld(handle_fatal_signal+0x303) [0x1166583]

/lib64/libpthread.so.0(+0xf630) [0x7f91a0042630]

/lib64/libc.so.6(abort+0x297) [0x7f919dee2bc7]

/usr/lib64/libgalera_smm.so(+0x1e6c7) [0x7f91909ec6c7]

/usr/lib64/libgalera_smm.so(+0x1ea54c) [0x7f9190bb854c]

/usr/lib64/libgalera_smm.so(+0x1f8c51) [0x7f9190bc6c51]

/usr/lib64/libgalera_smm.so(+0x1c8918) [0x7f9190b96918]

/usr/lib64/libgalera_smm.so(+0x1c8b98) [0x7f9190b96b98]

/usr/lib64/libgalera_smm.so(+0x1f0f60) [0x7f9190bbef60]

/usr/lib64/libgalera_smm.so(+0x2103cc) [0x7f9190bde3cc]

/usr/sbin/mysqld(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*)+0xe) [0x272c13e]

/usr/sbin/mysqld() [0x11a9b54]

/usr/sbin/mysqld(start_wsrep_THD+0x331) [0xebe6e1]

/usr/sbin/mysqld() [0x24e37e9]

/lib64/libpthread.so.0(+0x7ea5) [0x7f91a003aea5]

/lib64/libc.so.6(clone+0x6d) [0x7f919dfa98dd]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0): is an invalid pointer

Connection ID (thread ID): 2

Status: KILL_CONNECTION

You may download the Percona XtraDB Cluster operations manual by visiting

http://www.percona.com/software/percona-xtradb-cluster/. You may find information

in the manual which will help you identify the cause of the crash.

Writing a core file

vagrant@ubuntu-xenial:~/sp-composer/kub/helm$