Log of wsrep recovery (--wsrep-recover): INFO: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery_verbose.OhOTDU' --pid-file='/var/lib/mysql/file.pid' ERROR! WSREP: Failed to recover position: 2020-09-08T15:30:16.578461Z 0 [Warning] [MY-010091] [Server] Can't create test file /var/lib/mysql/mysqld_tmp_file_case_insensitive_test.lower-test 2020-09-08T15:30:16.578541Z 0 [Note] [MY-010949] [Server] Basedir set to /usr/. 2020-09-08T15:30:16.578556Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.19-10) starting as process 10571 2020-09-08T15:30:16.582119Z 0 [Warning] [MY-010091] [Server] Can't create test file /var/lib/mysql/mysqld_tmp_file_case_insensitive_test.lower-test 2020-09-08T15:30:16.582129Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive 2020-09-08T15:30:16.599202Z 0 [Note] [MY-012366] [InnoDB] Using Linux native AIO 2020-09-08T15:30:16.599872Z 0 [Note] [MY-010747] [Server] Plugin 'FEDERATED' is disabled. 2020-09-08T15:30:16.606897Z 1 [Note] [MY-012932] [InnoDB] PUNCH HOLE support available 2020-09-08T15:30:16.606960Z 1 [Note] [MY-012943] [InnoDB] Mutexes and rw_locks use GCC atomic builtins 2020-09-08T15:30:16.607002Z 1 [Note] [MY-012944] [InnoDB] Uses event mutexes 2020-09-08T15:30:16.607043Z 1 [Note] [MY-012945] [InnoDB] GCC builtin __atomic_thread_fence() is used for memory barrier 2020-09-08T15:30:16.607094Z 1 [Note] [MY-012948] [InnoDB] Compressed tables use zlib 1.2.11 2020-09-08T15:30:16.608870Z 1 [Note] [MY-013251] [InnoDB] Number of pools: 1 2020-09-08T15:30:16.609055Z 1 [Note] [MY-012951] [InnoDB] Using CPU crc32 instructions 2020-09-08T15:30:16.609617Z 1 [Note] [MY-012203] [InnoDB] Directories to scan './' 2020-09-08T15:30:16.609726Z 1 [Note] [MY-012204] [InnoDB] Scanning './' 2020-09-08T15:30:16.609999Z 1 [Note] [MY-013503] [InnoDB] Directory '/var/lib/mysql/.bash_history' will not be scanned because it is a hidden directory. 2020-09-08T15:30:16.612353Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 4 files. 2020-09-08T15:30:16.613430Z 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 128.000000M, instances = 1, chunk size =128.000000M 2020-09-08T15:30:16.621276Z 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool 2020-09-08T15:30:16.755895Z 0 [Note] [MY-011952] [InnoDB] If the mysqld execution user is authorized, page cleaner and LRU manager thread priority can be changed. See the man page of setpriority(). 2020-09-08T15:30:16.775720Z 1 [Note] [MY-011825] [InnoDB] Crash recovery did not find the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite 2020-09-08T15:30:16.857231Z 1 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 19733552, whereas checkpoint_lsn = 19733616 2020-09-08T15:30:16.860611Z 1 [Note] [MY-013083] [InnoDB] Log background threads are being started... 2020-09-08T15:30:17.109754Z 1 [Note] [MY-011825] [InnoDB] Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 3932160 bytes 2020-09-08T15:30:17.109978Z 1 [Note] [MY-012532] [InnoDB] Applying a batch of 0 redo log records ... 2020-09-08T15:30:17.110064Z 1 [Note] [MY-012535] [InnoDB] Apply batch completed! 2020-09-08T15:30:17.110245Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'. 2020-09-08T15:30:17.111254Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'. 2020-09-08T15:30:17.114298Z 1 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces. 2020-09-08T15:30:17.118153Z 1 [Note] [MY-011980] [InnoDB] GTID recovery trx_no: 1261 2020-09-08T15:30:17.168644Z 1 [Note] [MY-012923] [InnoDB] Creating shared tablespace for temporary tables 2020-09-08T15:30:17.168991Z 1 [Note] [MY-012265] [InnoDB] Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-09-08T15:30:17.203850Z 1 [Note] [MY-012266] [InnoDB] File './ibtmp1' size is now 12 MB. 2020-09-08T15:30:17.208041Z 1 [Note] [MY-011825] [InnoDB] Scanning temp tablespace dir:'./#innodb_temp/' 2020-09-08T15:30:17.262384Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active. 2020-09-08T15:30:17.293465Z 1 [Note] [MY-012976] [InnoDB] Percona XtraDB (http://www.percona.com) 8.0.19-10 started; log sequence number 19733616 2020-09-08T15:30:17.308972Z 1 [ERROR] [MY-011087] [Server] Different lower_case_table_names settings for server ('2') and data dictionary ('0'). 2020-09-08T15:30:17.309291Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed. 2020-09-08T15:30:17.309527Z 0 [ERROR] [MY-010119] [Server] Aborting 2020-09-08T15:30:17.309580Z 0 [Note] [MY-010120] [Server] Binlog end 2020-09-08T15:30:17.309660Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM' 2020-09-08T15:30:17.309689Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'InnoDB' 2020-09-08T15:30:17.309725Z 0 [Note] [MY-013072] [InnoDB] Starting shutdown... 2020-09-08T15:30:17.812046Z 0 [Note] [MY-013084] [InnoDB] Log background threads are being closed... 2020-09-08T15:30:18.321925Z 0 [Note] [MY-012980] [InnoDB] Shutdown completed; log sequence number 19733616 2020-09-08T15:30:18.322806Z 0 [Note] [MY-012255] [InnoDB] Removed temporary tablespace data file: "ibtmp1" 2020-09-08T15:30:18.322855Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV' 2020-09-08T15:30:18.323233Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.19-10) Percona XtraDB Cluster (GPL), Release rel10, Revision 727f180, WSREP version 26.4.3. 2020-09-08T15:30:18.679542Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.19-10) starting as process 10604 2020-09-08T15:30:18.682155Z 0 [Note] [MY-000000] [WSREP] New joining cluster node found needed SSL artifacts 2020-09-08T15:30:18.682204Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib64/galera4/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1 2020-09-08T15:30:18.682250Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib64/galera4/libgalera_smm.so' 2020-09-08T15:30:18.691031Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy loaded successfully. 2020-09-08T15:30:18.691094Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration. 2020-09-08T15:30:18.695490Z 0 [Note] [MY-000000] [Galera] Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1 2020-09-08T15:30:18.703603Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble: Version: 2 UUID: 0bfa5e83-ee9a-11ea-87c7-bef4ee9147f0 Seqno: -1 - -1 Offset: -1 Synced: 0 2020-09-08T15:30:18.703661Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 0bfa5e83-ee9a-11ea-87c7-bef4ee9147f0, offset: -1 2020-09-08T15:30:18.703790Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete. 2020-09-08T15:30:18.979141Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete. 2020-09-08T15:30:18.979237Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: didn't recover any events. 2020-09-08T15:30:18.980042Z 0 [Note] [MY-000000] [Galera] Complete reset of the galera cache 2020-09-08T15:30:19.283228Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk... 2020-09-08T15:30:20.114626Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 18.x.x.x; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem; 2020-09-08T15:30:20.128107Z 0 [Note] [MY-000000] [WSREP] Starting replication 2020-09-08T15:30:20.128214Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0 2020-09-08T15:30:20.128283Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1 2020-09-08T15:30:20.128405Z 0 [Note] [MY-000000] [Galera] protonet asio version 0 2020-09-08T15:30:20.128735Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums. 2020-09-08T15:30:20.128858Z 0 [Note] [MY-000000] [Galera] initializing ssl context 2020-09-08T15:30:20.129117Z 0 [Note] [MY-000000] [Galera] backend: asio 2020-09-08T15:30:20.129283Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 2020-09-08T15:30:20.129468Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk successful 2020-09-08T15:30:20.131437Z 0 [Note] [MY-000000] [Galera] GMCast version 0 2020-09-08T15:30:20.131634Z 0 [Note] [MY-000000] [Galera] (58a751c7, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567 2020-09-08T15:30:20.131724Z 0 [Note] [MY-000000] [Galera] (58a751c7, 'ssl://0.0.0.0:4567') multicast: , ttl: 1 2020-09-08T15:30:20.132099Z 0 [Note] [MY-000000] [Galera] EVS version 1 2020-09-08T15:30:20.132263Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-cluster', peer '18.n.n.n:' 2020-09-08T15:30:20.140876Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://18.n.n.n:4567 local endpoint ssl://172.y.y.y:37282 cipher: TLS_AES_256_GCM_SHA384 compression: none 2020-09-08T15:30:20.143149Z 0 [Note] [MY-000000] [Galera] (58a751c7, 'ssl://0.0.0.0:4567') connection established to 9b4b1c19 ssl://18.n.n.n:4567 2020-09-08T15:30:20.143335Z 0 [Note] [MY-000000] [Galera] (58a751c7, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2020-09-08T15:30:20.387965Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1 2020-09-08T15:30:20.388129Z 0 [Note] [MY-000000] [Galera] declaring 9b4b1c19 at ssl://18.n.n.n:4567 stable 2020-09-08T15:30:20.388255Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1 2020-09-08T15:30:20.388892Z 0 [Note] [MY-000000] [Galera] Node 9b4b1c19 state primary 2020-09-08T15:30:20.389490Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view (view_id(PRIM,58a751c7,26) memb { 58a751c7,0 9b4b1c19,0 } joined { } left { } partitioned { } ) 2020-09-08T15:30:20.389609Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk 2020-09-08T15:30:20.391157Z 0 [Note] [MY-000000] [Galera] clear restored view 2020-09-08T15:30:20.632918Z 0 [Note] [MY-000000] [Galera] gcomm: connected 2020-09-08T15:30:20.633121Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636 2020-09-08T15:30:20.633311Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0) 2020-09-08T15:30:20.633427Z 0 [Note] [MY-000000] [Galera] Opened channel 'pxc-cluster' 2020-09-08T15:30:20.633926Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1 2020-09-08T15:30:20.634215Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2 2020-09-08T15:30:20.634405Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 2020-09-08T15:30:20.634645Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 34a7e0bb-f1e8-11ea-b484-0ae95baea52b 2020-09-08T15:30:20.635494Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 34a7e0bb-f1e8-11ea-b484-0ae95baea52b 2020-09-08T15:30:20.636296Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 34a7e0bb-f1e8-11ea-b484-0ae95baea52b from 0 (pxc-cluster-node-2) 2020-09-08T15:30:20.636494Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 34a7e0bb-f1e8-11ea-b484-0ae95baea52b from 1 (pxc-cluster-node-1) 2020-09-08T15:30:20.636645Z 0 [Note] [MY-000000] [Galera] Quorum results: version = 6, component = PRIMARY, conf_id = 1, members = 1/2 (primary/total), act_id = 56, last_appl. = 55, protocols = 2/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 0bfa5e83-ee9a-11ea-87c7-bef4ee9147f0 2020-09-08T15:30:20.636848Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141] 2020-09-08T15:30:20.637000Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 57) 2020-09-08T15:30:20.637178Z 2 [Note] [MY-000000] [Galera] ####### processing CC 57, local, ordered 2020-09-08T15:30:20.637335Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1 2020-09-08T15:30:20.637503Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3) 2020-09-08T15:30:20.637667Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 58a751c7-f1e7-11ea-8f49-5f8d15e1c648 2020-09-08T15:30:20.637861Z 2 [Note] [MY-000000] [Galera] Server pxc-cluster-node-2 connected to cluster at position 0bfa5e83-ee9a-11ea-87c7-bef4ee9147f0:57 with ID 58a751c7-f1e7-11ea-8f49-5f8d15e1c648 2020-09-08T15:30:20.638025Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected 2020-09-08T15:30:20.638220Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification. 2020-09-08T15:30:20.638408Z 2 [Note] [MY-000000] [Galera] State transfer required: Group state: 0bfa5e83-ee9a-11ea-87c7-bef4ee9147f0:57 Local state: 00000000-0000-0000-0000-000000000000:-1 2020-09-08T15:30:20.638576Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner 2020-09-08T15:30:20.638760Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification. 2020-09-08T15:30:20.639395Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '18.x.x.x' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '10604' --mysqld-version '8.0.19-10' '' ) 2020-09-08T15:30:20.639840Z 0 [ERROR] [MY-000000] [WSREP] posix_spawnp() failed: 13 (Permission denied) 2020-09-08T15:30:20.640489Z 0 [ERROR] [MY-000000] [WSREP] Failed to execute: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '18.x.x.x' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '10604' --mysqld-version '8.0.19-10' '' : 13 (Permission denied) 2020-09-08T15:30:20.640618Z 2 [ERROR] [MY-000000] [WSREP] Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable. 2020-09-08T15:30:20.640724Z 2 [ERROR] [MY-000000] [Galera] SST request callback failed. This is unrecoverable, restart required. 2020-09-08T15:30:20.640757Z 2 [Note] [MY-000000] [Galera] ReplicatorSMM::abort() 2020-09-08T15:30:20.640779Z 2 [Note] [MY-000000] [Galera] Closing send monitor... 2020-09-08T15:30:20.640793Z 2 [Note] [MY-000000] [Galera] Closed send monitor. 2020-09-08T15:30:20.640820Z 2 [Note] [MY-000000] [Galera] gcomm: terminating thread 2020-09-08T15:30:20.640837Z 2 [Note] [MY-000000] [Galera] gcomm: joining thread 2020-09-08T15:30:20.640895Z 2 [Note] [MY-000000] [Galera] gcomm: closing backend 2020-09-08T15:30:20.651794Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation 15:30:20 UTC - mysqld got signal 11 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x46000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x20231f1] /usr/sbin/mysqld(handle_fatal_signal+0x33b) [0x1231cab] /lib64/libpthread.so.0(+0x12dd0) [0x7ff903703dd0] /lib64/libc.so.6(_IO_feof+0x7) [0x7ff90144d9c7] /usr/sbin/mysqld() [0x126685e] /lib64/libpthread.so.0(+0x82de) [0x7ff9036f92de] /lib64/libc.so.6(clone+0x43) [0x7ff9014d0e83] You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/. You may find information in the manual which will help you identify the cause of the crash.