Node is not starting 5.6

MySQL & MariaDB Percona Server for MySQL 5.6
1

Hello,

I’ve 2 nodes one running as bootstrap and i need the other node to join the cluster. But i got this errors in mysql error log.

The second node LOG:

Blockquote
Log of wsrep recovery (–wsrep-recover):
2023-04-06 15:57:04 16022 [Note] Plugin ‘FEDERATED’ is disabled.
2023-04-06 15:57:04 16022 [Note] InnoDB: Using atomics to ref count buffer pool pages
2023-04-06 15:57:04 16022 [Note] InnoDB: The InnoDB memory heap is disabled
2023-04-06 15:57:04 16022 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-04-06 15:57:04 16022 [Note] InnoDB: Memory barrier is not used
2023-04-06 15:57:04 16022 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-04-06 15:57:04 16022 [Note] InnoDB: Using Linux native AIO
2023-04-06 15:57:04 16022 [Note] InnoDB: Using CPU crc32 instructions
2023-04-06 15:57:04 16022 [Note] InnoDB: Initializing buffer pool, size = 6.3G
2023-04-06 15:57:04 16022 [Note] InnoDB: Completed initialization of buffer pool
2023-04-06 15:57:04 16022 [Note] InnoDB: Highest supported file format is Barracuda.
2023-04-06 15:57:04 16022 [Note] InnoDB: 128 rollback segment(s) are active.
2023-04-06 15:57:04 16022 [Note] InnoDB: Waiting for purge to start
2023-04-06 15:57:04 16022 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.41-84.1 started; log sequence number 1632006
2023-04-06 15:57:04 16022 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2023-04-06 15:57:04 16022 [Note] WSREP: Recovered position: 00000000-0000-0000-0000-000000000000:-1
2023-04-06 15:57:04 16022 [Note] Binlog end
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘partition’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_CHANGED_PAGES’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_DATAFILES’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_TABLESPACES’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN_COLS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_FIELDS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_COLUMNS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_INDEXES’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_TABLESTATS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_SYS_TABLES’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_FT_INDEX_TABLE’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_FT_INDEX_CACHE’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_FT_CONFIG’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_FT_BEING_DELETED’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_FT_DELETED’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_FT_DEFAULT_STOPWORD’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_METRICS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_BUFFER_POOL_STATS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE_LRU’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX_RESET’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_CMPMEM_RESET’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_CMPMEM’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_CMP_RESET’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_CMP’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_LOCK_WAITS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_LOCKS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘INNODB_TRX’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘XTRADB_ZIP_DICT_COLS’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘XTRADB_ZIP_DICT’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘XTRADB_RSEG’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘XTRADB_INTERNAL_HASH_TABLES’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘XTRADB_READ_VIEW’
2023-04-06 15:57:04 16022 [Note] Shutting down plugin ‘InnoDB’
2023-04-06 15:57:04 16022 [Note] InnoDB: FTS optimize thread exiting.
2023-04-06 15:57:04 16022 [Note] InnoDB: Starting shutdown…
2023-04-06 15:57:06 16022 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2023-04-06 15:57:07 16022 [Note] InnoDB: Shutdown completed; log sequence number 1632184
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘BLACKHOLE’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘ARCHIVE’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘PERFORMANCE_SCHEMA’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘MEMORY’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘CSV’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘MyISAM’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘MRG_MYISAM’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘sha256_password’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘mysql_old_password’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘mysql_native_password’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘wsrep’
2023-04-06 15:57:07 16022 [Note] Shutting down plugin ‘binlog’
2023-04-06 15:57:07 16022 [Note] /usr/sbin/mysqld: Shutdown complete

2023-04-06 15:57:08 16206 [Note] WSREP: Setting wsrep_ready to false
2023-04-06 15:57:08 16206 [Note] WSREP: Read nil XID from storage engines, skipping position init
2023-04-06 15:57:08 16206 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’
2023-04-06 15:57:08 16206 [Note] WSREP: wsrep_load(): Galera 3.28(rf216443) by Codership Oy info@codership.com loaded successfully.
2023-04-06 15:57:08 16206 [Note] WSREP: CRC-32C: using hardware acceleration.
2023-04-06 15:57:08 16206 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2023-04-06 15:57:08 16206 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.151.158.54; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT10S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.keepalive_period = PT3S; evs.max_install_timeouts = 3; evs.send_window = 1024; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 512; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 2048M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 64; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; g
2023-04-06 15:57:08 16206 [Note] WSREP: GCache history reset: 7c3a28db-d05a-11eb-bbd4-b6d142cfd387:0 → 00000000-0000-0000-0000-000000000000:-1
2023-04-06 15:57:08 16206 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2023-04-06 15:57:08 16206 [Note] WSREP: wsrep_sst_grab()
2023-04-06 15:57:08 16206 [Note] WSREP: Start replication
2023-04-06 15:57:08 16206 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2023-04-06 15:57:08 16206 [Note] WSREP: protonet asio version 0
2023-04-06 15:57:08 16206 [Note] WSREP: Using CRC-32C for message checksums.
2023-04-06 15:57:08 16206 [Note] WSREP: backend: asio
2023-04-06 15:57:08 16206 [Note] WSREP: gcomm thread scheduling priority set to other:0
2023-04-06 15:57:08 16206 [Note] WSREP: restore pc from disk successfully
2023-04-06 15:57:08 16206 [Note] WSREP: GMCast version 0
2023-04-06 15:57:08 16206 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
2023-04-06 15:57:08 16206 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
2023-04-06 15:57:08 16206 [Note] WSREP: EVS version 0
2023-04-06 15:57:08 16206 [Note] WSREP: gcomm: connecting to group ‘patientportal_cluster’, peer ‘10.151.158.26:,10.151.158.53:’
2023-04-06 15:57:08 16206 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) connection established to aabd5bb7 tcp://10.151.158.26:4567
2023-04-06 15:57:08 16206 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers:
2023-04-06 15:57:08 16206 [Note] WSREP: declaring aabd5bb7 at tcp://10.151.158.26:4567 stable
2023-04-06 15:57:08 16206 [Note] WSREP: Node aabd5bb7 state prim
2023-04-06 15:57:08 16206 [Note] WSREP: view(view_id(PRIM,aabd5bb7,61) memb {
aabd5bb7,0
ad6576ad,0
} joined {
} left {
} partitioned {
})
2023-04-06 15:57:08 16206 [Note] WSREP: save pc into disk
2023-04-06 15:57:08 16206 [Note] WSREP: discarding pending addr without UUID: tcp://10.151.158.53:4567
2023-04-06 15:57:08 16206 [Note] WSREP: clear restored view
2023-04-06 15:57:09 16206 [Note] WSREP: gcomm: connected
2023-04-06 15:57:09 16206 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2023-04-06 15:57:09 16206 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0)
2023-04-06 15:57:09 16206 [Note] WSREP: Opened channel ‘patientportal_cluster’
2023-04-06 15:57:09 16206 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2023-04-06 15:57:09 16206 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2023-04-06 15:57:09 16206 [Note] WSREP: Waiting for SST to complete.
2023-04-06 15:57:09 16206 [Note] WSREP: STATE EXCHANGE: sent state msg: ea8bf72a-d482-11ed-ad4e-cf48883648f7
2023-04-06 15:57:09 16206 [Note] WSREP: STATE EXCHANGE: got state msg: ea8bf72a-d482-11ed-ad4e-cf48883648f7 from 0 (ppdb01)
2023-04-06 15:57:09 16206 [Note] WSREP: STATE EXCHANGE: got state msg: ea8bf72a-d482-11ed-ad4e-cf48883648f7 from 1 (ppdb03)
2023-04-06 15:57:09 16206 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 30,
members = 1/2 (joined/total),
act_id = 12157679,
last_appl. = -1,
protocols = 0/9/3 (gcs/repl/appl),
group UUID = 7c3a28db-d05a-11eb-bbd4-b6d142cfd387
2023-04-06 15:57:09 16206 [Note] WSREP: Flow-control interval: [91, 91]
2023-04-06 15:57:09 16206 [Note] WSREP: Trying to continue unpaused monitor
2023-04-06 15:57:09 16206 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 12157679)
2023-04-06 15:57:09 16206 [Note] WSREP: State transfer required:
Group state: 7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157679
Local state: 00000000-0000-0000-0000-000000000000:-1
2023-04-06 15:57:09 16206 [Note] WSREP: New cluster view: global state: 7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157679, view# 31: Primary, number of nodes: 2, my index: 1, protocol version 3
2023-04-06 15:57:09 16206 [Note] WSREP: Setting wsrep_ready to true
2023-04-06 15:57:09 16206 [Warning] WSREP: Gap in state sequence. Need state transfer.
2023-04-06 15:57:09 16206 [Note] WSREP: Setting wsrep_ready to false
2023-04-06 15:57:09 16206 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.151.158.54’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16206’ --mysqld-version ‘5.6.41-84.1-56’ ‘’ ’
WSREP_SST: [INFO] Streaming with xbstream (2023-04-06 15:57:09)
WSREP_SST: [INFO] Using socat as streamer (2023-04-06 15:57:09)
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (2023-04-06 15:57:09)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr,retry=30 stdio | pigz -dc | xbstream -x; RC=( ${PIPESTATUS[@]} ) (2023-04-06 15:57:09)
2023-04-06 15:57:09 16206 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.151.158.54:4444/xtrabackup_sst//1
2023-04-06 15:57:09 16206 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 → 2) (Increment: 1 → 2)
2023-04-06 15:57:09 16206 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-04-06 15:57:09 16206 [Note] WSREP: REPL Protocols: 9 (4, 2)
2023-04-06 15:57:09 16206 [Note] WSREP: Assign initial position for certification: 12157679, protocol version: 4
2023-04-06 15:57:09 16206 [Note] WSREP: Service thread queue flushed.
2023-04-06 15:57:09 16206 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (7c3a28db-d05a-11eb-bbd4-b6d142cfd387): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():535. IST will be unavailable.
2023-04-06 15:57:09 16206 [Note] WSREP: Member 1.0 (ppdb03) requested state transfer from ‘any’. Selected 0.0 (ppdb01)(SYNCED) as donor.
2023-04-06 15:57:09 16206 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 12157679)
2023-04-06 15:57:09 16206 [Note] WSREP: Requesting state transfer: success, donor: 0
2023-04-06 15:57:09 16206 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 → 7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157679
WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /var/lib/mysql//.sst from previous state transfer. Removing (2023-04-06 15:57:10)
WSREP_SST: [INFO] Proceeding with SST (2023-04-06 15:57:10)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr,retry=30 stdio | pigz -dc | xbstream -x; RC=( ${PIPESTATUS[@]} ) (2023-04-06 15:57:10)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (2023-04-06 15:57:10)
removed ‘/var/lib/mysql/ib_logfile0’
removed ‘/var/lib/mysql/performance_schema/table_io_waits_summary_by_table.frm’
removed ‘/var/lib/mysql/performance_schema/table_lock_waits_summary_by_table.frm’
removed ‘/var/lib/mysql/performance_schema/events_stages_summary_global_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/socket_summary_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_summary_by_thread_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/threads.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_summary_by_digest.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_summary_by_account_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/db.opt’
removed ‘/var/lib/mysql/performance_schema/events_stages_history.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_summary_global_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_current.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_summary_by_account_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/setup_consumers.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_summary_by_thread_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/file_summary_by_instance.frm’
removed ‘/var/lib/mysql/performance_schema/setup_actors.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_summary_by_host_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/host_cache.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_summary_by_user_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_summary_by_instance.frm’
removed ‘/var/lib/mysql/performance_schema/objects_summary_global_by_type.frm’
removed ‘/var/lib/mysql/performance_schema/file_instances.frm’
removed ‘/var/lib/mysql/performance_schema/users.frm’
removed ‘/var/lib/mysql/performance_schema/cond_instances.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_history.frm’
removed ‘/var/lib/mysql/performance_schema/events_stages_summary_by_thread_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/session_account_connect_attrs.frm’
removed ‘/var/lib/mysql/performance_schema/events_stages_summary_by_host_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/table_io_waits_summary_by_index_usage.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_summary_by_user_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_current.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_summary_by_host_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/setup_objects.frm’
removed ‘/var/lib/mysql/performance_schema/events_stages_summary_by_account_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/setup_instruments.frm’
removed ‘/var/lib/mysql/performance_schema/file_summary_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_history_long.frm’
removed ‘/var/lib/mysql/performance_schema/events_stages_summary_by_user_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/rwlock_instances.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_history_long.frm’
removed ‘/var/lib/mysql/performance_schema/hosts.frm’
removed ‘/var/lib/mysql/performance_schema/socket_summary_by_instance.frm’
removed ‘/var/lib/mysql/performance_schema/accounts.frm’
removed ‘/var/lib/mysql/performance_schema/performance_timers.frm’
removed ‘/var/lib/mysql/performance_schema/socket_instances.frm’
removed ‘/var/lib/mysql/performance_schema/events_statements_summary_global_by_event_name.frm’
removed ‘/var/lib/mysql/performance_schema/events_stages_history_long.frm’
removed ‘/var/lib/mysql/performance_schema/mutex_instances.frm’
removed ‘/var/lib/mysql/performance_schema/events_waits_history.frm’
removed ‘/var/lib/mysql/performance_schema/session_connect_attrs.frm’
removed ‘/var/lib/mysql/performance_schema/events_stages_current.frm’
removed ‘/var/lib/mysql/performance_schema/setup_timers.frm’
removed directory: ‘/var/lib/mysql/performance_schema’
removed directory: ‘/var/lib/mysql/test’
removed ‘/var/lib/mysql/ibdata1’
removed ‘/var/lib/mysql/mysql/servers.MYI’
removed ‘/var/lib/mysql/mysql/time_zone_leap_second.MYI’
removed ‘/var/lib/mysql/mysql/slave_relay_log_info.frm’
removed ‘/var/lib/mysql/mysql/slave_worker_info.frm’
removed ‘/var/lib/mysql/mysql/proc.MYI’
removed ‘/var/lib/mysql/mysql/proxies_priv.MYI’
removed ‘/var/lib/mysql/mysql/servers.MYD’
removed ‘/var/lib/mysql/mysql/help_keyword.MYD’
removed ‘/var/lib/mysql/mysql/help_relation.frm’
removed ‘/var/lib/mysql/mysql/help_keyword.frm’
removed ‘/var/lib/mysql/mysql/db.frm’
removed ‘/var/lib/mysql/mysql/slave_master_info.frm’
removed ‘/var/lib/mysql/mysql/help_category.MYI’
removed ‘/var/lib/mysql/mysql/proc.MYD’
removed ‘/var/lib/mysql/mysql/slow_log.CSV’
removed ‘/var/lib/mysql/mysql/general_log.CSV’
removed ‘/var/lib/mysql/mysql/columns_priv.frm’
removed ‘/var/lib/mysql/mysql/time_zone_name.MYD’
removed ‘/var/lib/mysql/mysql/columns_priv.MYD’
removed ‘/var/lib/mysql/mysql/func.frm’
removed ‘/var/lib/mysql/mysql/ndb_binlog_index.MYI’
removed ‘/var/lib/mysql/mysql/slow_log.CSM’
removed ‘/var/lib/mysql/mysql/event.MYI’
removed ‘/var/lib/mysql/mysql/tables_priv.MYI’
removed ‘/var/lib/mysql/mysql/proxies_priv.MYD’
removed ‘/var/lib/mysql/mysql/columns_priv.MYI’
removed ‘/var/lib/mysql/mysql/func.MYD’
removed ‘/var/lib/mysql/mysql/help_topic.MYI’
removed ‘/var/lib/mysql/mysql/db.MYI’
removed ‘/var/lib/mysql/mysql/innodb_index_stats.ibd’
removed ‘/var/lib/mysql/mysql/slow_log.frm’
removed ‘/var/lib/mysql/mysql/time_zone_leap_second.frm’
removed ‘/var/lib/mysql/mysql/time_zone_transition_type.frm’
removed ‘/var/lib/mysql/mysql/help_topic.frm’
removed ‘/var/lib/mysql/mysql/innodb_table_stats.ibd’
removed ‘/var/lib/mysql/mysql/proxies_priv.frm’
removed ‘/var/lib/mysql/mysql/time_zone_transition_type.MYI’
removed ‘/var/lib/mysql/mysql/user.MYD’
removed ‘/var/lib/mysql/mysql/slave_relay_log_info.ibd’
removed ‘/var/lib/mysql/mysql/help_category.frm’
removed ‘/var/lib/mysql/mysql/innodb_index_stats.frm’
removed ‘/var/lib/mysql/mysql/plugin.MYI’
removed ‘/var/lib/mysql/mysql/tables_priv.MYD’
removed ‘/var/lib/mysql/mysql/ndb_binlog_index.MYD’
removed ‘/var/lib/mysql/mysql/help_relation.MYD’
removed ‘/var/lib/mysql/mysql/help_relation.MYI’
removed ‘/var/lib/mysql/mysql/plugin.MYD’
removed ‘/var/lib/mysql/mysql/time_zone_transition.MYD’
removed ‘/var/lib/mysql/mysql/help_topic.MYD’
removed ‘/var/lib/mysql/mysql/db.MYD’
removed ‘/var/lib/mysql/mysql/slave_master_info.ibd’
removed ‘/var/lib/mysql/mysql/time_zone_leap_second.MYD’
removed ‘/var/lib/mysql/mysql/procs_priv.frm’
removed ‘/var/lib/mysql/mysql/time_zone_name.frm’
removed ‘/var/lib/mysql/mysql/time_zone.MYI’
removed ‘/var/lib/mysql/mysql/time_zone.frm’
removed ‘/var/lib/mysql/mysql/plugin.frm’
removed ‘/var/lib/mysql/mysql/user.MYI’
removed ‘/var/lib/mysql/mysql/general_log.frm’
removed ‘/var/lib/mysql/mysql/proc.frm’
removed ‘/var/lib/mysql/mysql/tables_priv.frm’
removed ‘/var/lib/mysql/mysql/innodb_table_stats.frm’
removed ‘/var/lib/mysql/mysql/procs_priv.MYI’
removed ‘/var/lib/mysql/mysql/time_zone_name.MYI’
removed ‘/var/lib/mysql/mysql/help_keyword.MYI’
removed ‘/var/lib/mysql/mysql/func.MYI’
removed ‘/var/lib/mysql/mysql/user.frm’
removed ‘/var/lib/mysql/mysql/time_zone_transition.frm’
removed ‘/var/lib/mysql/mysql/time_zone_transition_type.MYD’
removed ‘/var/lib/mysql/mysql/event.MYD’
removed ‘/var/lib/mysql/mysql/help_category.MYD’
removed ‘/var/lib/mysql/mysql/event.frm’
removed ‘/var/lib/mysql/mysql/time_zone_transition.MYI’
removed ‘/var/lib/mysql/mysql/ndb_binlog_index.frm’
removed ‘/var/lib/mysql/mysql/time_zone.MYD’
removed ‘/var/lib/mysql/mysql/servers.frm’
removed ‘/var/lib/mysql/mysql/general_log.CSM’
removed ‘/var/lib/mysql/mysql/slave_worker_info.ibd’
removed ‘/var/lib/mysql/mysql/procs_priv.MYD’
removed directory: ‘/var/lib/mysql/mysql’
removed ‘/var/lib/mysql/ib_logfile1’
WSREP_SST: [INFO] Waiting for SST streaming to complete! (2023-04-06 15:57:10)
2023-04-06 15:57:11 16206 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (2023-04-06 15:57:21)
WSREP_SST: [ERROR] Cleanup after exit with status:2 (2023-04-06 15:57:21)
2023-04-06 15:57:21 16206 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.151.158.54’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘16206’ --mysqld-version ‘5.6.41-84.1-56’ ‘’ : 2 (No such file or directory)
2023-04-06 15:57:21 16206 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2023-04-06 15:57:21 16206 [ERROR] WSREP: SST script aborted with error 2 (No such file or directory)
2023-04-06 15:57:21 16206 [ERROR] WSREP: SST failed: 2 (No such file or directory)
2023-04-06 15:57:21 16206 [ERROR] Aborting

2023-04-06 15:57:21 16206 [Note] WSREP: Signalling cancellation of the SST request.
2023-04-06 15:57:21 16206 [Note] WSREP: SST request was cancelled
2023-04-06 15:57:21 16206 [Note] WSREP: Closing send monitor…
2023-04-06 15:57:21 16206 [Note] WSREP: Closed send monitor.
2023-04-06 15:57:21 16206 [Note] WSREP: gcomm: terminating thread
2023-04-06 15:57:21 16206 [Note] WSREP: gcomm: joining thread
2023-04-06 15:57:21 16206 [Note] WSREP: gcomm: closing backend
2023-04-06 15:57:23 16206 [Note] WSREP: Service disconnected.
2023-04-06 15:57:23 16206 [Note] WSREP: Waiting to close threads…
2023-04-06 15:57:23 16206 [Note] WSREP: rollbacker thread exiting
2023-04-06 15:57:24 16206 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) connection to peer aabd5bb7 with addr tcp://10.151.158.26:4567 timed out, no messages seen in PT3S
2023-04-06 15:57:24 16206 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://10.151.158.26:4567
2023-04-06 15:57:26 16206 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) reconnecting to aabd5bb7 (tcp://10.151.158.26:4567), attempt 0
2023-04-06 15:57:28 16206 [Note] WSREP: Some threads may fail to exit.
2023-04-06 15:57:28 16206 [Note] Binlog end
2023-04-06 15:57:28 16206 [Note] /usr/sbin/mysqld: Shutdown complete

Fatal signal 6 while backtracing

What is the problem there?

2

First node log (bootstraped node)

Blockquote
2023-04-06 16:03:16 5104 [Note] WSREP: (aabd5bb7, ‘tcp://0.0.0.0:4567’) connection established to ad6576ad tcp://10.151.158.54:4567
2023-04-06 16:03:16 5104 [Note] WSREP: (aabd5bb7, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers:
2023-04-06 16:03:17 5104 [Note] WSREP: declaring ad6576ad at tcp://10.151.158.54:4567 stable
2023-04-06 16:03:17 5104 [Note] WSREP: Node aabd5bb7 state prim
2023-04-06 16:03:17 5104 [Note] WSREP: view(view_id(PRIM,aabd5bb7,63) memb {
aabd5bb7,0
ad6576ad,0
} joined {
} left {
} partitioned {
})
2023-04-06 16:03:17 5104 [Note] WSREP: save pc into disk
2023-04-06 16:03:17 5104 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2023-04-06 16:03:17 5104 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7ab319f-d483-11ed-a0b7-fe082c73ffe5
2023-04-06 16:03:17 5104 [Note] WSREP: STATE EXCHANGE: sent state msg: c7ab319f-d483-11ed-a0b7-fe082c73ffe5
2023-04-06 16:03:17 5104 [Note] WSREP: STATE EXCHANGE: got state msg: c7ab319f-d483-11ed-a0b7-fe082c73ffe5 from 0 (ppdb01)
2023-04-06 16:03:17 5104 [Note] WSREP: STATE EXCHANGE: got state msg: c7ab319f-d483-11ed-a0b7-fe082c73ffe5 from 1 (ppdb03)
2023-04-06 16:03:17 5104 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 32,
members = 1/2 (joined/total),
act_id = 12157759,
last_appl. = 12157652,
protocols = 0/9/3 (gcs/repl/appl),
group UUID = 7c3a28db-d05a-11eb-bbd4-b6d142cfd387
2023-04-06 16:03:17 5104 [Note] WSREP: Flow-control interval: [91, 91]
2023-04-06 16:03:17 5104 [Note] WSREP: Trying to continue unpaused monitor
2023-04-06 16:03:17 5104 [Note] WSREP: New cluster view: global state: 7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157759, view# 33: Primary, number of nodes: 2, my index: 0, protocol version 3
2023-04-06 16:03:17 5104 [Note] WSREP: Setting wsrep_ready to true
2023-04-06 16:03:17 5104 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 → 1) (Increment: 1 → 2)
2023-04-06 16:03:17 5104 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-04-06 16:03:17 5104 [Note] WSREP: REPL Protocols: 9 (4, 2)
2023-04-06 16:03:17 5104 [Note] WSREP: Assign initial position for certification: 12157759, protocol version: 4
2023-04-06 16:03:17 5104 [Note] WSREP: Service thread queue flushed.
2023-04-06 16:03:17 5104 [Note] WSREP: Member 1.0 (ppdb03) requested state transfer from ‘any’. Selected 0.0 (ppdb01)(SYNCED) as donor.
2023-04-06 16:03:17 5104 [Note] WSREP: Shifting SYNCED → DONOR/DESYNCED (TO: 12157759)
2023-04-06 16:03:17 5104 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-04-06 16:03:18 5104 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘10.151.158.54:4444/xtrabackup_sst//1’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘5.6.41-84.1-56’ ‘’ --gtid ‘7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157759’ ’
2023-04-06 16:03:18 5104 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] Streaming with xbstream (2023-04-06 16:03:18)
WSREP_SST: [INFO] Using socat as streamer (2023-04-06 16:03:18)
WSREP_SST: [INFO] Streaming SST meta-info file before SST (2023-04-06 16:03:18)
WSREP_SST: [INFO] Evaluating xbstream -c ${FILE_TO_STREAM} | pigz | socat -u stdio TCP:10.151.158.54:4444,retry=30 ; RC=( ${PIPESTATUS[@]} ) (2023-04-06 16:03:18)
WSREP_SST: [INFO] Sleeping before data transfer for SST (2023-04-06 16:03:18)
2023-04-06 16:03:20 5104 [Note] WSREP: (aabd5bb7, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
WSREP_SST: [INFO] Streaming the backup to joiner at 10.151.158.54 4444 (2023-04-06 16:03:28)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/my.cnf --defaults-group=mysqld --no-version-check $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log | pigz | socat -u stdio TCP:10.151.158.54:4444,retry=30; RC=( ${PIPESTATUS[@]} ) (2023-04-06 16:03:28)
2023-04-06 16:03:32 5104 [Warning] Aborted connection 11930162 to db: ‘unconnected’ user: ‘root’ host: ‘localhost’ (Got an error reading communication packets)
WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log (2023-04-06 16:03:32)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (2023-04-06 16:03:32)
WSREP_SST: [INFO] Cleaning up temporary directories (2023-04-06 16:03:32)
2023-04-06 16:03:32 5104 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘10.151.158.54:4444/xtrabackup_sst//1’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘5.6.41-84.1-56’ ‘’ --gtid ‘7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157759’
2023-04-06 16:03:32 5104 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘10.151.158.54:4444/xtrabackup_sst//1’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘5.6.41-84.1-56’ ‘’ --gtid ‘7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157759’ : 22 (Invalid argument)
2023-04-06 16:03:33 5104 [Note] WSREP: forgetting ad6576ad (tcp://10.151.158.54:4567)
2023-04-06 16:03:33 5104 [Note] WSREP: Node aabd5bb7 state prim
2023-04-06 16:03:33 5104 [Note] WSREP: view(view_id(PRIM,aabd5bb7,64) memb {
aabd5bb7,0
} joined {
} left {
} partitioned {
ad6576ad,0
})
2023-04-06 16:03:33 5104 [Note] WSREP: save pc into disk
2023-04-06 16:03:33 5104 [Note] WSREP: forgetting ad6576ad (tcp://10.151.158.54:4567)
2023-04-06 16:03:33 5104 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2023-04-06 16:03:33 5104 [Note] WSREP: STATE_EXCHANGE: sent state UUID: d12d42fb-d483-11ed-9af3-270862b732fa
2023-04-06 16:03:33 5104 [Note] WSREP: STATE EXCHANGE: sent state msg: d12d42fb-d483-11ed-9af3-270862b732fa
2023-04-06 16:03:33 5104 [Note] WSREP: STATE EXCHANGE: got state msg: d12d42fb-d483-11ed-9af3-270862b732fa from 0 (ppdb01)
2023-04-06 16:03:33 5104 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 33,
members = 1/1 (joined/total),
act_id = 12157759,
last_appl. = 12157652,
protocols = 0/9/3 (gcs/repl/appl),
group UUID = 7c3a28db-d05a-11eb-bbd4-b6d142cfd387
2023-04-06 16:03:33 5104 [Note] WSREP: Flow-control interval: [64, 64]
2023-04-06 16:03:33 5104 [Note] WSREP: Trying to continue unpaused monitor
2023-04-06 16:03:33 5104 [Warning] WSREP: Could not find peer: ad6576ad-d482-11ed-83ef-ee205015a892
2023-04-06 16:03:33 5104 [Warning] WSREP: 0.0 (ppdb01): State transfer to -1.-1 (left the group) failed: -22 (Invalid argument)
2023-04-06 16:03:33 5104 [Note] WSREP: Shifting DONOR/DESYNCED → JOINED (TO: 12157759)
2023-04-06 16:03:33 5104 [Note] WSREP: Member 0.0 (ppdb01) synced with group.
2023-04-06 16:03:33 5104 [Note] WSREP: Shifting JOINED → SYNCED (TO: 12157759)
2023-04-06 16:03:33 5104 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘10.151.158.54:4444/xtrabackup_sst//1’ --socket ‘/var/lib/mysql/mysql.sock’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘5.6.41-84.1-56’ ‘’ --gtid ‘7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157759’
2023-04-06 16:03:33 5104 [Warning] WSREP: Protocol violation. JOIN message sender 0.0 (ppdb01) is not in state transfer (SYNCED). Message ignored.
2023-04-06 16:03:33 5104 [Note] WSREP: New cluster view: global state: 7c3a28db-d05a-11eb-bbd4-b6d142cfd387:12157759, view# 34: Primary, number of nodes: 1, my index: 0, protocol version 3
2023-04-06 16:03:33 5104 [Note] WSREP: Setting wsrep_ready to true
2023-04-06 16:03:33 5104 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 → 1) (Increment: 2 → 1)
2023-04-06 16:03:33 5104 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-04-06 16:03:33 5104 [Note] WSREP: REPL Protocols: 9 (4, 2)
2023-04-06 16:03:33 5104 [Note] WSREP: Assign initial position for certification: 12157759, protocol version: 4
2023-04-06 16:03:33 5104 [Note] WSREP: Service thread queue flushed.
2023-04-06 16:03:33 5104 [Note] WSREP: Synchronized with group, ready for connections
2023-04-06 16:03:33 5104 [Note] WSREP: Setting wsrep_ready to true
2023-04-06 16:03:33 5104 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-04-06 16:03:36 5104 [Note] WSREP: (aabd5bb7, ‘tcp://0.0.0.0:4567’) connection established to ad6576ad tcp://10.151.158.54:4567
2023-04-06 16:03:36 5104 [Warning] WSREP: discarding established (time wait) ad6576ad (tcp://10.151.158.54:4567)
2023-04-06 16:03:38 5104 [Note] WSREP: cleaning up ad6576ad (tcp://10.151.158.54:4567)

3

Hello @elkhedewy,
PXC 5.6 and MySQL 5.6 have been dead software for many years. PXC 5.7 and MySQL 5.7 will be dead in October 2023. Many bugs and features have been fixed/added between 5.6 and 8.0 (the current standard). I recommend you use PXC 8.0 before wasting time troubleshooting dead 5.6 software.

4

Ok i will upgrade to PXC 8.0 but first i need to solve all old issues

5

Old issues may be resolved by default in PXC 8. Unfortunately, the only solution/recommendation we can provide is first upgrade to PXC 8 and verify the problem remains.

6

Okay, is there any documentation for upgrading from 5.6 to PXC 8.0
NOTE:
Its production servers and already running

.
i will search for it, and i appreciate if you can help.

Regards,

7

What does this error means?

WSREP_SST: [INFO] Waiting for SST streaming to complete! (2023-04-07 16:33:53)
2023-04-07 16:33:54 23075 [Note] WSREP: (ad6576ad, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (2023-04-07 16:34:05)
WSREP_SST: [ERROR] Cleanup after exit with status:2 (2023-04-07 16:34:05)
2023-04-07 16:34:05 23075 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.151.158.54’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘23075’ --mysqld-version ‘5.6.41-84.1-56’ ‘’ : 2 (No such file or directory)
2023-04-07 16:34:05 23075 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2023-04-07 16:34:05 23075 [ERROR] WSREP: SST script aborted with error 2 (No such file or directory)
2023-04-07 16:34:05 23075 [ERROR] WSREP: SST failed: 2 (No such file or directory)
2023-04-07 16:34:05 23075 [ERROR] Aborting

8

It means the SST process failed which can happen for any number of reasons. You need to read through the donor logs to understand why.

9

There was no clear log for the error. But i changed the owner of /var/lib/mysql/ to mysql:mysql recursively and this solved the problem.

chown -R mysql:mysql /var/lib/mysql

This solved my problem and now all cluster is working fine !!