Hello, I have a three node Percona Xtradb Cluster 5.7. I resized on of the machine from 16 cores to 24 cores and after that MySQL get shutsdown randomly. Following are the logs when DB shutsdown. Please help me to identify the issue.
00:59:44 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://jira.percona.com/projects/PXC/issues
key_buffer_size=8388608
read_buffer_size=1048576
max_used_connections=119
max_threads=401
thread_count=29
connection_count=12
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1242919 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Build ID: fa6cdbf29396c07be2ad9ddbfeb78c94ff55972f
Server Version: 5.7.41-44-57-log Percona XtraDB Cluster (GPL), Release rel44, Revision b46a1c3, WSREP version 31.65, wsrep_31.65
Thread pointer: 0x7fa6c0090a50
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa8c567ad30 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf8417b]
/usr/sbin/mysqld(handle_fatal_signal+0x505)[0xd92b45]
/lib64/libpthread.so.0(+0xf630)[0x7faf913b4630]
/usr/sbin/mysqld[0xc9f7a8]
/usr/sbin/mysqld[0xc9f80d]
/usr/sbin/mysqld[0xcad73c]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x29bc)[0xcb05fc]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x172)[0xcb0d22]
/usr/sbin/mysqld(handle_connection+0x2c8)[0xd765f8]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf9d3a4]
/lib64/libpthread.so.0(+0x7ea5)[0x7faf913acea5]
/lib64/libc.so.6(clone+0x6d)[0x7faf8f78fb0d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fa6c06f05f0): is an invalid pointer
Connection ID (thread ID): 1177927
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Log of wsrep recovery (--wsrep-recover):
2023-05-30T01:15:45.092498Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-05-30T01:15:45.092570Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2023-05-30T01:15:45.094610Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.41-44-57-log) starting as process 8501 ...
2023-05-30T01:15:45.104373Z 0 [Warning] InnoDB: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead; Please refer to http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html
2023-05-30T01:15:45.104465Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-05-30T01:15:45.104485Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-05-30T01:15:45.104496Z 0 [Note] InnoDB: Uses event mutexes
2023-05-30T01:15:45.104506Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-05-30T01:15:45.104517Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-05-30T01:15:45.104527Z 0 [Note] InnoDB: Using Linux native AIO
2023-05-30T01:15:45.104949Z 0 [Note] InnoDB: Number of pools: 1
2023-05-30T01:15:45.105111Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-05-30T01:15:45.110561Z 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M
2023-05-30T01:15:45.709000Z 0 [Note] InnoDB: Completed initialization of buffer pool
2023-05-30T01:15:45.859063Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-05-30T01:15:45.878893Z 0 [Note] InnoDB: Recovering partial pages from the parallel doublewrite buffer at /data2/mysql/xb_doublewrite
2023-05-30T01:15:45.901276Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2023-05-30T01:15:45.990647Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 19202117284108
2023-05-30T01:15:46.115493Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19202122526720
2023-05-30T01:15:46.149920Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19202127769600
2023-05-30T01:15:46.191543Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19202133012480
2023-05-30T01:15
2023-05-30T01:15:49.432234Z 0 [Note] InnoDB: Apply batch completed
2023-05-30T01:15:49.432318Z 0 [Note] InnoDB: Last MySQL binlog file position 0 285670849, file name bin.029879
2023-05-30T01:15:53.964791Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-05-30T01:15:53.964841Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-05-30T01:15:53.965128Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-05-30T01:15:54.002247Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-05-30T01:15:54.006746Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2023-05-30T01:15:54.006776Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2023-05-30T01:15:54.006912Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 8148ms. The settings might not be optimal. (flushed=0, during the time.)
2023-05-30T01:15:54.008620Z 0 [Note] InnoDB: Waiting for purge to start
2023-05-30T01:15:54.058727Z 0 [Note] InnoDB: Percona XtraDB ([http://www.percona.com](http://www.percona.com/)) 5.7.41-44 started; log sequence number 19202255043709
2023-05-30T01:15:54.058757Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2023-05-30T01:15:54.059508Z 0 [Note] Plugin 'FEDERATED' is disabled.
2023-05-30T01:15:54.065888Z 0 [Note] Recovering after a crash using /logs/mysql/bin
2023-05-30T01:15:54.065932Z 0 [Note] WSREP: Before binlog recovery (wsrep position e2c227c5-3a82-11ec-97ca-2f03f14986b2:1719916734)
2023-05-30T01:15:54.386643Z 0 [Note] Starting crash recovery...
2023-05-30T01:15:54.386723Z 0 [Note] Crash recovery finished.
2023-05-30T01:15:54.386752Z 0 [Note] WSREP: After binlog recovery (wsrep position e2c227c5-3a82-11ec-97ca-2f03f14986b2:1719916734)
2023-05-30T01:15:54.388337Z 0 [Note] WSREP: Recovered position: e2c227c5-3a82-11ec-97ca-2f03f14986b2:1719916734
2023-05-30T01:15:54.388387Z 0 [Note] Binlog end
2023-05-30T01:15:54.388536Z 0 [Note] Shutting down plugin 'ngram'
2023-05-30T01:15:54.388554Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2023-05-30T01:15:54.388568Z 0 [Note] Shutting down plugin 'ARCHIVE'
2023-05-30T01:15:54.388580Z 0 [Note] Shutting down plugin 'partition'
2023-05-30T01:15:54.388568Z 0 [Note] Shutting down plugin 'ARCHIVE'
2023-05-30T01:15:54.388580Z 0 [Note] Shutting down plugin 'partition'
2023-05-30T01:15:54.388592Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2023-05-30T01:15:54.388604Z 0 [Note] Shutting down plugin 'MyISAM'
2023-05-30T01:15:54.388624Z 0 [Note] Shutting down plugin 'CSV'
2023-05-30T01:15:54.388637Z 0 [Note] Shutting down plugin 'MEMORY'
2023-05-30T01:15:54.388651Z 0 [Note] Shutting down plugin 'INNODB_TABLESPACES_SCRUBBING'
2023-05-30T01:15:54.388671Z 0 [Note] Shutting down plugin 'INNODB_TABLESPACES_ENCRYPTION'
2023-05-30T01:15:54.388678Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2023-05-30T01:15:54.388685Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2023-05-30T01:15:54.388692Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2023-05-30T01:15:54.388699Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2023-05-30T01:15:54.388706Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2023-05-30T01:15:54.388713Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2023-05-30T01:15:54.388721Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2023-05-30T01:15:54.388728Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2023-05-30T01:15:54.388736Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2023-05-30T01:15:54.388744Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2023-05-30T01:15:54.388752Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2023-05-30T01:15:54.388760Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2023-05-30T01:15:54.388768Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2023-05-30T01:15:54.388776Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2023-05-30T01:15:54.388785Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2023-05-30T01:15:54.388793Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2023-05-30T01:15:54.388801Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2023-05-30T01:15:54.388810Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2023-05-30T01:15:54.388818Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2023-05-30T01:15:54.388827Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2023-05-30T01:15:54.388836Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2023-05-30T01:15:54.388845Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2023-05-30T01:15:54.388855Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2023-05-30T01:15:54.388864Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2023-05-30T01:15:54.388873Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2023-05-30T01:15:54.388882Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2023-05-30T01:15:54.388892Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2023-05-30T01:15:54.388902Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2023-05-30T01:15:54.388911Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2023-05-30T01:15:54.388921Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2023-05-30T01:15:54.388931Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2023-05-30T01:15:54.388941Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
2023-05-30T01:15:54.388952Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
2023-05-30T01:15:54.388962Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
2023-05-30T01:15:54.388972Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2023-05-30T01:15:54.388983Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2023-05-30T01:15:54.388993Z 0 [Note] Shutting down plugin 'InnoDB'
2023-05-30T01:15:54.395729Z 0 [Note] InnoDB: FTS optimize thread exiting.
2023-05-30T01:15:54.396116Z 0 [Note] InnoDB: Starting shutdown...
2023-05-30T01:15:55.196903Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2023-05-30T01:15:57.053807Z 0 [Note] InnoDB: Shutdown completed; log sequence number 19202255069949
2023-05-30T01:15:57.054075Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-05-30T01:15:57.054110Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2023-05-30T01:15:57.054192Z 0 [Note] Shutting down plugin 'sha256_password'
2023-05-30T01:15:57.054205Z 0 [Note] Shutting down plugin 'mysql_native_password'
2023-05-30T01:15:57.054217Z 0 [Note] Shutting down plugin 'wsrep'
2023-05-30T01:15:57.054440Z 0 [Note] Shutting down plugin 'binlog'
2023-05-30T01:15:57.056215Z 0 [Note] /usr/sbin/mysqld: Shutdown complete