Database crash with `mysqld got signal 11`

MySQL & MariaDB Percona Server for MySQL 5.7
,
1

I have a database that stopped working starting up. I don’t think the server crashed before the error starting happening, but it is possible the application did. Here are some log outputs, I’m not sure where to go from here to pinpoint the error.

I restart the database and it’s every time the same, the percona server shuts down immediately after starting.

Running the docker.io/percona/5.7.25 container when the error occurred, updated to the latest 5.7.34 and still the same.

# without `--innodb_force_recovery` or with `--innodb_force_recovery=1`

22:42:13 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=152
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68338 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.


Build ID: Not Available 
Server Version: 5.7.34-37 Percona Server (GPL), Release 37, Revision 7c516e9

Thread pointer: 0x7f9ae4000ba0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f9af0912dc0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3f)[0xf1264f]
mysqld(handle_fatal_signal+0x5a1)[0xd43511]
/lib64/libpthread.so.0(+0x12b20)[0x7f9b1371db20]
mysqld(_Z28trx_undo_rec_get_partial_rowPKhP12dict_index_tPP8dtuple_tmP16mem_block_info_t+0x38a)[0x11364da]
mysqld(_Z14row_purge_stepP9que_thr_t+0xb30)[0x10ccbe0]
mysqld(_Z15que_run_threadsP9que_thr_t+0xc94)[0x1077f24]
mysqld(_Z9trx_purgemmb+0x400)[0x11347e0]
mysqld(srv_purge_coordinator_thread+0xb7d)[0x11048ad]
/lib64/libpthread.so.0(+0x814a)[0x7f9b1371314a]
/lib64/libc.so.6(clone+0x43)[0x7f9b11740f23]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 0
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

# with `--innodb_force_recovery=2/3`

2021-08-09T22:45:03.617981Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-08-09T22:45:03.618751Z 0 [Note] mysqld (mysqld 5.7.34-37) starting as process 1 ...
2021-08-09T22:45:03.620168Z 0 [Warning] option 'innodb-purge-threads': unsigned value 0 adjusted to 1
2021-08-09T22:45:03.620501Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-08-09T22:45:03.620508Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-08-09T22:45:03.620509Z 0 [Note] InnoDB: Uses event mutexes
2021-08-09T22:45:03.620510Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-08-09T22:45:03.620512Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-08-09T22:45:03.620513Z 0 [Note] InnoDB: Using Linux native AIO
2021-08-09T22:45:03.620691Z 0 [Note] InnoDB: Number of pools: 1
2021-08-09T22:45:03.620853Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-08-09T22:45:03.621999Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-08-09T22:45:03.624372Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-08-09T22:45:03.625507Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-08-09T22:45:03.636294Z 0 [Note] InnoDB: Recovering partial pages from the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2021-08-09T22:45:03.646822Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2021-08-09T22:45:03.679899Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 3932160 bytes
2021-08-09T22:45:03.689990Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-08-09T22:45:03.690002Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-08-09T22:45:03.690029Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-08-09T22:45:03.720424Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-08-09T22:45:03.720790Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2021-08-09T22:45:03.720796Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2021-08-09T22:45:03.720918Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.34-37 started; log sequence number 267655523260
2021-08-09T22:45:03.720923Z 0 [Note] InnoDB: !!! innodb_force_recovery is set to 2 !!!
2021-08-09T22:45:03.720959Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-08-09T22:45:03.721070Z 0 [Note] Plugin 'FEDERATED' is disabled.
2021-08-09T22:45:03.739168Z 0 [Note] InnoDB: Buffer pool(s) load completed at 210809 22:45:03
2021-08-09T22:45:03.748208Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2021-08-09T22:45:03.748220Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2021-08-09T22:45:03.749180Z 0 [Warning] CA certificate ca.pem is self signed.
2021-08-09T22:45:03.749224Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2021-08-09T22:45:03.749533Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2021-08-09T22:45:03.749554Z 0 [Note] IPv6 is available.
2021-08-09T22:45:03.749558Z 0 [Note]   - '::' resolves to '::';
2021-08-09T22:45:03.749566Z 0 [Note] Server socket created on IP: '::'.
2021-08-09T22:45:03.749599Z 0 [ERROR] Unix socket lock file is empty /var/lib/mysql/mysql.sock.lock.
2021-08-09T22:45:03.749602Z 0 [ERROR] Unable to setup unix socket lock file.
2021-08-09T22:45:03.749603Z 0 [ERROR] Aborting

2021-08-09T22:45:03.749609Z 0 [Note] Binlog end
2021-08-09T22:45:03.749646Z 0 [Note] Shutting down plugin 'ngram'
2021-08-09T22:45:03.749650Z 0 [Note] Shutting down plugin 'partition'
2021-08-09T22:45:03.749651Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2021-08-09T22:45:03.749652Z 0 [Note] Shutting down plugin 'ARCHIVE'
2021-08-09T22:45:03.749653Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2021-08-09T22:45:03.749654Z 0 [Note] Shutting down plugin 'MyISAM'
2021-08-09T22:45:03.749660Z 0 [Note] Shutting down plugin 'INNODB_TABLESPACES_SCRUBBING'
2021-08-09T22:45:03.749662Z 0 [Note] Shutting down plugin 'INNODB_TABLESPACES_ENCRYPTION'
2021-08-09T22:45:03.749663Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2021-08-09T22:45:03.749663Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2021-08-09T22:45:03.749664Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2021-08-09T22:45:03.749665Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2021-08-09T22:45:03.749666Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2021-08-09T22:45:03.749666Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2021-08-09T22:45:03.749667Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2021-08-09T22:45:03.749667Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2021-08-09T22:45:03.749668Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2021-08-09T22:45:03.749669Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2021-08-09T22:45:03.749670Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2021-08-09T22:45:03.749670Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2021-08-09T22:45:03.749671Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2021-08-09T22:45:03.749672Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2021-08-09T22:45:03.749672Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2021-08-09T22:45:03.749673Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2021-08-09T22:45:03.749674Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2021-08-09T22:45:03.749674Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2021-08-09T22:45:03.749675Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2021-08-09T22:45:03.749676Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2021-08-09T22:45:03.749676Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2021-08-09T22:45:03.749677Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2021-08-09T22:45:03.749678Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2021-08-09T22:45:03.749678Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2021-08-09T22:45:03.749679Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2021-08-09T22:45:03.749680Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2021-08-09T22:45:03.749680Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2021-08-09T22:45:03.749681Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2021-08-09T22:45:03.749682Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2021-08-09T22:45:03.749682Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2021-08-09T22:45:03.749683Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2021-08-09T22:45:03.749684Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
2021-08-09T22:45:03.749685Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
2021-08-09T22:45:03.749685Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
2021-08-09T22:45:03.749686Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2021-08-09T22:45:03.749687Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2021-08-09T22:45:03.749688Z 0 [Note] Shutting down plugin 'InnoDB'
2021-08-09T22:45:03.749719Z 0 [Note] InnoDB: FTS optimize thread exiting.
2021-08-09T22:45:03.749742Z 0 [Note] InnoDB: Starting shutdown...
2021-08-09T22:45:03.849837Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2021-08-09T22:45:03.850258Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 210809 22:45:03
2021-08-09T22:45:03.949913Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

# with `--innodb_force_recovery=4/5/6`

2021-08-09T22:46:39.499702Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-08-09T22:46:39.500511Z 0 [Note] mysqld (mysqld 5.7.34-37) starting as process 1 ...
2021-08-09T22:46:39.501945Z 0 [Warning] option 'innodb-purge-threads': unsigned value 0 adjusted to 1
2021-08-09T22:46:39.502240Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-08-09T22:46:39.502248Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-08-09T22:46:39.502250Z 0 [Note] InnoDB: Uses event mutexes
2021-08-09T22:46:39.502252Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-08-09T22:46:39.502254Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-08-09T22:46:39.502256Z 0 [Note] InnoDB: Using Linux native AIO
2021-08-09T22:46:39.502443Z 0 [Note] InnoDB: Number of pools: 1
2021-08-09T22:46:39.502602Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-08-09T22:46:39.503772Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-08-09T22:46:39.506240Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-08-09T22:46:39.507371Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-08-09T22:46:39.523459Z 0 [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2021-08-09T22:46:39.523772Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2021-08-09T22:46:39.565257Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 3932160 bytes
2021-08-09T22:46:39.568896Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-08-09T22:46:39.568944Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-08-09T22:46:39.635175Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-08-09T22:46:39.635550Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2021-08-09T22:46:39.635555Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2021-08-09T22:46:39.635882Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.34-37 started; log sequence number 267655523298
2021-08-09T22:46:39.635888Z 0 [Note] InnoDB: !!! innodb_force_recovery is set to 4 !!!
2021-08-09T22:46:39.635965Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-08-09T22:46:39.636251Z 0 [Note] InnoDB: Buffer pool(s) load completed at 210809 22:46:39
2021-08-09T22:46:39.636864Z 0 [Note] Plugin 'FEDERATED' is disabled.
2021-08-09T22:46:39.638782Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`plugin` in the cache. Attempting to load the tablespace with space id 2
2021-08-09T22:46:39.639317Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`gtid_executed` in the cache. Attempting to load the tablespace with space id 18
2021-08-09T22:46:39.641212Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2021-08-09T22:46:39.641219Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2021-08-09T22:46:39.641679Z 0 [Warning] CA certificate ca.pem is self signed.
2021-08-09T22:46:39.641712Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2021-08-09T22:46:39.641984Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2021-08-09T22:46:39.642003Z 0 [Note] IPv6 is available.
2021-08-09T22:46:39.642007Z 0 [Note]   - '::' resolves to '::';
2021-08-09T22:46:39.642015Z 0 [Note] Server socket created on IP: '::'.
2021-08-09T22:46:39.642055Z 0 [ERROR] Unix socket lock file is empty /var/lib/mysql/mysql.sock.lock.
2021-08-09T22:46:39.642057Z 0 [ERROR] Unable to setup unix socket lock file.
2021-08-09T22:46:39.642058Z 0 [ERROR] Aborting

2021-08-09T22:46:39.642062Z 0 [Note] Binlog end
2021-08-09T22:46:39.642088Z 0 [Note] Shutting down plugin 'ngram'
2021-08-09T22:46:39.642091Z 0 [Note] Shutting down plugin 'partition'
2021-08-09T22:46:39.642092Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2021-08-09T22:46:39.642093Z 0 [Note] Shutting down plugin 'ARCHIVE'
2021-08-09T22:46:39.642094Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2021-08-09T22:46:39.642095Z 0 [Note] Shutting down plugin 'MyISAM'
2021-08-09T22:46:39.642100Z 0 [Note] Shutting down plugin 'INNODB_TABLESPACES_SCRUBBING'
2021-08-09T22:46:39.642102Z 0 [Note] Shutting down plugin 'INNODB_TABLESPACES_ENCRYPTION'
2021-08-09T22:46:39.642103Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2021-08-09T22:46:39.642103Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2021-08-09T22:46:39.642104Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2021-08-09T22:46:39.642105Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2021-08-09T22:46:39.642106Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2021-08-09T22:46:39.642106Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2021-08-09T22:46:39.642107Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2021-08-09T22:46:39.642108Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2021-08-09T22:46:39.642108Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2021-08-09T22:46:39.642109Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2021-08-09T22:46:39.642110Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2021-08-09T22:46:39.642111Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2021-08-09T22:46:39.642111Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2021-08-09T22:46:39.642112Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2021-08-09T22:46:39.642113Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2021-08-09T22:46:39.642113Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2021-08-09T22:46:39.642114Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2021-08-09T22:46:39.642115Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2021-08-09T22:46:39.642115Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2021-08-09T22:46:39.642116Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2021-08-09T22:46:39.642117Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2021-08-09T22:46:39.642117Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2021-08-09T22:46:39.642118Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2021-08-09T22:46:39.642119Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2021-08-09T22:46:39.642120Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2021-08-09T22:46:39.642120Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2021-08-09T22:46:39.642121Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2021-08-09T22:46:39.642122Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2021-08-09T22:46:39.642122Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2021-08-09T22:46:39.642123Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2021-08-09T22:46:39.642124Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2021-08-09T22:46:39.642124Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
2021-08-09T22:46:39.642125Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
2021-08-09T22:46:39.642126Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
2021-08-09T22:46:39.642126Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2021-08-09T22:46:39.642127Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2021-08-09T22:46:39.642128Z 0 [Note] Shutting down plugin 'InnoDB'
2021-08-09T22:46:39.642152Z 0 [Note] InnoDB: FTS optimize thread exiting.
2021-08-09T22:46:39.642179Z 0 [Note] InnoDB: Starting shutdown...
2021-08-09T22:46:39.742309Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2021-08-09T22:46:39.742459Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 210809 22:46:39
2021-08-09T22:46:39.842321Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
1 Like
2

Looks like the database was running a purge, yes you’ll need to be using innodb_force_recovery

Looks like the database resolved the InnoDB crashing transaction, but got blocked due to a new error:

You should check to see if you have a file already at /var/lib/mysql/mysql.sock.lock

1 Like
3

Thanks for the response. There was indeed an empty mysql.sock.lock file. I removed it, and mysql starts in recovery mode, with some errors however about not being able to find tablespace for X table.

# with --innodb_force_recovery=6 --innodb_purge_threads=0

2021-08-10T02:41:50.117675Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-08-10T02:41:50.118408Z 0 [Note] mysqld (mysqld 5.7.34-37) starting as process 1 ...
2021-08-10T02:41:50.119699Z 0 [Warning] option 'innodb-purge-threads': unsigned value 0 adjusted to 1
2021-08-10T02:41:50.119977Z 0 [Note] InnoDB: Started in read only mode
2021-08-10T02:41:50.119984Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-08-10T02:41:50.119985Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-08-10T02:41:50.119986Z 0 [Note] InnoDB: Uses event mutexes
2021-08-10T02:41:50.119987Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-08-10T02:41:50.119989Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-08-10T02:41:50.119990Z 0 [Note] InnoDB: Using Linux native AIO
2021-08-10T02:41:50.120159Z 0 [Note] InnoDB: Number of pools: 1
2021-08-10T02:41:50.120302Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-08-10T02:41:50.120305Z 0 [Note] InnoDB: Disabling background log and ibuf IO write threads.
2021-08-10T02:41:50.120962Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-08-10T02:41:50.123247Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-08-10T02:41:50.124141Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-08-10T02:41:50.134246Z 0 [Note] InnoDB: Skipping doublewrite buffer processing due to InnoDB running in read only mode
2021-08-10T02:41:50.134514Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2021-08-10T02:41:50.134561Z 0 [Note] InnoDB: The user has set SRV_FORCE_NO_LOG_REDO on, skipping log redo
2021-08-10T02:41:50.134929Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-08-10T02:41:50.134966Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-08-10T02:41:50.191712Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-08-10T02:41:50.192052Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.34-37 started; log sequence number 0
2021-08-10T02:41:50.192057Z 0 [Note] InnoDB: !!! innodb_force_recovery is set to 6 !!!
2021-08-10T02:41:50.192180Z 0 [Note] Plugin 'FEDERATED' is disabled.
2021-08-10T02:41:50.193955Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`plugin` in the cache. Attempting to load the tablespace with space id 2
2021-08-10T02:41:50.194549Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`gtid_executed` in the cache. Attempting to load the tablespace with space id 18
2021-08-10T02:41:50.196262Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2021-08-10T02:41:50.196272Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2021-08-10T02:41:50.196785Z 0 [Warning] CA certificate ca.pem is self signed.
2021-08-10T02:41:50.196819Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2021-08-10T02:41:50.197096Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2021-08-10T02:41:50.197119Z 0 [Note] IPv6 is available.
2021-08-10T02:41:50.197122Z 0 [Note]   - '::' resolves to '::';
2021-08-10T02:41:50.197144Z 0 [Note] Server socket created on IP: '::'.
2021-08-10T02:41:50.203873Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`server_cost` in the cache. Attempting to load the tablespace with space id 19
2021-08-10T02:41:50.203994Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`engine_cost` in the cache. Attempting to load the tablespace with space id 20
2021-08-10T02:41:50.204855Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_leap_second` in the cache. Attempting to load the tablespace with space id 12
2021-08-10T02:41:50.204947Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_name` in the cache. Attempting to load the tablespace with space id 8
2021-08-10T02:41:50.205034Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone` in the cache. Attempting to load the tablespace with space id 9
2021-08-10T02:41:50.205100Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_transition_type` in the cache. Attempting to load the tablespace with space id 11
2021-08-10T02:41:50.205176Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_transition` in the cache. Attempting to load the tablespace with space id 10
2021-08-10T02:41:50.205458Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`servers` in the cache. Attempting to load the tablespace with space id 3
2021-08-10T02:41:50.208153Z 0 [Note] Event Scheduler: Loaded 0 events
2021-08-10T02:41:50.208242Z 0 [Note] mysqld: ready for connections.
Version: '5.7.34-37'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Percona Server (GPL), Release 37, Revision 7c516e9
2021-08-10T02:42:01.660147Z 2 [ERROR] InnoDB: Failed to find tablespace for table `application`.`cache_page` in the cache. Attempting to load the tablespace with space id 664
2021-08-10T02:42:01.690164Z 2 [ERROR] InnoDB: Failed to find tablespace for table `application`.`cache_bootstrap` in the cache. Attempting to load the tablespace with space id 636
2021-08-10T02:42:03.316917Z 3 [ERROR] InnoDB: Failed to find tablespace for table `application`.`semaphore` in the cache. Attempting to load the tablespace with space id 1065
2021-08-10T02:42:09.998115Z 3 [ERROR] InnoDB: Failed to find tablespace for table `application`.`cache` in the cache. Attempting to load the tablespace with space id 628

When running not in recovery mode, still getting the original mysqld got signal 11 error.

2 Likes
4

I run again with lower level recovery mode --innodb_force_recovery=2 --innodb_purge_threads=0 and the Failed to find tablespace ... errors were gone, then run in normal mode i.e. --innodb_force_recovery=0 and mysql started mysqld: ready for connections, I’m testing that now.

2 Likes
5

Mysql starts but it is hanging, can’t connect. Here are the logs.

2021-08-10T02:52:05.046706Z 0 [Note] mysqld: ready for connections.
Version: '5.7.34-37'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Percona Server (GPL), Release 37, Revision 7c516e9
2021-08-10T02:56:15.000898Z 0 [Warning] InnoDB: A long semaphore wait:
--Thread 140293903963904 has waited at trx0undo.ic line 179 for 241  seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7f98cbac4230 created in file buf0buf.cc line 1438
a writer (thread id 140293903963904) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file trx0undo.ic line 198
Last time write locked in file /mnt/workspace/percona-server-5.7-redhat-binary-rocks-new/label_exp/min-centos-8-x64/test/rpmbuild/BUILD/percona-server-5.7.34-37/percona-server-5.7.34-37/storage/innobase/include/trx0undo.ic line 179
2021-08-10T02:56:15.001082Z 0 [Warning] InnoDB: A long semaphore wait:
--Thread 140294060500736 has waited at srv0srv.cc line 2435 for 241  seconds the semaphore:
X-lock on RW-latch at 0x2f87da8 created in file dict0dict.cc line 1242
a writer (thread id 140293903963904) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file not yet reserved line 0
Last time write locked in file /mnt/workspace/percona-server-5.7-redhat-binary-rocks-new/label_exp/min-centos-8-x64/test/rpmbuild/BUILD/percona-server-5.7.34-37/percona-server-5.7.34-37/storage/innobase/row/row0trunc.cc line 1864
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0

=====================================
2021-08-10 02:56:25 0x7f98c3241700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 16 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1 srv_active, 0 srv_shutdown, 8 srv_idle
srv_master_thread log flush and writes: 8
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 4
--Thread 140293903963904 has waited at trx0undo.ic line 179 for 251  seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7f98cbac4230 created in file buf0buf.cc line 1438
a writer (thread id 140293903963904) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file trx0undo.ic line 198
Last time write locked in file /mnt/workspace/percona-server-5.7-redhat-binary-rocks-new/label_exp/min-centos-8-x64/test/rpmbuild/BUILD/percona-server-5.7.34-37/percona-server-5.7.34-37/storage/innobase/include/trx0undo.ic line 179
--Thread 140294060500736 has waited at srv0srv.cc line 2435 for 251  seconds the semaphore:
X-lock on RW-latch at 0x2f87da8 created in file dict0dict.cc line 1242
a writer (thread id 140293903963904) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file not yet reserved line 0
Last time write locked in file /mnt/workspace/percona-server-5.7-redhat-binary-rocks-new/label_exp/min-centos-8-x64/test/rpmbuild/BUILD/percona-server-5.7.34-37/percona-server-5.7.34-37/storage/innobase/row/row0trunc.cc line 1864
OS WAIT ARRAY INFO: signal count 2
RW-shared spins 0, rounds 4, OS waits 2
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 4.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 559456773
Purge done for trx's n:o < 559414843 undo n:o < 0 state: running
History list length 70
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 559456772, ACTIVE 252 sec truncating table
mysql tables in use 1, locked 1
0 lock struct(s), heap size 1136, 0 row lock(s)
MySQL thread id 2, OS thread handle 140293903963904, query id 4 127.0.0.1 application System lock
TRUNCATE cache_page
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
538 OS file reads, 57 OS file writes, 8 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 305, seg size 307, 0 merges
merged operations:
 insert 0, delete mark 0, delete 0
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 34673, node heap has 2 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 2 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 267655549392
Log flushed up to   267655549392
Pages flushed up to 267655549392
Last checkpoint at  267655549383
Max checkpoint age    80826164
Checkpoint age target 78300347
Modified age          0
Checkpoint age        9
0 pending log flushes, 0 pending chkp writes
10 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 139722752
Dictionary memory allocated 146764
Internal hash tables (constant factor + variable factor)
    Adaptive hash index 2318272         (2219072 + 99200)
    Page hash           139112 (buffer pool 0 only)
    Dictionary cache    701532  (554768 + 146764)
    File system         1175440         (812272 + 363168)
    Lock system         334008  (332872 + 1136)
    Recovery system     0       (0 + 0)
Buffer pool size   8191
Buffer pool size, bytes 134201344
Free buffers       7646
Database pages     541
Old database pages 219
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 507, created 34, written 36
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 541, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
1 RW transactions active inside InnoDB
Process ID=1, Main thread ID=140294060500736, state: enforcing dict cache limit
Number of rows inserted 0, updated 0, deleted 0, read 8
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
InnoDB: ###### Diagnostic info printed to the standard error stream
2021-08-10T02:56:46.001260Z 0 [Warning] InnoDB: A long semaphore wait:
--Thread 140293903963904 has waited at trx0undo.ic line 179 for 272  seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7f98cbac4230 created in file buf0buf.cc line 1438
a writer (thread id 140293903963904) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file trx0undo.ic line 198
Last time write locked in file /mnt/workspace/percona-server-5.7-redhat-binary-rocks-new/label_exp/min-centos-8-x64/test/rpmbuild/BUILD/percona-server-5.7.34-37/percona-server-5.7.34-37/storage/innobase/include/trx0undo.ic line 179
2021-08-10T02:56:46.001276Z 0 [Warning] InnoDB: A long semaphore wait:
--Thread 140294060500736 has waited at srv0srv.cc line 2435 for 272  seconds the semaphore:
X-lock on RW-latch at 0x2f87da8 created in file dict0dict.cc line 1242
a writer (thread id 140293903963904) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file not yet reserved line 0
Last time write locked in file /mnt/workspace/percona-server-5.7-redhat-binary-rocks-new/label_exp/min-centos-8-x64/test/rpmbuild/BUILD/percona-server-5.7.34-37/percona-server-5.7.34-37/storage/innobase/row/row0trunc.cc line 1864
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0

=====================================
2021-08-10 02:57:01 0x7f98c3241700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 36 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1 srv_active, 0 srv_shutdown, 8 srv_idle
srv_master_thread log flush and writes: 8
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 4
--Thread 140293903963904 has waited at trx0undo.ic line 179 for 287  seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7f98cbac4230 created in file buf0buf.cc line 1438
a writer (thread id 140293903963904) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file trx0undo.ic line 198
Last time write locked in file /mnt/workspace/percona-server-5.7-redhat-binary-rocks-new/label_exp/min-centos-8-x64/test/rpmbuild/BUILD/percona-server-5.7.34-37/percona-server-5.7.34-37/storage/innobase/include/trx0undo.ic line 179
--Thread 140294060500736 has waited at srv0srv.cc line 2435 for 287  seconds the semaphore:
X-lock on RW-latch at 0x2f87da8 created in file dict0dict.cc line 1242
a writer (thread id 140293903963904) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file not yet reserved line 0
Last time write locked in file /mnt/workspace/percona-server-5.7-redhat-binary-rocks-new/label_exp/min-centos-8-x64/test/rpmbuild/BUILD/percona-server-5.7.34-37/percona-server-5.7.34-37/storage/innobase/row/row0trunc.cc line 1864
OS WAIT ARRAY INFO: signal count 2
RW-shared spins 0, rounds 4, OS waits 2
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 4.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 559456773
Purge done for trx's n:o < 559414843 undo n:o < 0 state: running
History list length 70
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 559456772, ACTIVE 288 sec truncating table
mysql tables in use 1, locked 1
0 lock struct(s), heap size 1136, 0 row lock(s)
MySQL thread id 2, OS thread handle 140293903963904, query id 4 127.0.0.1 application System lock
TRUNCATE cache_page
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
538 OS file reads, 57 OS file writes, 8 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 305, seg size 307, 0 merges
merged operations:
 insert 0, delete mark 0, delete 0
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 34673, node heap has 2 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 2 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 267655549392
Log flushed up to   267655549392
Pages flushed up to 267655549392
Last checkpoint at  267655549383
Max checkpoint age    80826164
Checkpoint age target 78300347
Modified age          0
Checkpoint age        9
0 pending log flushes, 0 pending chkp writes
10 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 139722752
Dictionary memory allocated 146764
Internal hash tables (constant factor + variable factor)
    Adaptive hash index 2318272         (2219072 + 99200)
    Page hash           139112 (buffer pool 0 only)
    Dictionary cache    701532  (554768 + 146764)
    File system         1175440         (812272 + 363168)
    Lock system         334008  (332872 + 1136)
    Recovery system     0       (0 + 0)
Buffer pool size   8191
Buffer pool size, bytes 134201344
Free buffers       7646
Database pages     541
Old database pages 219
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 507, created 34, written 36
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 541, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
1 RW transactions active inside InnoDB
Process ID=1, Main thread ID=140294060500736, state: enforcing dict cache limit
Number of rows inserted 0, updated 0, deleted 0, read 8
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
2 Likes
6

I’ve run it again with --innodb_adaptive_hash_index=0, things worked with connecting with the mysql client, then run again with that option and things work when connecting with mysql client but not from the application. Things looking better, troubleshooting the application not connecting.

2 Likes
7

Great to hear you can recover the server. The first task you should initiate is to take a full logical backup of the database using mysqldump or mydumper. Then use this backup to restore into a clean new datadir that doesn’t have any past occurrences of corruption.

Since you happen to be running Percona Server, you may then also want to configure
[innodb_corrupt_table_action=salvage](https://www.percona.com/doc/percona-server/8.0/reliability/innodb_corrupt_table_action.html#innodb_corrupt_table_action) so that the database attempts to read corrupted pages and will recover the maximum amount of data.

1 Like