So this evening, my mysql (8.0.31) instance restarted, looks like the OOM took it down for which I need to investigate.
However, upon restart… its not yet come back up, I’m looking for some help / assistance / guidance here on what I can do to bring it back up (even to drop broken tables / deal with data recovery).
When it starts back up, it gets so far, then crashes wiht an error about possible corruption of an index page - but no clue which one? I’ve been through various innodb_force_recovery modes with no success, I’ve also set innocheck running on all idb files - it found absolutely nothing wrong.
2023-05-18T19:38:14.995693Z 0 [Warning] [MY-010139] [Server] Changed limits: max_open_files: 10000 (requested 12500) 2023-05-18T19:38:14.995699Z 0 [Warning] [MY-010142] [Server] Changed limits: table_open_cache: 3745 (requested 4000) 2023-05-18T19:38:15.177912Z 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release. 2023-05-18T19:38:15.177920Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release. 2023-05-18T19:38:15.177932Z 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replica_updates instead. 2023-05-18T19:38:15.177942Z 0 [Warning] [MY-011069] [Server] The syntax '--transaction-write-set-extraction' is deprecated and will be removed in a future release. 2023-05-18T19:38:15.177997Z 0 [Warning] [MY-010915] [Server] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release. 2023-05-18T19:38:15.179186Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.31) starting as process 3953821 2023-05-18T19:38:15.182025Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in orde$ 2023-05-18T19:38:15.182038Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8mb3_unicode_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead. 2023-05-18T19:38:15.198423Z 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to compute innodb_redo_log_capacity=4294967296. Please use innodb_redo$ 2023-05-18T19:38:15.201987Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2023-05-18T19:38:18.850561Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2023-05-18T19:38:19.559295Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2023-05-18T19:38:19.559335Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. 2023-05-18T19:38:19.578247Z 0 [Warning] [MY-010604] [Repl] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=db1-r$ 2023-05-18T19:38:19.593617Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.31' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server - GPL. 2023-05-18T19:38:19.593640Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock 2023-05-18T19:42:20.406260Z 892 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.' 2023-05-18T19:42:45.684507Z 3169 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.' 2023-05-18T19:42:45.685661Z 892 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv4 localhost address to the allowlist. It is mandatory that it is added.' 2023-05-18T19:42:45.685693Z 892 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv6 localhost address to the allowlist. It is mandatory that it is added.' 2023-05-18T19:42:45.687532Z 3328 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''$ 2023-05-18T19:42:46.792217Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor' 2023-05-18T19:42:46.792607Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 10.10.0.100:3306 on view 16844389657918695:1.' 2023-05-18T19:42:46.792999Z 0 [System] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within the replication group.' 2023-05-18T19:42:46.793353Z 3527 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.' 2023-05-18T19:46:18.236118Z 0 [ERROR] [MY-012721] [InnoDB] N heap is wrong 179, 155 len 16384; hex 6ca889eb000000de0000461400002d6000001d80696ad87045bf00000000000000000000021300223f6380b3162e0d6f02a100050000008b0000000000000000000000000000000002fe0000000000000000000000000000000000000000010002001c696e66696d756d0001000b$ InnoDB: End of page dump InnoDB: Page may be an index page where index id is 766 2023-05-18T19:46:18.304963Z 0 [ERROR] [MY-011937] [InnoDB] [FATAL] Apparent corruption of an index page [page id: space=531, page number=222] to be written to data file. We intentionally crash the server to prevent corrupt data from end$ 2023-05-18T19:46:18.304985Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: buf0dblwr.cc:1537:ib::fatal triggered thread 139846900352768 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be
I’m at a bit of a loss here as to what I can do to bring this server back up.
Any help appriciated.