Percona server 8.0.35-27 crash on temporary tables

MySQL & MariaDB
1

Hello.
My server crashed with following stack and errors.
There is errors like:
[ERROR] [MY-011899] [InnoDB] [FATAL] Unable to read page [page id: space=4243767282, page number=0]
And I see this space in INNODB_SESSION_TEMP_TABLESPACES (actually there is two spaces that noted in the log and both are temporary)
Any idea how I can investigate it future?

/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x29) [0x55ae9dc01089]
/usr/sbin/mysqld(print_fatal_signal(int)+0x291) [0x55ae9c689491]
/usr/sbin/mysqld(my_server_abort()+0x66) [0x55ae9c6896a6]
/usr/sbin/mysqld(my_abort()+0xa) [0x55ae9dbfa42a]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2f2) [0x55ae9df5c302]
/usr/sbin/mysqld(+0x2c1019e) [0x55ae9df5f19e]
/usr/sbin/mysqld(Buf_fetch<Buf_fetch_other>::read_page()+0x183) [0x55ae9dfe4303]
/usr/sbin/mysqld(Buf_fetch_other::get(buf_block_t*&)+0xea) [0x55ae9dfed2ea]
/usr/sbin/mysqld(Buf_fetch<Buf_fetch_other>::single_page()+0x41) [0x55ae9dfedaf1]
/usr/sbin/mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, ut::Location, mtr_t*, bool)+0x136) [0x55ae9dfee106]
/usr/sbin/mysqld(fsp_get_space_header_block(unsigned int, page_size_t const&, mtr_t*, buf_block_t**)+0x3f) [0x55ae9e17276f]
/usr/sbin/mysqld(fsp_reserve_free_extents(unsigned long*, unsigned int, unsigned long, fsp_reserve_t, mtr_t*, unsigned int)+0xe2) [0x55ae9e17df72]
/usr/sbin/mysqld(fseg_create_general(unsigned int, unsigned int, unsigned long, bool, mtr_t*)+0x14d) [0x55ae9e185ebd]
/usr/sbin/mysqld(btr_create(unsigned long, unsigned int, unsigned long, dict_index_t*, mtr_t*)+0x16e) [0x55ae9dfb1c0e]
/usr/sbin/mysqld(dict_create_index_tree_in_mem(dict_index_t*, trx_t*)+0x171) [0x55ae9e09fd91]
/usr/sbin/mysqld(row_create_index_for_mysql(dict_index_t*, trx_t*, unsigned long const*, dict_table_t*)+0x3a5) [0x55ae9de56925]
/usr/sbin/mysqld(create_table_info_t::create_table(dd::Table const*, dd::Table const*)+0x8d9) [0x55ae9dc80cb9]
/usr/sbin/mysqld(int innobase_basic_ddl::create_impl<dd::Table>(THD*, char const*, TABLE*, HA_CREATE_INFO*, dd::Table*, bool, bool, bool, unsigned int, unsigned int, dd::Table const*)+0x1a8
) [0x55ae9dc81048]
/usr/sbin/mysqld(ha_innobase::create(char const*, TABLE*, HA_CREATE_INFO*, dd::Table*)+0x7c) [0x55ae9dc8214c]
/usr/sbin/mysqld(+0x12546d8) [0x55ae9c5a36d8]
/usr/sbin/mysqld(instantiate_tmp_table(THD*, TABLE*)+0x14b) [0x55ae9c5a658b]
/usr/sbin/mysqld(create_tmp_table(THD*, Temp_table_param*, mem_root_deque<Item*> const&, ORDER*, bool, bool, unsigned long long, unsigned long long, char const*)+0x3740) [0x55ae9c5ade50]
/usr/sbin/mysqld(mysql_schema_table(THD*, LEX*, Table_ref*)+0x375) [0x55ae9c539045]
/usr/sbin/mysqld(open_tables(THD*, Table_ref**, unsigned int*, unsigned int, Prelocking_strategy*)+0x28b) [0x55ae9c3d8f1b]
/usr/sbin/mysqld(open_tables_for_query(THD*, Table_ref*, unsigned int)+0x7e) [0x55ae9c3daa3e]
/usr/sbin/mysqld(Sql_cmd_dml::prepare(THD*)+0xf1) [0x55ae9c512931]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0xdf) [0x55ae9c51446f]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xca0) [0x55ae9c499070]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x609) [0x55ae9c49cfb9]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x2718) [0x55ae9c49fc68]
/usr/sbin/mysqld(do_command(THD*)+0x1b1) [0x55ae9c4a06d1]
/usr/sbin/mysqld(+0x13299b8) [0x55ae9c6789b8]
/usr/sbin/mysqld(+0x2f31d73) [0x55ae9e280d73]
/lib64/libc.so.6(+0x82d99) [0x7fbb27960d99]
/lib64/libc.so.6(+0xfc4ec) [0x7fbb279da4ec]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f5456f6ace0):                               is an invalid pointer
Connection ID (thread ID): 36697406
Status: NOT_KILLED

Thanks

2

Hello @egel,
If MySQL crashes while it is starting up like this, then you will need to perform a logical dump/reload to clean out the broken data dictionary. You could also try to modify the tables yourself, but that’s beyond the scope here.

https://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html

3

The crash seems to occurs not during startup.
And the “corruption” is related only to two tables in INNODB_SESSION_TEMP_TABLESPACES.
Since the temp tablespaces re-created during startup I am really curious about your opinion what and how can I investigate there.

Thanks

4

The crash seems to occurs not during startup.

Ok. You didn’t specify when it happens so I made an assumption. When exactly does this crash happen? If this is repeatable, please provide the steps so we can test it out.

Temporary tables do not persist beyond sessions. After disconnecting a session, temporary tables are dropped. If for some reason Innodb has held onto a handle for a temporary table, even after restart, there might be some corruption in the data dictionary that needs to be fixed by doing a logical dump/restore.

5

Thanks for your reply,
Sorry went off for few days.
The crash happened, I’ve rebooted server and since then the server was ok.
The crash seems that, happened during massive online change (rebuild large table with new PK).
The table got rebuilt despite the crash.
I understand that this is not enough information, but this is all I have.
Therefore I am asking for suggestion, how to collect more relevant information in such situation.

Best Regards