Mysql gives signal 11 during upgrade

MySQL & MariaDB
,
1

When trying to upgrade from 8.0.21 to 8.0.31 or 8.0.33, i get:

"2023-06-19T13:18:20.351132Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version ‘80021’ to ‘80033’.
…
2023-06-19T13:18:58Z UTC - mysqld got signal 11 ;
…
Thread pointer: 0x7ff601820000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7ff6003fea80 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x216624d]
…
/lib64/libc.so.6(clone+0x6d) [0x7ff615a66b0d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 1
Status: NOT_KILLED"

2

i tried mysqlcheck -c --all-databases. nothing seems corrupted.

3 
2023-06-19T13:18:20.351132Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version '80021' to '80033'.
2023-06-19T13:18:20.426237Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2023-06-19T13:18:31.345570Z 0 [Note] [MY-012350] [InnoDB] 6threads have validated 23205 out of 51004 tablespaces so far.
2023-06-19T13:18:41.345958Z 0 [Note] [MY-012350] [InnoDB] 6threads have validated 46761 out of 51004 tablespaces so far.
2023-06-19T13:18:43.150218Z 1 [Note] [MY-012356] [InnoDB] Scanned 51004 tablespaces. Validated 51004.
2023-06-19T13:18:58Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=ca2d331a4fa161d963a8b5ca191d44e8528cb59c
Server Version: 8.0.33-25 Percona Server (GPL), Release 25, Revision 9468fd1d

Thread pointer: 0x7ff601820000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff6003fea80 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x216624d]
/usr/sbin/mysqld(print_fatal_signal(int)+0x39f) [0x11fddef]
/usr/sbin/mysqld(handle_fatal_signal+0xc5) [0x11fdec5]
/lib64/libpthread.so.0(+0xf630) [0x7ff6176b3630]
/usr/sbin/mysqld(dd::Column_type_element_impl::validate() const+0x45) [0x20a4ad5]
/usr/sbin/mysqld(bool dd::Collection<dd::Column_type_element*>::restore_items<dd::Column_impl, bool (*)(dd::Column_type_element_impl const*, dd::Column_type_element_impl const*)>(dd::Column_impl*, dd::Open_dictionary_tables_ctx*, dd::Raw_table*, dd::Object_key*, bool (*)(dd::Column_type_element_impl const*, dd::Column_type_element_impl const*))+0x216) [0x1ea15a6]
/usr/sbin/mysqld(bool dd::Collection<dd::Column*>::restore_items<dd::Abstract_table_impl, bool (*)(dd::Column_impl const*, dd::Column_impl const*)>(dd::Abstract_table_impl*, dd::Open_dictionary_tables_ctx*, dd::Raw_table*, dd::Object_key*, bool (*)(dd::Column_impl const*, dd::Column_impl const*))+0x18c) [0x1ea0e0c]
/usr/sbin/mysqld(dd::Table_impl::restore_children(dd::Open_dictionary_tables_ctx*)+0x41) [0x20f4621]
/usr/sbin/mysqld(dd::Entity_object_table_impl::restore_object_from_record(dd::Open_dictionary_tables_ctx*, dd::Raw_record const&, dd::Entity_object**) const+0xc5) [0x20a6065]
/usr/sbin/mysqld(bool dd::cache::Dictionary_client::foreach<dd::Table>(dd::Object_key const*, std::function<bool (std::unique_ptr<dd::Table, std::default_delete<dd::Table> >&)> const&) const+0x1f1) [0x1f1ca31]
/usr/sbin/mysqld() [0x210fdb2]
/usr/sbin/mysqld(bool dd::cache::Dictionary_client::foreach<dd::Schema>(dd::Object_key const*, std::function<bool (std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&)> const&) const+0x142) [0x1f1e6d2]
/usr/sbin/mysqld(dd::upgrade::do_server_upgrade_checks(THD*)+0x1b7) [0x210f387]
/usr/sbin/mysqld(dd::bootstrap::restart(THD*)+0x13d) [0x1f0448d]
/usr/sbin/mysqld() [0x2131e31]
/usr/sbin/mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0xc40) [0x2139990]
/usr/sbin/mysqld() [0x12c8226]
/usr/sbin/mysqld() [0x2592465]
/lib64/libpthread.so.0(+0x7ea5) [0x7ff6176abea5]
/lib64/libc.so.6(clone+0x6d) [0x7ff615a66b0d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 1
Status: NOT_KILLED
4

Hi @FrankD,
Do you have an ENUM or SET columns in your schema? That aside, I’ll pass this along to our devs and they may need more info from you like a schema dump. Is this repeatable?

5

Hi @matthewb ,
the thing is that this is one of 30 servers i manage. And all of those servers have the same databases (1 db per client) . And this is the only server giving me that issue, all the others have been migrated to 8.0.31. They should ( in theory) all be the same config.

Is there a way i could get more info in the log file? To help us pinpoint where the issue is.

6

I 've checked and yes, i have a “set” column type in my customer database.

7

Oups, i have Enum also

8

There is nothing else you can get in the log file. Looking at the stack trace, the last function to be called is
dd::Column_type_element_impl::validate() and so we zip over to the source code and find this function:

L70-72 would have printed a specific error message, which you didn’t get. 75-77 could have been the issue since there is an assert on the character set. What charset are you using on this customer?

79-80 deal with checking ENUM/SET lengths. And that’s the end of the function. So it’s possibly something to do with charset or ENUM/SET.

9

If you want more info, you can download/install the debug package for PS 8.0.33 and then start MySQL inside gdb in order to grab a complete stacktrace, but that’s pretty advanced debugging.

10

he he, my only customer in the APAC region. Character set makes a lot of sense, will check and test that. Thanks.

11

If you can, please post the troublesome schema. I’d like to attempt to repeat/reproduce myself and report back to our dev team.

12

charset is the same across all servers:
[root@prod-apac-tc-db-201 ~]# grep ‘collation|character’ /etc/my.cnf
default-character-set=utf8mb4
collation-server = utf8mb4_0900_ai_ci
character-set-server = utf8mb4

13

Could be linked to collation of a database:
CREATE DATABASE cust_gmonikko /*!40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci_ks / /!80016 DEFAULT ENCRYPTION=‘N’ */

14

all other clients use : DEFAULT_CHARACTER_SET_NAME:utf8mb4
DEFAULT_COLLATION_NAME: utf8mb4_0900_ai_ci

15

Sorry i cannot upload schema, sensitive info and trade secrets…

16

utf8mb4_0900_ai_ci_ks → How did the customer end up using this?

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 8.0.21 MySQL Community Server - GPL

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE DATABASE test /*!40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci_ks  */;
ERROR 1273 (HY000): Unknown collation: 'utf8mb4_0900_ai_ci_ks'
mysql> CREATE DATABASE test /*!40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci  */;
Query OK, 1 row affected (0.02 sec)
17

utf8mb4_ja_0900_ai_ci_ks was added in 8.0.2:
https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-2.html
But that is not what @FrankD is using it seems.

@Frank, this certainly seems like the issue. If you can please explain how you originally used utf8mb4_0900_ai_ci_ks (which does not appear to be a valid collation), that would be helpful. I went back to 5.7 and it isn’t there either.

18

Hi all,
ok so we are using a custom collation in /usr/share/mysql/charsets/Index.xml
i m uploading the file here:
Index.xml.txt (109.0 KB)

excerpt:




ﾞ

This is to be able to differentiate 2 Japanese characters that looks the same in utf8 standard.

19

Hello @FrankD,
Ok. Custom collation. Unfortunately, support for these is now deprecated:

https://dev.mysql.com/doc/refman/8.0/en/adding-collation.html

Warning
User-defined collations are deprecated; you should expect support for them to be removed in a future version of MySQL.

I suggest that you first take a look at the proper utf8mb4_ja_0900_ai_ci_ks collation and see if that fits your needs and if so, convert your tables to use this collation. After that, you can proceed with upgrading to 8.0.33.

20

So i confirm, but putting back my custom collation in the Index.xml that comes with 8.0.31, upgrade ran fine. Thanks a lot @matthewb for all the help, really appreciated!