Mysqlrouter "Application got fatal signal: 11" after upgrade to percona mysql 8.0.43-34-1.bullseye

MySQL & MariaDB Percona Distribution for MySQL
1

Hi

I just upgraded percona from version (8.0.42-33-1.bullseye) to (8.0.43-34-1.bullseye) and the percona-mysql-router is dumping:

/usr/bin/mysqlrouter -c mysqlrouter.conf
Application got fatal signal: 11
stack_bottom = 0 thread_stack 0x0
/usr/lib/mysqlrouter/private/libmysqlharness.so.1(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x7f97ce701a5e]
/usr/lib/mysqlrouter/private/libmysqlharness.so.1(+0x9d31d) [0x7f97ce6f331d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f97ceeff140]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(file_info::RegisterFilename(int, char const*, file_info::OpenType)+0x58) [0x7f97ce88df88]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(my_open(char const*, int, int)+0x5e) [0x7f97ce80e39e]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(+0x1a40ad) [0x7f97ce8d90ad]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(+0xd6680) [0x7f97ce80b680]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x1034f) [0x7f97ceefc34f]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(my_charset_get_by_name(MY_CHARSET_LOADER*, char const*, unsigned int, int)+0x72) [0x7f97ce80bb22]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(get_charset_by_csname(char const*, unsigned int, int)+0x80) [0x7f97ce80bc30]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(mysql_init_character_set(MYSQL*)+0x6f) [0x7f97ce7e3dcf]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(+0xb22fb) [0x7f97ce7e72fb]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(connect_helper+0x29) [0x7f97ce7e77a9]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(mysql_real_connect+0x110) [0x7f97ce7e18a0]
/usr/lib/mysqlrouter/private/libmysqlrouter.so.1(mysqlrouter::MySQLSession::connect(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std:/usr/lib/mysqlrouter/private/libmysqlrouter_metadata_cache.so.1(ClusterMetadata::do_connect(mysqlrouter::MySQLSession&, mysql_harness::TCPAddress const&)+0x152) [0x7f97cccf20a2]
/usr/lib/mysqlrouter/private/libmysqlrouter_metadata_cache.so.1(ClusterMetadata::connect_and_setup_session(mysql_harness::TCPAddress const&)+0x121) [0x7f97cccf2261]
/usr/lib/mysqlrouter/private/libmysqlrouter_metadata_cache.so.1(GRClusterMetadata::fetch_cluster_topology(std::atomic const&, mysqlrouter::TargetCluster&, unsigned int, std::vector<mysql_harness::TCPAddress, std::allocator<mysql_harness::TCPAddress> > const&, bool, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, unsigned long&)+0x3dd) [0x7f97cccfdbfd]
/usr/lib/mysqlrouter/private/libmysqlrouter_metadata_cache.so.1(GRMetadataCache::refresh(bool)+0x7f) [0x7f97ccd0f42f]
/usr/lib/mysqlrouter/private/libmysqlrouter_metadata_cache.so.1(MetadataCache::refresh_thread()+0x8d) [0x7f97ccd0e58d]
/usr/lib/mysqlrouter/private/libmysqlrouter_metadata_cache.so.1(MetadataCache::run_thread(void*)+0x1a) [0x7f97ccd0ebaa]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f97ceef3ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f97ce00dadf]

Could any one help in resolving this

2

Hi @Salvatore_Rotella

It seems a bug, I found a verified one which is very similar to the backtrace

3

Hi @Salvatore_Rotella,

I investigated this crash in depth. The backtrace and build configuration point to a specific root cause.

Root cause: LTO in 8.0.43-34 Debian packages

Percona Server 8.0.43-34 introduced -DWITH_LTO=ON in the Debian package build rules (build-ps/debian/rules). This is the only meaningful change between 8.0.42 and 8.0.43. All source code in the crash path (mysys/my_file.cc, mysys/my_open.cc, sql-common/client.cc, strings/ctype.cc) is byte-for-byte identical between both versions.

The crash mechanism relates to how LTO changes symbol resolution. mysys (the MySQL system library) is a static convenience library. In 8.0.42 (no LTO), only libmysqlrouter.so.1 contains mysys. The mysqlrouter binary and plugin libraries resolve mysys functions at runtime via libmysqlrouter.so.1.

Under LTO (8.0.43), the linker merges the entire MySQL client and mysys into multiple components: the mysqlrouter binary, libmysqlrouter.so.1, and plugin .so files. Each gets its own copy of fivp (the file info vector pointer in my_file.cc), which lives in an anonymous namespace (internal linkage per object).

The crash occurs because the binary’s my_init() resolves at link time to the binary’s own embedded copy (LTO inlined it). This initializes only the binary’s fivp. libmysqlrouter.so.1’s fivp is never initialized because its my_init() is never called. When the metadata cache thread calls mysql_real_connect, the charset initialization path in libmysqlrouter.so.1 (mysql_init_character_set → my_charset_get_by_name → my_open → RegisterFilename) dereferences the library’s NULL fivp, causing SIGSEGV. Your backtrace confirms this: every frame in the crash path is in libmysqlrouter.so.1.

The plugin duplication is a secondary effect: metadata_cache.so grows from 85KB to 7,667KB (90x) and gains its own RegisterFilename symbol. This would cause the same crash during metadata cache runtime, but the library crash triggers first.

Key evidence: The -Wno-error=free-nonheap-object flag was also added to the 8.0.43 build, which suggests Percona was already seeing LTO-related memory warnings during compilation.

Note on Bug #111177: @yunus.uyanik linked upstream MySQL Bug #111177, but that is a different crash. Bug #111177 involves a REST API race condition in get_connections / JsonDocument::SetMember. Your backtrace goes through mysql_init_character_set -> my_open -> RegisterFilename, which is a charset initialization path, not a REST API issue.

Reproduction and source build verification

I reproduced this crash in a controlled environment: single-node InnoDB Cluster on Debian Bullseye x86_64, using the official Percona APT packages. Router 8.0.42-33 connects to the cluster normally. After upgrading Router to 8.0.43-34, it crashes immediately with the same backtrace you reported (MetadataCache::refresh_thread → mysql_real_connect → RegisterFilename → SIGSEGV). Downgrading back to 8.0.42-33 restores normal operation.

I then built Router from the patched Percona-Server-8.0.43-34 source on a native x86_64 EC2 instance (c7i.8xlarge, Debian Bullseye, GCC 10). The patch strips -flto flags from the Router subdirectory in router/CMakeLists.txt, following the same pattern already used in that file for stripping -fuse-ld=gold (lines 37-38).

Three-way comparison of the crash-path plugin (metadata_cache.so):

Binary LTO Size RegisterFilename Bootstrap
APT 8.0.42-33 No 85KB 0 Clean
APT 8.0.43-34 Yes 7,667KB 1 SIGSEGV
Source-built 8.0.43-34 (patched) No 513KB 0 Clean

The patched build eliminates RegisterFilename from the crash-path plugins. These plugins resolve mysys symbols via the PLT to libmysqlrouter.so.1, the same behavior as 8.0.42.

I filed JIRA PS-10592 to track this. The LTO flag was added via PKG-870 (commit 5964e341). The patch and full reproduction environment are at nogueiraanderson/ps-10592-router-lto-fix.

Workaround (verified)

Downgrade percona-mysql-router to the pre-LTO version while keeping the server at 8.0.43:

apt install percona-mysql-router=8.0.42-33-1.bullseye

Router 8.0.42 uses its own private libraries and communicates with the server via the MySQL wire protocol (backward compatible). Running Router 8.0.42 with Server 8.0.43 is safe.

Could you confirm which packages were upgraded? Running dpkg -l | grep percona would clarify whether Router was also upgraded to 8.0.43 or remained at 8.0.42. If Router was upgraded, the LTO hypothesis explains the crash directly.

References