Xtradb crash on ubuntu upgrade

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Hello,
We are trying to upgrade our ubuntu cluster from Ubuntu 14.04 to Ubuntu 16.04.
We got a problem with percona xtradb cluster.
We use, on the old cluster, percona xtradb 5.6.41-28.28-1.trusty package from percona repo. If we upgrade ubuntu with this package, this package run fine on ubuntu 16.04. But if we upgrade to 5.6.41-28.28-1.xenial, all seem to work good (clustercheck is ok, sst is ok, etc…) but as soon as we direct some trafic to the upgraded node, it crash.
The only message we have in logs is:

Thread pointer: 0x2969f80
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7f7f351ace38 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x92b9bb]
/usr/sbin/mysqld(handle_fatal_signal+0x49a)[0x691bba]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f83117d9390]
/lib/x86_64-linux-gnu/libc.so.6(+0x14dec2)[0x7f8310caaec2]
/usr/sbin/mysqld(_Z21find_or_create_digestP10PFS_threadPK18sql_digest_storagePKcj+0x39b)[0x93ef1b]
/usr/sbin/mysqld[0x968843]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x386)[0x7209b6]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x244)[0x7243e4]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x182)[0x6e85b2]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6e87c0]
/usr/sbin/mysqld(pfs_spawn_thread+0x146)[0x964416]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f83117cf6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f8310c6441d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 75
Status: NOT_KILLED

When we try to execute manually some requests (select, inserts, etc…) we have no problem, but as soon as we add the real trafic it crash. We have try to log all request, and to replay requests concerned by the thread ID that crash, without any crash.
How is it possible that trusty version have no problem on xenial but xenial version have ? How can we upgrade ?

Thanks !

2

Hello,

This crash is identical as in [URL][PXC-1909] LP #1597831: (xenial package only) xtradb-cluster-server SIGSEGV in find_or_create_digest - Percona JIRA and this function:

/usr/sbin/mysqld(_Z21find_or_create_digestP10PFS_threadPK18s ql_digest_storagePKcj+0x39b)[0x93ef1b]

refers to EVENTS_STATEMENTS_SUMMARY_BY_DIGEST in Performance Schema.

Unfortunately, we are missing the test case in this bug report. I have tried to trigger this crash by enabling all the P_S instruments and consumers, and running various sysbench tests in multiple threads, and in the same time, concurrent queries to performance_schema.events_statements_summary_by_digest, but I am not able to trigger any crash.
Used the following packages:

root@ubuntu-xenial:~# dpkg -l|grep -i percona
ii percona-release 0.1-6.xenial all Package to install Percona gpg key and APT repo
ii percona-xtrabackup 2.3.10-1.xenial amd64 Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.6 5.6.41-28.28-1.xenial amd64 Percona XtraDB Cluster database client binaries
ii percona-xtradb-cluster-common-5.6 5.6.41-28.28-1.xenial amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-3 3.28-1.xenial amd64 Metapackage for latest version of galera3.
ii percona-xtradb-cluster-galera-3.x 3.28-1.xenial amd64 Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.6 5.6.41-28.28-1.xenial amd64 Percona XtraDB Cluster database server binaries

Can you send more details on what your workload looks like? What kind of queries are you executing for performance_schema tables? What are your P_S settings?