*** buffer overflow detected *** trying to query a PXC XtraDB cluster 5.7

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

hi,

i have a running PXC 5.7 running with proxysql.
the setup is made using percona docker images

all nodes are up and show status like '%wsrep%'; shows that all nodes are up and synced.

when i try to access one node from another host in the same network using mysql client. i have these messages:

from the client:


Warning: Using a password on the command line interface can be insecure.
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 2

from the node:


2017-03-21T15:30:53.784961Z 0 [Note] mysqld: ready for connections.
Version: '5.7.17-11-57' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel11, Revision e2a7fdd, WSREP version 27.20, wsrep_27.20
2017-03-21T15:35:23.062096Z 0 [Note] WSREP: (4d3c097c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.20.2.3:4567 
*** buffer overflow detected ***: mysqld terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7f6f4a7c51af]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f6f4a84aaa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7f6f4a848cc0]
mysqld(_Z19find_or_create_hostP10PFS_threadPKcj+0x395)[0x1264035]
mysqld(_Z22find_or_create_accountP10PFS_threadPKcjS2_j+0x3d2)[0x12ae8f2]
mysqld(_Z18set_thread_accountP10PFS_thread+0x36)[0x126bde6]
mysqld(pfs_set_thread_account_v1+0xa0)[0x124df40]
mysqld(_Z16acl_authenticateP3THD19enum_server_commandb+0xc52)[0x7c4882]
mysqld[0xc4a054]
mysqld(_Z22thd_prepare_connectionP3THDb+0x5a)[0xc4b8ba]
mysqld(handle_connection+0x30d)[0xd590bd]
mysqld(pfs_spawn_thread+0x1b4)[0x124d874]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f6f4c7fa064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6f4a83a62d]
======= Memory map: ========
***snip***
15:35:23 UTC - mysqld got signal 6 ;
***snip***


key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=152
thread_count=4
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68407 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f6f1c2afae0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6f187f7ab0 thread_stack 0x30000
mysqld(my_print_stacktrace+0x2c)[0xebe56c]
mysqld(handle_fatal_signal+0x479)[0x7a4b89]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f6f4c801890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f6f4a787067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f6f4a788448]
/lib/x86_64-linux-gnu/libc.so.6(+0x731b4)[0x7f6f4a7c51b4]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f6f4a84aaa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7f6f4a848cc0]
mysqld(_Z19find_or_create_hostP10PFS_threadPKcj+0x395)[0x1264035]
mysqld(_Z22find_or_create_accountP10PFS_threadPKcjS2_j+0x3d2)[0x12ae8f2]
mysqld(_Z18set_thread_accountP10PFS_thread+0x36)[0x126bde6]
mysqld(pfs_set_thread_account_v1+0xa0)[0x124df40]
mysqld(_Z16acl_authenticateP3THD19enum_server_commandb+0xc52)[0x7c4882]
mysqld[0xc4a054]
mysqld(_Z22thd_prepare_connectionP3THDb+0x5a)[0xc4b8ba]
mysqld(handle_connection+0x30d)[0xd590bd]
mysqld(pfs_spawn_thread+0x1b4)[0x124d874]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f6f4c7fa064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6f4a83a62d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 60
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.

what the heck is going on ? how come a simply connexion breaks a mysql node ?

any clues appreciated :slight_smile:

Regards,

2

That’s a bad crash… but this immediate crash usually points that something wrong with the environment.
Do you use proper binaries?
I mean using .deb on CentOS or .rpm on Ubuntu would be problematic.

Also the crashed function
find_or_create_host may point that there is some problem with DNS. is name resolution working properly on this host?

3

Hi!

I’m using official Docker image from percona.

I’ll double check dns resolution but it should be OK.

I’m testing the proxysql /pxc Docker swarm setup.

As pxc 5.6 docker image is broken, I’m using 5.7 Docker image.

Very strange bug btw!

Regards,

4

Hi,

Checked everything around.
For wathever the reason it seems related to swarm itself, using Docker only makes it work perfectly!

I still don’t know why, but if you may have any clue, I’d be glad to hear

Regards,

5

Ok, two years five months after this thread was made, I got the exact same error as you did. The issue behind the scene is indeed related to network/DNS. Although not sure about what the root cause is, after turning on [U], the issue is gone. So essentially, mysqld reaches a bug/defect or whatever when handling authentication during a connection. It tries to resolve the host name and fails and terminated the daemon instead of gracefully logging a warning and maybe failing the request. Hope this helps.

[mysqld]
skip-name-resolve