# My cluster crash randomly **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [June 15, 2015, 2:11am UTC](https://forums.percona.com/t/my-cluster-crash-randomly/4264 "2015-06-15T02:11:11Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Rossonero224](https://avatars.discourse-cdn.com/v4/letter/r/8491ac/32.png) [@Rossonero224](https://forums.percona.com/u/Rossonero224)\ **Post date:** [June 15, 2015, 2:11am UTC](https://forums.percona.com/t/my-cluster-crash-randomly/4264/1 "2015-06-15T02:11:11Z")
Hi, i ran a cluster with 3 nodes. It’s created and run normally in about 1 day before 1 node crash randomly. My information 1. Server version: 5.6.21-70.1-56-log Percona XtraDB Cluster (GPL), Release rel70.1, Revision 938, WSREP version 25.8, wsrep\_25.8.r4150 2. Ram 32G, CPU 24 core, 4 HDD - raid 10 3. datafile: 4G 4. file: /etc/my.conf [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 ## REPLICATE # Path to Galera library wsrep\_provider=/usr/lib64/libgalera\_smm.so wsrep\_provider\_options=“gcache.size = 1G; gcache.page\_size = 512M; gcs.fc\_limit = 512” wsrep\_slave\_threads=24 wsrep\_restart\_slave=1 wsrep\_forced\_binlog\_format=ROW # Cluster connection URL contains IPs of node#1, node#2 and node#3 wsrep\_cluster\_address=gcomm://192.168.1.83,192.168.1.84,192.168.1.85 # In order for Galera to work correctly binlog format should be ROW binlog\_format=ROW # MyISAM storage engine has only experimental support default\_storage\_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb\_autoinc\_lock\_mode=2 # Node #2 address wsrep\_node\_address=192.168.1.xx # Cluster name wsrep\_cluster\_name=my\_centos\_cluster # SST method wsrep\_sst\_method=xtrabackup-v2 #Authentication for SST method wsrep\_sst\_auth=“xxx:xxx” # Maximum number of rows in write set wsrep\_max\_ws\_rows=262144 # Maximum size of write set wsrep\_max\_ws\_size=2147483648 #################### TUNNING ######################## ###### Slow query log slow\_query\_log=1 slow\_query\_log\_file =/var/log/mysql/slow\_queries.log long\_query\_time=4 connect\_timeout=300 skip\_name\_resolve innodb\_flush\_log\_at\_trx\_commit=2 innodb\_file\_per\_table=1 max\_allowed\_packet=1G max\_connect\_errors=1000000 innodb\_buffer\_pool\_size=4G read\_buffer\_size=4M read\_rnd\_buffer\_size=4M join\_buffer\_size=8M sort\_buffer\_size=4M innodb\_log\_buffer\_size=16M thread\_cache\_size=256 innodb\_additional\_mem\_pool\_size=32M innodb\_flush\_method=O\_DIRECT log\_queries\_not\_using\_indexes=1 innodb\_thread\_concurrency=0 wait\_timeout=300 interactive\_timeout=300 max\_connections=800 innodb\_fast\_shutdown=0 open\_files\_limit=10000 table\_open\_cache=3000 tmp\_table\_size=32M max\_heap\_table\_size=32M ##### Set Ramdisk tmpdir = /usr/mysqltmp ####################### 1. Error message 05:05:00 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at [https://bugs.launchpad.net/percona-xtradb-cluster](https://bugs.launchpad.net/percona-xtradb-cluster) key\_buffer\_size=8388608 read\_buffer\_size=4194304 max\_used\_connections=45 max\_threads=802 thread\_count=27 connection\_count=2 It is possible that mysqld could use up to key\_buffer\_size + (read\_buffer\_size + sort\_buffer\_size)\*max\_threads = 6590372 K bytes of memory Hope that’s ok; if not, decrease some variables in the equation. Thread pointer: 0xa09eba0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong… stack\_bottom = 7fa617fedd38 thread\_stack 0x40000 /usr/sbin/mysqld(my\_print\_stacktrace+0x35)[0x8f97d5] /usr/sbin/mysqld(handle\_fatal\_signal+0x4b4)[0x6655c4] /lib64/libpthread.so.0(+0xf710)[0x7fa78c98e710] /usr/sbin/mysqld(\_Z11ull\_get\_keyPKhPmc+0x14)[0x5fb2b4] /usr/sbin/mysqld(my\_hash\_first\_from\_hash\_value+0x6b)[0x8e2c4b] /usr/sbin/mysqld(my\_hash\_search+0x11)[0x8e2e31] /usr/sbin/mysqld(\_ZN22Item\_func\_release\_lock7val\_intEv+0x10f)[0x60068f] /usr/sbin/mysqld(\_ZN4Item4sendEP8ProtocolP6String+0x1c4)[0x5b06d4] /usr/sbin/mysqld(\_ZN8Protocol19send\_result\_set\_rowEP4ListI4ItemE+0xc7)[0x65ef47] /usr/sbin/mysqld(\_ZN11select\_send9send\_dataER4ListI4ItemE+0x67)[0x6ae287] /usr/sbin/mysqld(\_ZN4JOIN4execEv+0x521)[0x6c9e81] /usr/sbin/mysqld(\_Z12mysql\_selectP3THDP10TABLE\_LISTjR4ListI4ItemEPS4\_P10SQL\_I\_ListI8st\_orderESB\_S7\_yP13select\_resultP18st\_select\_lex\_unitP13st\_select\_lex+0x250)[0x7120e0] /usr/sbin/mysqld(\_Z13handle\_selectP3THDP13select\_resultm+0x187)[0x712967] /usr/sbin/mysqld[0x6e836d] /usr/sbin/mysqld(\_Z21mysql\_execute\_commandP3THD+0x3cdb)[0x6ed50b] /usr/sbin/mysqld(\_ZN18Prepared\_statement7executeEP6Stringb+0x40e)[0x7002ae] /usr/sbin/mysqld(_ZN18Prepared\_statement12execute\_loopEP6StringbPhS2_+0xde)[0x7044ae] /usr/sbin/mysqld(\_Z22mysql\_sql\_stmt\_executeP3THD+0xbe)[0x70500e] /usr/sbin/mysqld(\_Z21mysql\_execute\_commandP3THD+0x1324)[0x6eab54] /usr/sbin/mysqld(\_Z11mysql\_parseP3THDPcjP12Parser\_state+0x658)[0x6f0338] /usr/sbin/mysqld[0x6f0491] /usr/sbin/mysqld(\_Z16dispatch\_command19enum\_server\_commandP3THDPcj+0x19d5)[0x6f2675] /usr/sbin/mysqld(\_Z10do\_commandP3THD+0x22b)[0x6f3b5b] /usr/sbin/mysqld(\_Z24do\_handle\_one\_connectionP3THD+0x17f)[0x6bc30f] /usr/sbin/mysqld(handle\_one\_connection+0x47)[0x6bc4f7] /usr/sbin/mysqld(pfs\_spawn\_thread+0x12a)[0xaf38ba] /lib64/libpthread.so.0(+0x79d1)[0x7fa78c9869d1] /lib64/libc.so.6(clone+0x6d)[0x7fa78ae8a8fd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7fa4dc004178): is an invalid pointer Connection ID (thread ID): 6222895 Status: NOT\_KILLED You may download the Percona XtraDB Cluster operations manual by visiting [http://www.percona.com/software/percona-xtradb-cluster/](http://www.percona.com/software/percona-xtradb-cluster/). You may find information in the manual which will help you identify the cause of the crash. 150608 12:05:00 mysqld\_safe Number of processes running now: 0 150608 12:05:00 mysqld\_safe WSREP: not restarting wsrep node automatically 150608 12:05:00 mysqld\_safe mysqld from pid file /var/lib/mysql/RR-Cluster-DB2.pid ended I dig the internet for a month but nothing can fix my error. Hope some here, in percona forum can help me. Thanks million times. ---
**Author:** ![Rossonero224](https://avatars.discourse-cdn.com/v4/letter/r/8491ac/32.png) [@Rossonero224](https://forums.percona.com/u/Rossonero224)\ **Post date:** [June 16, 2015, 9:32pm UTC](https://forums.percona.com/t/my-cluster-crash-randomly/4264/2 "2015-06-16T21:32:51Z")
hi, anyone can help? ---
**Author:** ![Rossonero224](https://avatars.discourse-cdn.com/v4/letter/r/8491ac/32.png) [@Rossonero224](https://forums.percona.com/u/Rossonero224)\ **Post date:** [June 18, 2015, 11:07pm UTC](https://forums.percona.com/t/my-cluster-crash-randomly/4264/3 "2015-06-18T23:07:23Z")
pleaseee helppppp