# Percona cluster - one server crashed - memory bug? **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [April 28, 2015, 5:19am UTC](https://forums.percona.com/t/percona-cluster-one-server-crashed-memory-bug/4192 "2015-04-28T05:19:20Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![francoisstark](https://avatars.discourse-cdn.com/v4/letter/f/a698b9/32.png) [@francoisstark](https://forums.percona.com/u/francoisstark)\ **Post date:** [April 28, 2015, 5:19am UTC](https://forums.percona.com/t/percona-cluster-one-server-crashed-memory-bug/4192/1 "2015-04-28T05:19:20Z")
I have three servers in a cluster - running Percona XtraDB 5.6.22-72.0-56 on ubuntu 12.04, xeon systems with ECC ram and raid 5 arrays, 32GB ram each. Unlikely to be a hardware issue. This is what the error log said: 18:19:44 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at [URL][https://bugs.launchpad.net/percona-xtradb-cluster[/URL]](https://bugs.launchpad.net/percona-xtradb-cluster%5B/URL%5D) key\_buffer\_size=8388608 read\_buffer\_size=131072 max\_used\_connections=37 max\_threads=153 thread\_count=19 connection\_count=2 It is possible that mysqld could use up to key\_buffer\_size + (read\_buffer\_size + sort\_buffer\_size)\*max\_threads = 69252 K bytes of memory Hope that’s ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2d18000990 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong… stack\_bottom = 7f30005e0a70 thread\_stack 0x40000 /usr/sbin/mysqld(my\_print\_stacktrace+0x2e)[0x8e811e] /usr/sbin/mysqld(handle\_fatal\_signal+0x392)[0x65ffa2] /lib/x86\_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f303d3e5cb0] /usr/lib/libgalera\_smm.so(_ZN6galera13Certification16purge_ for\_trx\_v3EPNS\_9TrxHandleE+0xa0)[0x7f302225a0f0] /usr/lib/libgalera\_smm.so(_ZN6galera13Certification16purge_ trxs\_upto\_Elb+0x158)[0x7f302225b8c8] /usr/lib/libgalera\_smm.so(\_ZN6galera13ReplicatorSMM18proces s\_commit\_cutEll+0x85)[0x7f3022288215] /usr/lib/libgalera\_smm.so(\_ZN6galera15GcsActionSource8dispa tchEPvRK10gcs\_actionRb+0x405)[0x7f3022269d75] /usr/lib/libgalera\_smm.so(\_ZN6galera15GcsActionSource7proce ssEPvRb+0x5e)[0x7f302226a8ee] /usr/lib/libgalera\_smm.so(_ZN6galera13ReplicatorSMM10async_ recvEPv+0x78)[0x7f302228f958] /usr/lib/libgalera\_smm.so(galera\_recv+0x1e)[0x7f30222a4c8e] /usr/sbin/mysqld[0x5a491c] /usr/sbin/mysqld(start\_wsrep\_THD+0x287)[0x58d247] /lib/x86\_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f303d3dde9a] /lib/x86\_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f303c8f88bd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 12 Status: NOT\_KILLED ---
**Author:** ![francoisstark](https://avatars.discourse-cdn.com/v4/letter/f/a698b9/32.png) [@francoisstark](https://forums.percona.com/u/francoisstark)\ **Post date:** [April 28, 2015, 7:13am UTC](https://forums.percona.com/t/percona-cluster-one-server-crashed-memory-bug/4192/2 "2015-04-28T07:13:40Z")
Trying to restart this server’s mysql the SST fails… I accidentally started it in bootstrap mode and it ran for about a minute by itself, causing some writes to the databases. Then I edited my.cnf and restarted mysql service, expecting a full SST but get this error: 2015-04-28 12:53:54 45962 [Note] WSREP: Running: 'wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘10.X.X.X’ --auth ‘sstuser:sdfgdfghdry56’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘45962’ ‘’ ’ WSREP\_SST: [INFO] Streaming with tar (20150428 12:53:55.080) WSREP\_SST: [INFO] Using socat as streamer (20150428 12:53:55.081) WSREP\_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20150428 12:53:55.110) 2015-04-28 12:53:55 45962 [Note] WSREP: Prepared SST request: xtrabackup|10.x.X.X:4444/xtrabackup\_sst 2015-04-28 12:53:55 45962 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2015-04-28 12:53:55 45962 [Note] WSREP: REPL Protocols: 5 (3, 1) 2015-04-28 12:53:55 45962 [Note] WSREP: Assign initial position for certification: 14184918, protocol version: 3 2015-04-28 12:53:55 45962 [Note] WSREP: Service thread queue flushed. 2015-04-28 12:53:55 45962 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.x.x.x:4568 2015-04-28 12:53:55 45962 [Note] WSREP: Node 1.0 (perc1) requested state transfer from ‘_any_’. Selected 0.0 (dxss3)(SYNCED) as donor. 2015-04-28 12:53:55 45962 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 14184918) 2015-04-28 12:53:55 45962 [Note] WSREP: Requesting state transfer: success, donor: 0 2015-04-28 12:53:55 45962 [Note] WSREP: 0.0 (dxss3): State transfer to 1.0 (dxss1) complete. 2015-04-28 12:53:55 45962 [Note] WSREP: Member 0 (dxss3) synced with group. WSREP\_SST: [INFO] xtrabackup\_ist received from donor: Running IST (20150428 12:53:55.425) WSREP\_SST: [INFO] Total time on joiner: 0 seconds (20150428 12:53:55.427) WSREP\_SST: [INFO] Removing the sst\_in\_progress file (20150428 12:53:55.428) 2015-04-28 12:53:55 45962 [Note] WSREP: SST complete, seqno: 14119148 2015-04-28 12:53:55 45962 [Note] Plugin ‘FEDERATED’ is disabled. 2015-04-28 12:53:55 45962 [Note] InnoDB: Using atomics to ref count buffer pool pages 2015-04-28 12:53:55 45962 [Note] InnoDB: The InnoDB memory heap is disabled 2015-04-28 12:53:55 45962 [Note] InnoDB: Mutexes and rw\_locks use GCC atomic builtins 2015-04-28 12:53:55 45962 [Note] InnoDB: Memory barrier is not used 2015-04-28 12:53:55 45962 [Note] InnoDB: Compressed tables use zlib 1.2.3.4 2015-04-28 12:53:55 45962 [Note] InnoDB: Using Linux native AIO 2015-04-28 12:53:55 45962 [Note] InnoDB: Using CPU crc32 instructions 2015-04-28 12:53:55 45962 [Note] InnoDB: Initializing buffer pool, size = 10.0G 2015-04-28 12:53:55 45962 [Note] InnoDB: Completed initialization of buffer pool 2015-04-28 12:53:55 45962 [Note] InnoDB: Highest supported file format is Barracuda. 2015-04-28 12:53:56 45962 [Note] InnoDB: 128 rollback segment(s) are active. 2015-04-28 12:53:56 45962 [Note] InnoDB: Waiting for purge to start 2015-04-28 12:53:56 45962 [Note] InnoDB: Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 5.6.22-72.0 started; log sequence number 30463102736 2015-04-28 12:53:56 45962 [Note] RSA private key file not found: /var/lib/mysql//private\_key.pem. Some authentication plugins will not work. 2015-04-28 12:53:56 45962 [Note] RSA public key file not found: /var/lib/mysql//public\_key.pem. Some authentication plugins will not work. 2015-04-28 12:53:56 45962 [Note] Server hostname (bind-address): ‘\*’; port: 3306 2015-04-28 12:53:56 45962 [Note] IPv6 is available. 2015-04-28 12:53:56 45962 [Note] - ‘::’ resolves to ‘::’; 2015-04-28 12:53:56 45962 [Note] Server socket created on IP: ‘::’. 2015-04-28 12:53:56 45962 [Note] Event Scheduler: Loaded 0 events 2015-04-28 12:53:56 45962 [Note] WSREP: Signalling provider to continue. 2015-04-28 12:53:56 45962 [Note] WSREP: inited wsrep sidno 1 2015-04-28 12:53:56 45962 [Note] WSREP: SST received: 5b18cbf7-sdfgsdfgsdfg8379-11e3-92395df271:14119148 2015-04-28 12:53:56 45962 [Note] WSREP: Receiving IST: 65770 writesets, seqnos 14119148-14184918 2015-04-28 12:53:56 45962 [Note] /usr/sbin/mysqld: ready for connections. Version: ‘5.6.22-72.0-56’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel72.0, Revision 978, WSREP version 25.8, wsrep\_25.8.r4150 2015-04-28 12:53:56 45962 [ERROR] Slave SQL: Could not execute Delete\_rows event on table dxss.codesecrets; Can’t find record in ‘codesecrets’, Error\_code: 1032; handler error HA\_ERR\_KEY\_NOT\_FOUND; the event’s master log FIRST, end\_log\_pos 213, Error\_code: 1032 2015-04-28 12:53:56 45962 [Warning] WSREP: RBR event 3 Delete\_rows apply warning: 120, 14119181 2015-04-28 12:53:56 45962 [Warning] WSREP: Failed to apply app buffer: seqno: 14119181, status: 1 at galera/src/trx\_handle.cpp:apply():340 Retrying 2th time 2015-04-28 12:53:56 45962 [ERROR] Slave SQL: Could not execute Delete\_rows event on table dxss.codesecrets; Can’t find record in ‘codesecrets’, Error\_code: 1032; handler error HA\_ERR\_KEY\_NOT\_FOUND; the event’s master log FIRST, end\_log\_pos 213, Error\_code: 1032 2015-04-28 12:53:56 45962 [Warning] WSREP: RBR event 3 Delete\_rows apply warning: 120, 14119181 2015-04-28 12:53:56 45962 [Warning] WSREP: Failed to apply app buffer: seqno: 14119181, status: 1 at galera/src/trx\_handle.cpp:apply():340 Retrying 3th time 2015-04-28 12:53:56 45962 [ERROR] Slave SQL: Could not execute Delete\_rows event on table dxss.codesecrets; Can’t find record in ‘codesecrets’, Error\_code: 1032; handler error HA\_ERR\_KEY\_NOT\_FOUND; the event’s master log FIRST, end\_log\_pos 213, Error\_code: 1032 2015-04-28 12:53:56 45962 [Warning] WSREP: RBR event 3 Delete\_rows apply warning: 120, 14119181 2015-04-28 12:53:56 45962 [Warning] WSREP: Failed to apply app buffer: seqno: 14119181, status: 1 at galera/src/trx\_handle.cpp:apply():340 Retrying 4th time 2015-04-28 12:53:56 45962 [ERROR] Slave SQL: Could not execute Delete\_rows event on table dxss.codesecrets; Can’t find record in ‘codesecrets’, Error\_code: 1032; handler error HA\_ERR\_KEY\_NOT\_FOUND; the event’s master log FIRST, end\_log\_pos 213, Error\_code: 1032 2015-04-28 12:53:56 45962 [Warning] WSREP: RBR event 3 Delete\_rows apply warning: 120, 14119181 2015-04-28 12:53:56 45962 [ERROR] WSREP: receiving IST failed, node restart required: Failed to apply trx 14119181 4 times 2015-04-28 12:53:56 45962 [Note] WSREP: Closing send monitor… It seems I must delete the local innodb data files and somehow force a complete SST. The other two systems in the cluster are running fine. Thanks Francois ---
**Author:** ![francoisstark](https://avatars.discourse-cdn.com/v4/letter/f/a698b9/32.png) [@francoisstark](https://forums.percona.com/u/francoisstark)\ **Post date:** [April 29, 2015, 2:54am UTC](https://forums.percona.com/t/percona-cluster-one-server-crashed-memory-bug/4192/3 "2015-04-29T02:54:26Z")
Still trying to join the cluster. I deleted all GRA and grastate files from /var/lib/mysql. This forces a SST in stead of a IST. Deleted the related database directories as well and tried to join. The joiner gives me this error: WSREP\_SST: [ERROR] xtrabackup process ended without creating ‘/var/lib/mysql//xtrabackup\_galera\_info’ It created three other files in /var/lib/mysql: xtrabackup\_checkpoints xtrabackup\_info xtrabackup\_logfile aaaaaaand problem found: The failed system had this line in my.cnf: wsrep\_sst\_method=xtrabackup While in the other two running systems, this line was commented out. So I commented out the line, deleted the grastate file again and reqstarted mysql service. It forced a full SST, and from what I can see in the logfile, the default sst method is now set to xtrabackup-v2 Problem solved. For now. The original crash still baffles.