# XtraDB Cluster node failure **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [November 7, 2012, 10:53am UTC](https://forums.percona.com/t/xtradb-cluster-node-failure/2498 "2012-11-07T10:53:07Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![rdab100](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rdab100/32/8_2.png) [@rdab100](https://forums.percona.com/u/rdab100)\ **Post date:** [November 7, 2012, 10:53am UTC](https://forums.percona.com/t/xtradb-cluster-node-failure/2498/1 "2012-11-07T10:53:07Z")
How do you diagnose what caused an XtraDB cluster node to fail? What are ALL the log files etc that I should be looking at? Thanks. Dom A node (6 node cluster, split three members on two sites (1000MB or more between sites) failed at Nov 2 00:55:14 and failed to restart, several times, mysql\_safe seems to have captured the failure but failed to restart it. We had to do a HARD OS restart and a manual mysql restart as you can see from the mysql error log. There was no user activity on the db at the time of the failure. We lost a second node 6 hours after the first and a third about 5 hours later, again nothing seems to explain it. Environment: scientific 6.2 64bit vmware 5 2vcpu 8GB ram 4GB db 5.9GB innoDB buffer Version 5.5.24-23.6 my.cnf file: [client] port = 3306 socket = /var/lib/mysql/mysql.sock [mysqld] skip-name-resolv character-set-server=utf8 collation-server=utf8\_bin datadir=/var/lib/mysql user=mysql expire\_logs\_days=3 port = 3306 socket = /var/lib/mysql/mysql.sock back\_log = 50 max\_connections = 250 max\_connect\_errors = 10 table\_open\_cache = 2048 max\_allowed\_packet = 32M thread\_cache\_size = 8 query\_cache\_size = 0 query\_cache\_type = 0 ft\_min\_word\_len = 4 default-storage-engine = InnoDB thread\_stack = 192K transaction\_isolation = REPEATABLE-READ tmp\_table\_size = 64M log-bin=mysql-bin binlog\_format=ROW log\_slave\_updates slow\_query\_log=1 slow\_query\_log\_file=/var/log/mysql/slow.log long\_query\_time = 2 general\_log=0 general\_log\_file=/var/log/mysql/general.log log-error=/var/log/mysql/error.log server-id=183371624 key\_buffer\_size = 32M bulk\_insert\_buffer\_size = 64M myisam\_sort\_buffer\_size = 128M myisam\_max\_sort\_file\_size = 10G myisam\_repair\_threads = 1 myisam\_recover innodb\_additional\_mem\_pool\_size = 16M innodb\_buffer\_pool\_size = 6000M innodb\_data\_file\_path = ibdata1:10M:autoextend innodb\_thread\_concurrency = 0 innodb\_flush\_log\_at\_trx\_commit = 0 innodb\_doublewrite=1 innodb\_log\_buffer\_size = 8M innodb\_log\_file\_size = 384M innodb\_log\_files\_in\_group = 2 innodb\_max\_dirty\_pages\_pct = 90 innodb\_file\_per\_table innodb\_autoinc\_lock\_mode=2 innodb\_locks\_unsafe\_for\_binlog=1 wait\_timeout=300 wsrep\_provider=/usr/lib64/libgalera\_smm.so wsrep\_provider\_options = “gcache.size=4G; gmcast.listen\_addr=tcp://0.0.0.0:4010;evs.send\_window=512; evs.user\_send\_window=512;” wsrep\_slave\_threads=8 wsrep\_cluster\_name=lptlive wsrep\_sst\_method=xtrabackup wsrep\_sst\_auth=replic:5vaeNeIiTYBRU wsrep\_notify\_cmd=/usr/bin/wsrep\_notify.sh [xtrabackup] target-dir = /backup [mysqldump] quick max\_allowed\_packet = 32M [mysql] no-auto-rehash [myisamchk] key\_buffer\_size = 512M sort\_buffer\_size = 512M read\_buffer = 8M write\_buffer = 8M [mysqlhotcopy] interactive-timeout [mysqld\_safe] open-files-limit = 8192 wsrep\_urls= gcomm://192.168.100.101:4010,gcomm://192.168.100.102:4010,gc omm://192.168.100.103:4010,gcomm://192.168.101.104:4010,gcom m://192.168.101.105:4010,gcomm://192.168.101.106:4010,gcomm: // I have checked the mysql error log and it says: 121102 00:55:14 mysqld\_safe Number of processes running now: 0 121102 00:55:14 mysqld\_safe mysqld restarted 121102 0:55:15 [Note] Flashcache bypass: disabled 121102 0:55:15 [Note] Flashcache setup error is : open flash device failed 121102 0:55:15 [Note] WSREP: Read nil XID from storage engines, skipping position init 121102 0:55:15 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 121102 0:55:15 [Note] WSREP: wsrep\_load(): Galera 2.1(r113) by Codership Oy \<[info@codership.com](mailto:info@codership.com)\> loaded succesfully. 121102 0:55:15 [Note] WSREP: Found saved state: 72de1b90-022d-11e2-0800-484cdf256d28:-1 121102 0:55:15 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’. 121102 0:55:15 [Note] WSREP: Passing config to GCS: base\_host = 192.168.100.103; evs.send\_window = 512; evs.user\_send\_window = 512; gcache.dir = /var/lib/mysql/; gcache.keep\_pages \_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 4G; gcs.fc\_debug = 0; gcs.fc\_factor = 0.5; gcs.fc\_limit = 16; gcs.fc\_master\_slave = NO; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = NO; g mcast.listen\_addr = tcp://0.0.0.0:4010; replicator.causal\_read\_timeout = PT30S; replicator.commit\_order = 3 121102 0:55:15 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 121102 0:55:15 [Note] WSREP: wsrep\_sst\_grab() 121102 0:55:15 [Note] WSREP: Start replication 121102 0:55:15 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 121102 0:55:15 [Note] WSREP: protonet asio version 0 121102 0:55:15 [Note] WSREP: backend: asio 121102 0:55:15 [Note] WSREP: GMCast version 0 121102 0:55:15 [Note] WSREP: (f6ba0383-2487-11e2-0800-0e4230df1ac6, ‘tcp://0.0.0.0:4010’) listening at tcp://0.0.0.0:4010 121102 0:55:15 [Note] WSREP: (f6ba0383-2487-11e2-0800-0e4230df1ac6, ‘tcp://0.0.0.0:4010’) multicast: , ttl: 1 121102 0:55:15 [Note] WSREP: EVS version 0 121102 0:55:15 [Note] WSREP: PC version 0 121102 0:55:15 [Note] WSREP: gcomm: connecting to group ‘lptlive’, peer ‘192.168.100.101:4010’ 121102 0:55:15 [Note] WSREP: (f6ba0383-2487-11e2-0800-0e4230df1ac6, ‘tcp://0.0.0.0:4010’) turning message relay requesting on, nonlive peers: tcp://192.168.100.57:4010 tcp://192.168.101.105:4010 tcp://192.168.101.104:4010 tcp://192.168.100.102:4010 tcp://192.168.100.139:4010 121102 0:55:15 [Note] WSREP: (f6ba0383-2487-11e2-0800-0e4230df1ac6, ‘tcp://0.0.0.0:4010’) cleaning up duplicate 0x28b0660 after established 0x289c490 121102 0:55:15 [Note] WSREP: (f6ba0383-2487-11e2-0800-0e4230df1ac6, ‘tcp://0.0.0.0:4010’) turning message relay requesting off 121102 0:55:15 [Note] WSREP: (f6ba0383-2487-11e2-0800-0e4230df1ac6, ‘tcp://0.0.0.0:4010’) cleaning up duplicate 0x28ae630 after established 0x28dc140 121102 0:55:15 [Note] WSREP: (f6ba0383-2487-11e2-0800-0e4230df1ac6, ‘tcp://0.0.0.0:4010’) cleaning up duplicate 0x28b0000 after established 0x28db420 121102 0:55:19 [Note] WSREP: declaring 72f89f95-0729-11e2-0800-e80c813e3134 stable 121102 0:55:19 [Note] WSREP: declaring a8511994-190a-11e2-0800-580d0a80208f stable 121102 0:55:19 [Note] WSREP: declaring a882f2cb-190a-11e2-0800-00171a908230 stable 121102 0:55:19 [Note] WSREP: declaring a8b2f666-190a-11e2-0800-56702e7cb352 stable 121102 0:55:19 [Note] WSREP: declaring a8beb15d-190a-11e2-0800-81fcb2686439 stable 121102 0:55:19 [Note] WSREP: declaring a8c0cfb1-190a-11e2-0800-7533cf9b72c7 stable 121102 0:55:19 [Note] WSREP: view(view\_id(PRIM,72f89f95-0729-11e2-0800-e80c813e3134,117) memb { 72f89f95-0729-11e2-0800-e80c813e3134, a8511994-190a-11e2-0800-580d0a80208f, a882f2cb-190a-11e2-0800-00171a908230, a8b2f666-190a-11e2-0800-56702e7cb352, a8beb15d-190a-11e2-0800-81fcb2686439, a8c0cfb1-190a-11e2-0800-7533cf9b72c7, f6ba0383-2487-11e2-0800-0e4230df1ac6, } joined { } left { } partitioned { aae97de1-1831-11e2-0800-ee58636f1b39, }) 121102 0:55:19 [Note] WSREP: gcomm: connected 121102 0:55:19 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 121102 0:55:19 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) 121102 0:55:19 [Note] WSREP: Opened channel ‘lptlive’ 121102 0:55:19 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 6, memb\_num = 7 121102 0:55:19 [Note] WSREP: Waiting for SST to complete. 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: sent state msg: f95d07c6-2487-11e2-0800-d49782a4e38d 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: got state msg: f95d07c6-2487-11e2-0800-d49782a4e38d from 0 (garb) 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: got state msg: f95d07c6-2487-11e2-0800-d49782a4e38d from 1 (mysqldbdrv03) 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: got state msg: f95d07c6-2487-11e2-0800-d49782a4e38d from 2 (mysqldbdrv01) 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: got state msg: f95d07c6-2487-11e2-0800-d49782a4e38d from 4 (mysqldb03) 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: got state msg: f95d07c6-2487-11e2-0800-d49782a4e38d from 5 (mysqldb02) 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: got state msg: f95d07c6-2487-11e2-0800-d49782a4e38d from 3 (mysqldbdrv02) 121102 0:55:19 [Note] WSREP: STATE EXCHANGE: got state msg: f95d07c6-2487-11e2-0800-d49782a4e38d from 6 (mysqldb01) 121102 0:55:19 [Note] WSREP: ‘garb’ demoted SYNCED-\>PRIMARY due to gap in history: 8585650 - 8585807 121102 0:55:19 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf\_id = 112, members = 5/7 (joined/total), act\_id = 8585807, last\_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = 72de1b90-022d-11e2-0800-484cdf256d28 121102 0:55:19 [Note] WSREP: Flow-control interval: [21, 42] 121102 0:55:19 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 8585807) 121102 0:55:19 [Note] WSREP: State transfer required: Group state: 72de1b90-022d-11e2-0800-484cdf256d28:8585807 Local state: 72de1b90-022d-11e2-0800-484cdf256d28:-1 121102 0:55:19 [Note] WSREP: New cluster view: global state: 72de1b90-022d-11e2-0800-484cdf256d28:8585807, view# 113: Primary, number of nodes: 7, my index: 6, protocol version 2 121102 0:55:19 [Warning] WSREP: Gap in state sequence. Need state transfer. 121102 0:55:19 [Note] WSREP: Node 0 (garb) requested state transfer from ‘_any_’. Selected 1 (mysqldbdrv03)(SYNCED) as donor. 121102 0:55:19 [Note] WSREP: 0 (garb): State transfer from 1 (mysqldbdrv03) complete. 121102 0:55:19 [Note] WSREP: Member 0 (garb) synced with group. 121102 0:55:20 [Warning] WSREP: 1 (mysqldbdrv03): State transfer to 0 (garb) failed: -1 (Operation not permitted) 121102 0:55:20 [Note] WSREP: Member 1 (mysqldbdrv03) synced with group. 121102 0:55:21 [Note] WSREP: Running: ‘wsrep\_sst\_xtrabackup ‘joiner’ ‘192.168.100.103’ ‘replic:5vaeNeIiTYBRU’ ‘/var/lib/mysql/’ ‘/etc/my.cnf’ ‘4981’ 2\>sst.err’ 121102 0:55:21 [Note] WSREP: Prepared SST request: xtrabackup|192.168.100.103:4444/xtrabackup\_sst ERROR 2003 (HY000): Can’t connect to MySQL server on ‘127.0.0.1’ (111) sed: couldn’t flush stdout: Broken pipe 121102 0:55:21 [ERROR] WSREP: Process completed with error: /usr/bin/wsrep\_notify.sh --status Joiner --uuid 72de1b90-022d-11e2-0800-484cdf256d28 --primary yes --index 6 --members 72f89f95-0729-11e2-0800-e80c813e3134/garb/,a8511994-190a-11e 2-0800-580d0a80208f/mysqldbdrv03/192.168.100. 121102 0:55:21 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/bin/wsrep\_notify.sh --status Joiner --uuid 72de1b90-022d-11e2-0800-484cdf256d28 --primary yes --index 6 --members 72f89f95-0729-11e2-0800-e80c813e3134/garb/,a8511994-190a-11e 2-0800-580d0a802 121102 0:55:21 [Note] WSREP: Assign initial position for certification: 8585807, protocol version: 2 121102 0:55:21 [Note] WSREP: Prepared IST receiver, listening at: tcp://192.168.100.103:4011 121102 0:55:21 [Note] WSREP: Node 6 (mysqldb01) requested state transfer from ‘_any_’. Selected 1 (mysqldbdrv03)(SYNCED) as donor. 121102 0:55:21 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 8585830) 121102 0:55:21 [Note] WSREP: Requesting state transfer: success, donor: 1 121102 0:56:57 [Note] WSREP: 1 (mysqldbdrv03): State transfer to 6 (mysqldb01) complete. 121102 0:56:57 [Note] WSREP: Member 1 (mysqldbdrv03) synced with group. 121102 0:57:17 [Note] WSREP: SST complete, seqno: 8585950 121102 0:57:17 [Note] Plugin ‘FEDERATED’ is disabled. 121102 0:57:17 InnoDB: The InnoDB memory heap is disabled 121102 0:57:17 InnoDB: Mutexes and rw\_locks use GCC atomic builtins 121102 0:57:17 InnoDB: Compressed tables use zlib 1.2.3 121102 0:57:17 InnoDB: Using Linux native AIO 121102 0:57:17 InnoDB: Initializing buffer pool, size = 5.9G InnoDB: mmap(6463488000 bytes) failed; errno 12 121102 0:57:17 InnoDB: Completed initialization of buffer pool 121102 0:57:17 InnoDB: Fatal error: cannot allocate memory for the buffer pool 121102 0:57:17 [ERROR] Plugin ‘InnoDB’ init function returned error. 121102 0:57:17 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed. 121102 0:57:17 [ERROR] Unknown/unsupported storage engine: InnoDB 121102 0:57:17 [ERROR] Aborting 121102 0:57:19 [Note] WSREP: Closing send monitor… 121102 0:57:19 [Note] WSREP: Closed send monitor. 121102 0:57:19 [Note] WSREP: gcomm: terminating thread 121102 0:57:19 [Note] WSREP: gcomm: joining thread 121102 0:57:19 [Note] WSREP: gcomm: closing backend 121102 0:57:19 [Note] WSREP: view(view\_id(NON\_PRIM,72f89f95-0729-11e2-0800-e80c813e3134,1 17) memb { f6ba0383-2487-11e2-0800-0e4230df1ac6, } joined { } left { } partitioned { 72f89f95-0729-11e2-0800-e80c813e3134, a8511994-190a-11e2-0800-580d0a80208f, a882f2cb-190a-11e2-0800-00171a908230, a8b2f666-190a-11e2-0800-56702e7cb352, a8beb15d-190a-11e2-0800-81fcb2686439, a8c0cfb1-190a-11e2-0800-7533cf9b72c7, }) 121102 0:57:19 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1 121102 0:57:19 [Note] WSREP: view((empty)) 121102 0:57:19 [Note] WSREP: gcomm: closed 121102 0:57:19 [Note] WSREP: Flow-control interval: [8, 16] 121102 0:57:19 [Note] WSREP: Received NON-PRIMARY. 121102 0:57:19 [Note] WSREP: Shifting JOINER → OPEN (TO: 8586007) 121102 0:57:19 [Note] WSREP: Received self-leave message. 121102 0:57:19 [Note] WSREP: Flow-control interval: [0, 0] 121102 0:57:19 [Note] WSREP: Received SELF-LEAVE. Closing connection. 121102 0:57:19 [Note] WSREP: Shifting OPEN → CLOSED (TO: 8586007) 121102 0:57:19 [Note] WSREP: RECV thread exiting 0: Success 121102 0:57:19 [Note] WSREP: recv\_thread() joined. 121102 0:57:19 [Note] WSREP: Closing slave action queue. 121102 0:57:19 [Note] WSREP: Service disconnected. 121102 0:57:19 [Note] WSREP: rollbacker thread exiting 121102 0:57:20 [Note] WSREP: Some threads may fail to exit. 121102 0:57:20 [Note] /usr/sbin/mysqld: Shutdown complete Error in my\_thread\_global\_end(): 2 threads didn’t exit 121102 00:57:25 mysqld\_safe mysqld from pid file /var/lib/mysql/mysqldb01.pid ended 121102 13:02:33 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql 121102 13:02:33 [Note] Flashcache bypass: disabled 121102 13:02:33 [Note] Flashcache setup error is : open flash device failed 121102 13:02:33 [Note] WSREP: Read nil XID from storage engines, skipping position init 121102 13:02:33 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 121102 13:02:33 [Note] WSREP: wsrep\_load(): Galera 2.1(r113) by Codership Oy \<[info@codership.com](mailto:info@codership.com)\> loaded succesfully. 121102 13:02:33 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 121102 13:02:33 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’. 121102 13:02:33 [Note] WSREP: Passing config to GCS: base\_host = 192.168.100.103; evs.send\_window = 512; evs.user\_send\_window = 512; gcache.dir = /var/lib/mysql/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 4G; gcs.fc\_debug = 0; gcs.fc\_factor = 0.5; gcs.fc\_limit = 16; gcs.fc\_master\_slave = NO; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = NO; gmcast.listen\_addr = tcp://0.0.0.0:4010; replicator.causal\_read\_timeout = PT30S; replicator.commit\_order = 3 121102 13:02:33 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 121102 13:02:33 [Note] WSREP: wsrep\_sst\_grab() 121102 13:02:33 [Note] WSREP: Start replication 121102 13:02:33 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 121102 13:02:33 [Note] WSREP: protonet asio version 0 121102 13:02:33 [Note] WSREP: backend: asio 121102 13:02:33 [Note] WSREP: GMCast version 0 121102 13:02:33 [Note] WSREP: (914eeace-24ed-11e2-0800-f3b6d7ea9afa, ‘tcp://0.0.0.0:4010’) listening at tcp://0.0.0.0:4010 121102 13:02:33 [Note] WSREP: (914eeace-24ed-11e2-0800-f3b6d7ea9afa, ‘tcp://0.0.0.0:4010’) multicast: , ttl: 1 121102 13:02:33 [Note] WSREP: EVS version 0 121102 13:02:33 [Note] WSREP: PC version 0 121102 13:02:33 [Note] WSREP: gcomm: connecting to group ‘lptlive’, peer ‘192.168.100.101:4010’ 121102 13:02:33 [Note] WSREP: (914eeace-24ed-11e2-0800-f3b6d7ea9afa, ‘tcp://0.0.0.0:4010’) turning message relay requesting on, nonlive peers: tcp://192.168.100.57:4010 tcp://192.168.101.105:4010 tcp://192.168.101.104:4010 tcp://192.168.100.139:4010 121102 13:02:33 [Note] WSREP: (914eeace-24ed-11e2-0800-f3b6d7ea9afa, ‘tcp://0.0.0.0:4010’) cleaning up duplicate 0x13d9290 after established 0x13d05e0 121102 13:02:33 [Note] WSREP: (914eeace-24ed-11e2-0800-f3b6d7ea9afa, ‘tcp://0.0.0.0:4010’) cleaning up duplicate 0x13f3110 after established 0x13ea570 121102 13:02:34 [Note] WSREP: (914eeace-24ed-11e2-0800-f3b6d7ea9afa, ‘tcp://0.0.0.0:4010’) turning message relay requesting off 121102 13:02:34 [Note] WSREP: declaring 72f89f95-0729-11e2-0800-e80c813e3134 stable 121102 13:02:34 [Note] WSREP: declaring a8511994-190a-11e2-0800-580d0a80208f stable 121102 13:02:34 [Note] WSREP: declaring a882f2cb-190a-11e2-0800-00171a908230 stable 121102 13:02:34 [Note] WSREP: declaring a8b2f666-190a-11e2-0800-56702e7cb352 stable 121102 13:02:34 [Note] WSREP: declaring a8beb15d-190a-11e2-0800-81fcb2686439 stable 121102 13:02:34 [Note] WSREP: view(view\_id(PRIM,72f89f95-0729-11e2-0800-e80c813e3134,121) memb { 72f89f95-0729-11e2-0800-e80c813e3134, 914eeace-24ed-11e2-0800-f3b6d7ea9afa, a8511994-190a-11e2-0800-580d0a80208f, a882f2cb-190a-11e2-0800-00171a908230, a8b2f666-190a-11e2-0800-56702e7cb352, a8beb15d-190a-11e2-0800-81fcb2686439, } joined { } left { } partitioned { }) 121102 13:02:34 [Note] WSREP: gcomm: connected 121102 13:02:34 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 121102 13:02:34 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) 121102 13:02:34 [Note] WSREP: Opened channel ‘lptlive’ 121102 13:02:34 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 1, memb\_num = 6 121102 13:02:34 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 121102 13:02:34 [Note] WSREP: Waiting for SST to complete. 121102 13:02:34 [Note] WSREP: STATE EXCHANGE: sent state msg: 917d7c9d-24ed-11e2-0800-7b9a2dd6372a 121102 13:02:34 [Note] WSREP: STATE EXCHANGE: got state msg: 917d7c9d-24ed-11e2-0800-7b9a2dd6372a from 0 (garb) 121102 13:02:34 [Note] WSREP: STATE EXCHANGE: got state msg: 917d7c9d-24ed-11e2-0800-7b9a2dd6372a from 2 (mysqldbdrv03) 121102 13:02:34 [Note] WSREP: STATE EXCHANGE: got state msg: 917d7c9d-24ed-11e2-0800-7b9a2dd6372a from 3 (mysqldbdrv01) 121102 13:02:34 [Note] WSREP: STATE EXCHANGE: got state msg: 917d7c9d-24ed-11e2-0800-7b9a2dd6372a from 4 (mysqldbdrv02) 121102 13:02:34 [Note] WSREP: STATE EXCHANGE: got state msg: 917d7c9d-24ed-11e2-0800-7b9a2dd6372a from 5 (mysqldb03) 121102 13:02:34 [Note] WSREP: STATE EXCHANGE: got state msg: 917d7c9d-24ed-11e2-0800-7b9a2dd6372a from 1 (mysqldb01) 121102 13:02:34 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf\_id = 116, members = 5/6 (joined/total), act\_id = 8643168, last\_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = 72de1b90-022d-11e2-0800-484cdf256d28 121102 13:02:34 [Note] WSREP: Flow-control interval: [20, 39] 121102 13:02:34 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 8643168) 121102 13:02:34 [Note] WSREP: State transfer required: Group state: 72de1b90-022d-11e2-0800-484cdf256d28:8643168 Local state: 00000000-0000-0000-0000-000000000000:-1 121102 13:02:34 [Note] WSREP: New cluster view: global state: 72de1b90-022d-11e2-0800-484cdf256d28:8643168, view# 117: Primary, number of nodes: 6, my index: 1, protocol version 2 121102 13:02:34 [Warning] WSREP: Gap in state sequence. Need state transfer. 121102 13:02:36 [Note] WSREP: Running: ‘wsrep\_sst\_xtrabackup ‘joiner’ ‘192.168.100.103’ ‘replic:5vaeNeIiTYBRU’ ‘/var/lib/mysql/’ ‘/etc/my.cnf’ ‘22659’ 2\>sst.err’ 121102 13:02:36 [Note] WSREP: Prepared SST request: xtrabackup|192.168.100.103:4444/xtrabackup\_sst ERROR 2003 (HY000): Can’t connect to MySQL server on ‘127.0.0.1’ (111) sed: couldn’t flush stdout: Broken pipe 121102 13:02:36 [ERROR] WSREP: Process completed with error: /usr/bin/wsrep\_notify.sh --status Joiner --uuid 72de1b90-022d-11e2-0800-484cdf256d28 --primary yes --index 1 --members 72f89f95-0729-11e2-0800-e80c813e3134/garb/,914eeace-24ed-11e 2-0800-f3b6d7ea9afa/mysqldb01/10.238. 121102 13:02:36 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/bin/wsrep\_notify.sh --status Joiner --uuid 72de1b90-022d-11e2-0800-484cdf256d28 --primary yes --index 1 --members 72f89f95-0729-11e2-0800-e80c813e3134/garb/,914eeace-24ed-11e 2-0800-f3b6d7ea9 121102 13:02:36 [Note] WSREP: Assign initial position for certification: 8643168, protocol version: 2 121102 13:02:36 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (72de1b90-022d-11e2-0800-484cdf256d28): 1 (Operation not permitted) at galera/src/replicator\_str.cpp:prepare\_for\_IST():439. IST will be unavailable. 121102 13:02:36 [Note] WSREP: Node 1 (mysqldb01) requested state transfer from ‘_any_’. Selected 2 (mysqldbdrv03)(SYNCED) as donor. 121102 13:02:36 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 8643172) 121102 13:02:36 [Note] WSREP: Requesting state transfer: success, donor: 2 121102 13:04:09 [Note] WSREP: 2 (mysqldbdrv03): State transfer to 1 (mysqldb01) complete. 121102 13:04:09 [Note] WSREP: Member 2 (mysqldbdrv03) synced with group. 121102 13:04:32 [Note] WSREP: SST complete, seqno: 8643297 121102 13:04:32 [Note] Plugin ‘FEDERATED’ is disabled. 121102 13:04:32 InnoDB: The InnoDB memory heap is disabled 121102 13:04:32 InnoDB: Mutexes and rw\_locks use GCC atomic builtins 121102 13:04:32 InnoDB: Compressed tables use zlib 1.2.3 121102 13:04:32 InnoDB: Using Linux native AIO 121102 13:04:32 InnoDB: Initializing buffer pool, size = 5.9G InnoDB: mmap(6463488000 bytes) failed; errno 12 121102 13:04:32 InnoDB: Completed initialization of buffer pool 121102 13:04:32 InnoDB: Fatal error: cannot allocate memory for the buffer pool 121102 13:04:32 [ERROR] Plugin ‘InnoDB’ init function returned error. 121102 13:04:32 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed. 121102 13:04:32 [ERROR] Unknown/unsupported storage engine: InnoDB 121102 13:04:32 [ERROR] Aborting 121102 13:04:34 [Note] WSREP: Closing send monitor… 121102 13:04:34 [Note] WSREP: Closed send monitor. 121102 13:04:34 [Note] WSREP: gcomm: terminating thread 121102 13:04:34 [Note] WSREP: gcomm: joining thread 121102 13:04:34 [Note] WSREP: gcomm: closing backend 121102 13:04:34 [Note] WSREP: view(view\_id(NON\_PRIM,72f89f95-0729-11e2-0800-e80c813e3134,1 21) memb { 914eeace-24ed-11e2-0800-f3b6d7ea9afa, } joined { } left { } partitioned { 72f89f95-0729-11e2-0800-e80c813e3134, a8511994-190a-11e2-0800-580d0a80208f, a882f2cb-190a-11e2-0800-00171a908230, a8b2f666-190a-11e2-0800-56702e7cb352, a8beb15d-190a-11e2-0800-81fcb2686439, }) 121102 13:04:34 [Note] WSREP: view((empty)) 121102 13:04:34 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1 121102 13:04:34 [Note] WSREP: gcomm: closed 121102 13:04:34 [Note] WSREP: Flow-control interval: [8, 16] 121102 13:04:34 [Note] WSREP: Received NON-PRIMARY. 121102 13:04:34 [Note] WSREP: Shifting JOINER → OPEN (TO: 8643326) 121102 13:04:34 [Note] WSREP: Received self-leave message. 121102 13:04:34 [Note] WSREP: Flow-control interval: [0, 0] 121102 13:04:34 [Note] WSREP: Received SELF-LEAVE. Closing connection. 121102 13:04:34 [Note] WSREP: Shifting OPEN → CLOSED (TO: 8643326) 121102 13:04:34 [Note] WSREP: RECV thread exiting 0: Success 121102 13:04:34 [Note] WSREP: recv\_thread() joined. 121102 13:04:34 [Note] WSREP: Closing slave action queue. 121102 13:04:34 [Note] WSREP: Service disconnected. 121102 13:04:34 [Note] WSREP: rollbacker thread exiting 121102 13:04:35 [Note] WSREP: Some threads may fail to exit. 121102 13:04:35 [Note] /usr/sbin/mysqld: Shutdown complete Error in my\_thread\_global\_end(): 2 threads didn’t exit 121102 13:04:40 mysqld\_safe mysqld from pid file /var/lib/mysql/mysqldb01.pid ended 121102 13:11:44 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql 121102 13:11:45 [Note] Flashcache bypass: disabled 121102 13:11:45 [Note] Flashcache setup error is : open flash device failed 121102 13:11:45 [Note] WSREP: Read nil XID from storage engines, skipping position init 121102 13:11:45 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 121102 13:11:45 [Note] WSREP: wsrep\_load(): Galera 2.1(r113) by Codership Oy \<[info@codership.com](mailto:info@codership.com)\> loaded succesfully. 121102 13:11:45 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 121102 13:11:45 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’. 121102 13:11:45 [Note] WSREP: Passing config to GCS: base\_host = 192.168.100.103; evs.send\_window = 512; evs.user\_send\_window = 512; gcache.dir = /var/lib/mysql/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 4G; gcs.fc\_debug = 0; gcs.fc\_factor = 0.5; gcs.fc\_limit = 16; gcs.fc\_master\_slave = NO; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = NO; gmcast.listen\_addr = tcp://0.0.0.0:4010; replicator.causal\_read\_timeout = PT30S; replicator.commit\_order = 3 121102 13:11:45 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 121102 13:11:45 [Note] WSREP: wsrep\_sst\_grab() 121102 13:11:45 [Note] WSREP: Start replication 121102 13:11:45 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 121102 13:11:45 [Note] WSREP: protonet asio version 0 121102 13:11:45 [Note] WSREP: backend: asio 121102 13:11:45 [Note] WSREP: GMCast version 0 121102 13:11:45 [Note] WSREP: (d9fd24c5-24ee-11e2-0800-044cf4047e4f, ‘tcp://0.0.0.0:4010’) listening at tcp://0.0.0.0:4010 121102 13:11:45 [Note] WSREP: (d9fd24c5-24ee-11e2-0800-044cf4047e4f, ‘tcp://0.0.0.0:4010’) multicast: , ttl: 1 121102 13:11:45 [Note] WSREP: EVS version 0 121102 13:11:45 [Note] WSREP: PC version 0 121102 13:11:45 [Note] WSREP: gcomm: connecting to group ‘lptlive’, peer ‘192.168.100.101:4010’ 121102 13:11:45 [Note] WSREP: (d9fd24c5-24ee-11e2-0800-044cf4047e4f, ‘tcp://0.0.0.0:4010’) turning message relay requesting on, nonlive peers: tcp://192.168.100.57:4010 tcp://192.168.101.105:4010 tcp://192.168.101.104:4010 tcp://192.168.100.139:4010 121102 13:11:45 [Note] WSREP: (d9fd24c5-24ee-11e2-0800-044cf4047e4f, ‘tcp://0.0.0.0:4010’) turning message relay requesting off 121102 13:11:45 [Note] WSREP: declaring 72f89f95-0729-11e2-0800-e80c813e3134 stable 121102 13:11:45 [Note] WSREP: declaring a8511994-190a-11e2-0800-580d0a80208f stable 121102 13:11:45 [Note] WSREP: declaring a882f2cb-190a-11e2-0800-00171a908230 stable 121102 13:11:45 [Note] WSREP: declaring a8b2f666-190a-11e2-0800-56702e7cb352 stable 121102 13:11:45 [Note] WSREP: declaring a8beb15d-190a-11e2-0800-81fcb2686439 stable 121102 13:11:45 [Note] WSREP: view(view\_id(PRIM,72f89f95-0729-11e2-0800-e80c813e3134,123) memb { 72f89f95-0729-11e2-0800-e80c813e3134, a8511994-190a-11e2-0800-580d0a80208f, a882f2cb-190a-11e2-0800-00171a908230, a8b2f666-190a-11e2-0800-56702e7cb352, a8beb15d-190a-11e2-0800-81fcb2686439, d9fd24c5-24ee-11e2-0800-044cf4047e4f, } joined { } left { } partitioned { }) 121102 13:11:45 [Note] WSREP: gcomm: connected 121102 13:11:45 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 121102 13:11:45 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) 121102 13:11:45 [Note] WSREP: Opened channel ‘lptlive’ 121102 13:11:45 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 5, memb\_num = 6 121102 13:11:45 [Note] WSREP: Waiting for SST to complete. 121102 13:11:45 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 121102 13:11:45 [Note] WSREP: STATE EXCHANGE: sent state msg: da2023f8-24ee-11e2-0800-d38b758c56bd 121102 13:11:45 [Note] WSREP: STATE EXCHANGE: got state msg: da2023f8-24ee-11e2-0800-d38b758c56bd from 0 (garb) 121102 13:11:45 [Note] WSREP: STATE EXCHANGE: got state msg: da2023f8-24ee-11e2-0800-d38b758c56bd from 1 (mysqldbdrv03) 121102 13:11:45 [Note] WSREP: STATE EXCHANGE: got state msg: da2023f8-24ee-11e2-0800-d38b758c56bd from 3 (mysqldbdrv02) 121102 13:11:45 [Note] WSREP: STATE EXCHANGE: got state msg: da2023f8-24ee-11e2-0800-d38b758c56bd from 4 (mysqldb03) 121102 13:11:45 [Note] WSREP: STATE EXCHANGE: got state msg: da2023f8-24ee-11e2-0800-d38b758c56bd from 2 (mysqldbdrv01) 121102 13:11:45 [Note] WSREP: STATE EXCHANGE: got state msg: da2023f8-24ee-11e2-0800-d38b758c56bd from 5 (mysqldb01) 121102 13:11:45 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf\_id = 118, members = 5/6 (joined/total), act\_id = 8643880, last\_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = 72de1b90-022d-11e2-0800-484cdf256d28 121102 13:11:45 [Note] WSREP: Flow-control interval: [20, 39] 121102 13:11:45 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 8643880) 121102 13:11:45 [Note] WSREP: State transfer required: Group state: 72de1b90-022d-11e2-0800-484cdf256d28:8643880 Local state: 00000000-0000-0000-0000-000000000000:-1 121102 13:11:45 [Note] WSREP: New cluster view: global state: 72de1b90-022d-11e2-0800-484cdf256d28:8643880, view# 119: Primary, number of nodes: 6, my index: 5, protocol version 2 121102 13:11:45 [Warning] WSREP: Gap in state sequence. Need state transfer. 121102 13:11:47 [Note] WSREP: Running: ‘wsrep\_sst\_xtrabackup ‘joiner’ ‘192.168.100.103’ ‘replic:5vaeNeIiTYBRU’ ‘/var/lib/mysql/’ ‘/etc/my.cnf’ ‘12805’ 2\>sst.err’ 121102 13:11:47 [Note] WSREP: Prepared SST request: xtrabackup|192.168.100.103:4444/xtrabackup\_sst ERROR 2003 (HY000): Can’t connect to MySQL server on ‘127.0.0.1’ (111) sed: couldn’t flush stdout: Broken pipe 121102 13:11:47 [ERROR] WSREP: Process completed with error: /usr/bin/wsrep\_notify.sh --status Joiner --uuid 72de1b90-022d-11e2-0800-484cdf256d28 --primary yes --index 5 --members 72f89f95-0729-11e2-0800-e80c813e3134/garb/,a8511994-190a-11e 2-0800-580d0a80208f/mysqldbdrv03/192.168.100. 121102 13:11:47 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/bin/wsrep\_notify.sh --status Joiner --uuid 72de1b90-022d-11e2-0800-484cdf256d28 --primary yes --index 5 --members 72f89f95-0729-11e2-0800-e80c813e3134/garb/,a8511994-190a-11e 2-0800-580d0a802 121102 13:11:47 [Note] WSREP: Assign initial position for certification: 8643880, protocol version: 2 121102 13:11:47 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (72de1b90-022d-11e2-0800-484cdf256d28): 1 (Operation not permitted) at galera/src/replicator\_str.cpp:prepare\_for\_IST():439. IST will be unavailable. 121102 13:11:47 [Note] WSREP: Node 5 (mysqldb01) requested state transfer from ‘_any_’. Selected 1 (mysqldbdrv03)(SYNCED) as donor. 121102 13:11:47 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 8643883) 121102 13:11:47 [Note] WSREP: Requesting state transfer: success, donor: 1 121102 13:13:24 [Note] WSREP: 1 (mysqldbdrv03): State transfer to 5 (mysqldb01) complete. 121102 13:13:24 [Note] WSREP: Member 1 (mysqldbdrv03) synced with group. 121102 13:13:38 [Note] WSREP: SST complete, seqno: 8644014 121102 13:13:38 [Note] Plugin ‘FEDERATED’ is disabled. 121102 13:13:38 InnoDB: The InnoDB memory heap is disabled 121102 13:13:38 InnoDB: Mutexes and rw\_locks use GCC atomic builtins 121102 13:13:38 InnoDB: Compressed tables use zlib 1.2.3 121102 13:13:38 InnoDB: Using Linux native AIO 121102 13:13:38 InnoDB: Initializing buffer pool, size = 5.9G 121102 13:13:38 InnoDB: Completed initialization of buffer pool 121102 13:13:38 InnoDB: highest supported file format is Barracuda. 121102 13:13:38 InnoDB: Waiting for the background threads to start 121102 13:13:39 Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 1.1.8-rel25.3 started; log sequence number 31925464588 121102 13:13:39 [Note] Recovering after a crash using mysql-bin 121102 13:13:40 [Note] Starting crash recovery… 121102 13:13:40 [Note] Crash recovery finished. 121102 13:13:40 [Note] Server hostname (bind-address): ‘(null)’; port: 3306 121102 13:13:40 [Note] - ‘(null)’ resolves to ‘0.0.0.0’; 121102 13:13:40 [Note] - ‘(null)’ resolves to ‘::’; 121102 13:13:40 [Note] Server socket created on IP: ‘0.0.0.0’. 121102 13:13:40 [Warning] ‘proxies\_priv’ entry ‘@ root@mysqldb02’ ignored in --skip-name-resolve mode. 121102 13:13:40 [Note] Event Scheduler: Loaded 0 events 121102 13:13:40 [Note] WSREP: Signalling provider to continue. 121102 13:13:40 [Note] WSREP: Received SST: 72de1b90-022d-11e2-0800-484cdf256d28:8644014 121102 13:13:40 [Note] WSREP: SST received: 72de1b90-022d-11e2-0800-484cdf256d28:8644014 121102 13:13:40 [Note] WSREP: 5 (mysqldb01): State transfer from 1 (mysqldbdrv03) complete The /var/log/messages at the time of failure says: Nov 2 00:55:12 mysqldbsrv01 xinetd[8461]: START: mysqlchk pid=4850 from=::ffff:192.168.100.102 Nov 2 00:55:12 mysqldbsrv01 xinetd[8461]: START: mysqlchk pid=4851 from=::ffff:192.168.100.101 Nov 2 00:55:13 mysqldbsrv01 xinetd[8461]: EXIT: mysqlchk status=141 pid=4851 duration=1(sec) Nov 2 00:55:13 mysqldbsrv01 xinetd[8461]: EXIT: mysqlchk status=141 pid=4850 duration=1(sec) Nov 2 00:55:13 mysqldbsrv01 xinetd[8461]: START: mysqlchk pid=4909 from=::ffff:192.168.100.101 Nov 2 00:55:13 mysqldbsrv01 xinetd[8461]: START: mysqlchk pid=4910 from=::ffff:192.168.100.102 Nov 2 00:55:14 mysqldbsrv01 kernel: mysqld invoked oom-killer: gfp\_mask=0x201da, order=0, oom\_adj=0, oom\_score\_adj=0 Nov 2 00:55:14 mysqldbsrv01 kernel: mysqld cpuset=/ mems\_allowed=0 Nov 2 00:55:14 mysqldbsrv01 kernel: Pid: 4457, comm: mysqld Not tainted 2.6.32-220.el6.x86\_64 #1 Nov 2 00:55:14 mysqldbsrv01 kernel: Call Trace: Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? cpuset\_print\_task\_mems\_allowed+0x91/0xb0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? dump\_header+0x90/0x1b0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? security\_real\_capable\_noaudit+0x3c/0x70 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? oom\_kill\_process+0x8a/0x2c0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? select\_bad\_process+0xe1/0x120 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? out\_of\_memory+0x220/0x3c0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? \_\_alloc\_pages\_nodemask+0x89e/0x940 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? alloc\_pages\_current+0xaa/0x110 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? \_\_page\_cache\_alloc+0x87/0x90 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? \_\_do\_page\_cache\_readahead+0xdb/0x210 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? ra\_submit+0x21/0x30 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? filemap\_fault+0x4c3/0x500 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? \_\_do\_fault+0x54/0x510 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? handle\_pte\_fault+0xf7/0xb50 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? sock\_aio\_read+0x181/0x190 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? handle\_mm\_fault+0x1e4/0x2b0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? \_\_do\_page\_fault+0x139/0x480 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? lock\_sock\_nested+0xac/0xc0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? \_spin\_unlock\_bh+0x1b/0x20 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? release\_sock+0xce/0xe0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? sock\_setsockopt+0x16d/0x6f0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? do\_page\_fault+0x3e/0xa0 Nov 2 00:55:14 mysqldbsrv01 kernel: [] ? page\_fault+0x25/0x30 Nov 2 00:55:14 mysqldbsrv01 kernel: Mem-Info: Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 DMA per-cpu: Nov 2 00:55:14 mysqldbsrv01 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Nov 2 00:55:14 mysqldbsrv01 kernel: CPU 1: hi: 0, btch: 1 usd: 0 Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 DMA32 per-cpu: Nov 2 00:55:14 mysqldbsrv01 kernel: CPU 0: hi: 186, btch: 31 usd: 21 Nov 2 00:55:14 mysqldbsrv01 kernel: CPU 1: hi: 186, btch: 31 usd: 46 Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 Normal per-cpu: Nov 2 00:55:14 mysqldbsrv01 kernel: CPU 0: hi: 186, btch: 31 usd: 34 Nov 2 00:55:14 mysqldbsrv01 kernel: CPU 1: hi: 186, btch: 31 usd: 53 Nov 2 00:55:14 mysqldbsrv01 kernel: active\_anon:1648167 inactive\_anon:322976 isolated\_anon:0 Nov 2 00:55:14 mysqldbsrv01 kernel: active\_file:210 inactive\_file:3276 isolated\_file:0 Nov 2 00:55:14 mysqldbsrv01 kernel: unevictable:0 dirty:0 writeback:0 unstable:0 Nov 2 00:55:14 mysqldbsrv01 kernel: free:25820 slab\_reclaimable:2474 slab\_unreclaimable:12998 Nov 2 00:55:14 mysqldbsrv01 kernel: mapped:759 shmem:0 pagetables:7759 bounce:0 Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 DMA free:15660kB min:124kB low:152kB high:184kB active\_anon:0kB inactive\_anon:0kB active\_file:0kB inactive\_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15268kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab\_reclaimable:0kB slab\_unreclaimable:0kB kernel\_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback\_tmp:0kB pages\_scanned:0 all\_unreclaimable? yes Nov 2 00:55:14 mysqldbsrv01 kernel: lowmem\_reserve: 0 3000 8050 8050 Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 DMA32 free:45348kB min:25140kB low:31424kB high:37708kB active\_anon:2310072kB inactive\_anon:578088kB active\_file:24kB inactive\_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:24kB shmem:0kB slab\_reclaimable:144kB slab\_unreclaimable:0kB kernel\_stack:0kB pagetables:1636kB unstable:0kB bounce:0kB writeback\_tmp:0kB pages\_scanned:33 all\_unreclaimable? no Nov 2 00:55:14 mysqldbsrv01 kernel: lowmem\_reserve: 0 0 5050 5050 Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 Normal free:42272kB min:42316kB low:52892kB high:63472kB active\_anon:4282596kB inactive\_anon:713816kB active\_file:816kB inactive\_file:13104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:0kB writeback:0kB mapped:3184kB shmem:0kB slab\_reclaimable:9752kB slab\_unreclaimable:51992kB kernel\_stack:1352kB pagetables:29400kB unstable:0kB bounce:0kB writeback\_tmp:0kB pages\_scanned:608 all\_unreclaimable? yes Nov 2 00:55:14 mysqldbsrv01 kernel: lowmem\_reserve: 0 0 0 0 Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 DMA: 1_4kB 1_8kB 0_16kB 1_32kB 2_64kB 1_128kB 0_256kB 0_512kB 1_1024kB 1_2048kB 3_4096kB = 15660kB Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 Normal: 416_4kB 282_8kB 205_16kB 148_32kB 122_64kB 64_128kB 28_256kB 10_512kB 2_1024kB 0_2048kB 0_4096kB = 42272kB Nov 2 00:55:14 mysqldbsrv01 kernel: 9145 total pagecache pages Nov 2 00:55:14 mysqldbsrv01 kernel: 5631 pages in swap cache Nov 2 00:55:14 mysqldbsrv01 kernel: Swap cache stats: add 684333, delete 678702, find 182870270/182890443 Nov 2 00:55:14 mysqldbsrv01 kernel: Free swap = 0kB Nov 2 00:55:14 mysqldbsrv01 kernel: Total swap = 999992kB Nov 2 00:55:14 mysqldbsrv01 kernel: 2097136 pages RAM Nov 2 00:55:14 mysqldbsrv01 kernel: 48654 pages reserved Nov 2 00:55:14 mysqldbsrv01 kernel: 2079 pages shared Nov 2 00:55:14 mysqldbsrv01 kernel: 2018022 pages non-shared Nov 2 00:55:14 mysqldbsrv01 kernel: [pid] uid tgid total\_vm rss cpu oom\_adj oom\_score\_adj name Nov 2 00:55:14 mysqldbsrv01 kernel: [453] 0 453 2880 0 1 -17 -1000 udevd Nov 2 00:55:14 mysqldbsrv01 kernel: [1013] 0 1013 37677 114 0 0 0 vmtoolsd Nov 2 00:55:14 mysqldbsrv01 kernel: [1266] 0 1266 23306 31 0 -17 -1000 auditd Nov 2 00:55:14 mysqldbsrv01 kernel: [1282] 0 1282 62714 105 0 0 0 rsyslogd Nov 2 00:55:14 mysqldbsrv01 kernel: [1432] 0 1432 29321 24 1 0 0 crond Nov 2 00:55:14 mysqldbsrv01 kernel: [1460] 0 1460 1029 1 0 0 0 mingetty Nov 2 00:55:14 mysqldbsrv01 kernel: [1462] 0 1462 1029 1 0 0 0 mingetty Nov 2 00:55:14 mysqldbsrv01 kernel: [1464] 0 1464 3077 0 0 -17 -1000 udevd Nov 2 00:55:14 mysqldbsrv01 kernel: [1465] 0 1465 3077 0 0 -17 -1000 udevd Nov 2 00:55:14 mysqldbsrv01 kernel: [1466] 0 1466 1029 1 1 0 0 mingetty Nov 2 00:55:14 mysqldbsrv01 kernel: [1468] 0 1468 1029 1 1 0 0 mingetty Nov 2 00:55:14 mysqldbsrv01 kernel: [1470] 0 1470 1029 1 1 0 0 mingetty Nov 2 00:55:14 mysqldbsrv01 kernel: [3338] 81 3338 7920 1 1 0 0 dbus-daemon Nov 2 00:55:14 mysqldbsrv01 kernel: [3366] 0 3366 5767 0 0 0 0 oddjobd Nov 2 00:55:14 mysqldbsrv01 kernel: [3476] 0 3476 16017 24 1 0 0 sshd Nov 2 00:55:14 mysqldbsrv01 kernel: [3527] 38 3527 7014 39 0 0 0 ntpd Nov 2 00:55:14 mysqldbsrv01 kernel: [3894] 0 3894 44097 54 1 0 0 sssd Nov 2 00:55:14 mysqldbsrv01 kernel: [3896] 0 3896 635966 456032 0 0 0 sssd\_be Nov 2 00:55:14 mysqldbsrv01 kernel: [3898] 0 3898 156200 41587 0 0 0 sssd\_nss Nov 2 00:55:14 mysqldbsrv01 kernel: [3899] 0 3899 41845 28 0 0 0 sssd\_pam Nov 2 00:55:14 mysqldbsrv01 kernel: [4893] 0 4893 12909 23 1 0 0 master Nov 2 00:55:14 mysqldbsrv01 kernel: [4896] 89 4896 13463 17 0 0 0 qmgr Nov 2 00:55:14 mysqldbsrv01 kernel: [5541] 0 5541 33684 8872 0 0 0 ruby Nov 2 00:55:14 mysqldbsrv01 kernel: [6798] 0 6798 1029 1 1 0 0 mingetty Nov 2 00:55:14 mysqldbsrv01 kernel: [8461] 0 8461 5536 32 0 0 0 xinetd Nov 2 00:55:14 mysqldbsrv01 kernel: [23283] 0 23283 2873 17 0 0 0 mysqld\_safe Nov 2 00:55:14 mysqldbsrv01 kernel: [26543] 0 26543 49177 142 1 0 0 snmpd Nov 2 00:55:14 mysqldbsrv01 kernel: [15222] 498 15222 3092711 1457974 1 0 0 mysqld Nov 2 00:55:14 mysqldbsrv01 kernel: [22692] 20504 22692 2614 38 1 0 0 nrpe Nov 2 00:55:14 mysqldbsrv01 kernel: [27036] 89 27036 13445 140 1 0 0 pickup Nov 2 00:55:14 mysqldbsrv01 kernel: [4871] 20504 4871 2614 34 0 0 0 nrpe Nov 2 00:55:14 mysqldbsrv01 kernel: [4872] 20504 4872 2615 34 1 0 0 nrpe Nov 2 00:55:14 mysqldbsrv01 kernel: [4873] 20504 4873 27054 69 1 0 0 pmp-check-mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4875] 20504 4875 2614 34 0 0 0 nrpe Nov 2 00:55:14 mysqldbsrv01 kernel: [4876] 20504 4876 2615 34 1 0 0 nrpe Nov 2 00:55:14 mysqldbsrv01 kernel: [4877] 20504 4877 27054 69 0 0 0 pmp-check-mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4879] 20504 4879 2614 34 0 0 0 nrpe Nov 2 00:55:14 mysqldbsrv01 kernel: [4880] 20504 4880 2615 34 1 0 0 nrpe Nov 2 00:55:14 mysqldbsrv01 kernel: [4881] 20504 4881 27054 69 0 0 0 pmp-check-mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4882] 20504 4882 27087 77 1 0 0 pmp-check-mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4883] 20504 4883 27087 77 0 0 0 pmp-check-mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4884] 20504 4884 27087 77 0 0 0 pmp-check-mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4906] 20504 4906 49488 391 0 0 0 mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4907] 20504 4907 49488 391 0 0 0 mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4908] 20504 4908 49488 388 0 0 0 mysql Nov 2 00:55:14 mysqldbsrv01 kernel: [4909] 99 4909 26521 176 0 0 0 clustercheck Nov 2 00:55:14 mysqldbsrv01 kernel: [4910] 99 4910 26521 177 0 0 0 clustercheck Nov 2 00:55:14 mysqldbsrv01 kernel: Out of memory: Kill process 15222 (mysqld) score 654 or sacrifice child Nov 2 00:55:14 mysqldbsrv01 kernel: Killed process 15222, UID 498, (mysqld) total-vm:12370844kB, anon-rss:5830224kB, file-rss:1672kB Nov 2 00:55:14 mysqldbsrv01 xinetd[8461]: START: mysqlchk pid=4922 from=::ffff:192.168.100.101 Nov 2 00:55:14 mysqldbsrv01 xinetd[8461]: START: mysqlchk pid=4923 from=::ffff:192.168.100.102 Nov 2 00:55:14 mysqldbsrv01 xinetd[8461]: EXIT: mysqlchk status=141 pid=4909 duration=1(sec) Nov 2 00:55:14 mysqldbsrv01 xinetd[8461]: EXIT: mysqlchk status=141 pid=4923 duration=0(sec) Nov 2 00:55:14 mysqldbsrv01 xinetd[8461]: EXIT: mysqlchk status=141 pid=4922 duration=0(sec) Nov 2 00:55:14 mysqldbsrv01 xinetd[8461]: EXIT: mysqlchk status=141 pid=4910 duration=1(sec) Nov 2 00:55:15 mysqldbsrv01 xinetd[8461]: START: mysqlchk pid=4983 from=::ffff:192.168.100.101 Nov 2 00:55:14 mysqldbsrv01 kernel: Node 0 DMA32: 353_4kB 146_8kB 71_16kB 53_32kB 36_64kB 28_128kB 21_256kB 20_512kB 10_1024kB 4_2048kB 0\*4096kB = 45348kB ---
**Author:** ![rdab100](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rdab100/32/8_2.png) [@rdab100](https://forums.percona.com/u/rdab100)\ **Post date:** [November 12, 2012, 6:11am UTC](https://forums.percona.com/t/xtradb-cluster-node-failure/2498/2 "2012-11-12T06:11:33Z")
No one able to comment? ---
**Author:** ![karileh](https://avatars.discourse-cdn.com/v4/letter/k/bb73d2/32.png) [@karileh](https://forums.percona.com/u/karileh)\ **Post date:** [November 12, 2012, 1:08pm UTC](https://forums.percona.com/t/xtradb-cluster-node-failure/2498/3 "2012-11-12T13:08:00Z")
Well this is just a guess, but in your log messages I see some notices about running out of memory, so that’s probably the root cause. 121102 0:57:17 InnoDB: Fatal error: cannot allocate memory for the buffer pool And this point kernel oom killer has step in by killing mysqld: Nov 2 00:55:14 mysqldbsrv01 kernel: mysqld invoked oom-killer: gfp\_mask=0x201da, order=0, oom\_adj=0, oom\_score\_adj=0 I’m currently evaluating Galera cluster by myself and I’m having problem with excessive memory usage compared to standalone mysql server, so maybe these issues are related. [URL][Percona Community Forum - Technical forum for help with Percona's open source software for MySQL®, PostgreSQL, and MongoDB® databases.](http://forum.percona.com/index.php?t=msg&th=2871%5B/URL%5D) ---
**Author:** ![rdab100](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rdab100/32/8_2.png) [@rdab100](https://forums.percona.com/u/rdab100)\ **Post date:** [November 15, 2012, 7:36am UTC](https://forums.percona.com/t/xtradb-cluster-node-failure/2498/4 "2012-11-15T07:36:15Z")
Thanks for the response… The error messages that it had run out of memory are noted but there isn’t any explanation as to why it happened (why I asked were there some other logs that should be looked at) the servers and databases were not under load high IO or had anything that could obviously explain it … the nodes run fine, and then pop and go down… but why? I also note the apparent memory leak in 5.5.27 cluster and that 5.5.28 standalone doesn’t have the same leak after your post. Maybe the problems I am seeing are related. Dom