# Help **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [October 8, 2014, 5:21pm UTC](https://forums.percona.com/t/help/3809 "2014-10-08T17:21:32Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 8, 2014, 5:21pm UTC](https://forums.percona.com/t/help/3809/1 "2014-10-08T17:21:32Z")
Hi, guys i have succesfully setup Xtradb cluster with a total of 2 nodes however it ramdonly crashes, can anyone help? so far the error log has the following: 2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_6”.“rlk\_users” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_6”.“rlk\_usermeta” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_6”.“rlk\_comments” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-08 18:47:16 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-08 18:47:16 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_9”.“wp\_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-08 18:47:25 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-08 18:47:25 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. why would this happen? thank you for all your help ---
**Author:** ![przemek](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/przemek/32/3_2.png) [@przemek](https://forums.percona.com/u/przemek)\ **Post date:** [October 10, 2014, 3:59am UTC](https://forums.percona.com/t/help/3809/2 "2014-10-10T03:59:00Z")
For some reason this system table seems missing. Did you do any manual work on mysql database? How was the second node created? Can you try with mysql\_upgrade script if it fixes it? ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:30am UTC](https://forums.percona.com/t/help/3809/3 "2014-10-10T08:30:13Z")
Hi, No i have not, they are both fresh with same versions of everything, it lasts 1 or 2 days and then crashes. Last time i ended up restoring database. This time crashed but didnt damage the database here is the error log: 2014-10-10 05:05:57 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:05:57 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_1”.“rlk\_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:06:00 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:06:00 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_9”.“wp\_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:06:18 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:06:18 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:06:35 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:06:35 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_7”.“wp\_commentmeta” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:10:19 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:10:19 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:15 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:16 19556 [Warning] Too many connections 2014-10-10 05:12:17 19556 [Warning] Too many connections 2014-10-10 05:12:17 19556 [Warning] Too many connections 2014-10-10 05:12:17 19556 [Warning] Too many connections 2014-10-10 05:12:17 19556 [Warning] Too many connections 2014-10-10 05:12:17 19556 [Warning] Too many connections 2014-10-10 05:12:17 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:22 19556 [Warning] Too many connections 2014-10-10 05:12:55 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:12:55 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:13:35 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:13:35 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_9”.“wp\_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_6”.“rlk\_wfHoover” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_6”.“rlk\_wfHoover” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:19:00 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:19:00 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_wfScanners” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:19:00 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:19:00 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:21:40 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:21:40 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_1”.“rlk\_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 05:23:27 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 05:23:27 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 09:23:40 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at [url][https://bugs.launchpad.net/percona-xtradb-cluster[/url]](https://bugs.launchpad.net/percona-xtradb-cluster%5B/url%5D) key\_buffer\_size=8388608 read\_buffer\_size=131072 max\_used\_connections=152 max\_threads=153 thread\_count=3 connection\_count=1 It is possible that mysqld could use up to key\_buffer\_size + (read\_buffer\_size + sort\_buffer\_size)\*max\_threads = 69257 K bytes of memory Hope that’s ok; if not, decrease some variables in the equation. ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:31am UTC](https://forums.percona.com/t/help/3809/4 "2014-10-10T08:31:36Z")
Thread pointer: 0x124f84e0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong… stack\_bottom = 7fd855f2ed38 thread\_stack 0x40000 /usr/sbin/mysqld(my\_print\_stacktrace+0x35)[0x8f5835] /usr/sbin/mysqld(handle\_fatal\_signal+0x4b4)[0x664384] /lib64/libpthread.so.0(+0xf710)[0x7fdb46db1710] /usr/sbin/mysqld(\_ZNK7handler22ha\_statistic\_incrementEM17system\_status\_vary+0x8)[0x59f768] /usr/sbin/mysqld(\_ZN9ha\_myisam9write\_rowEPh+0x20)141010 05:23:40 mysqld\_safe Number of processes running now: 0 141010 05:23:40 mysqld\_safe WSREP: not restarting wsrep node automatically 141010 05:23:40 mysqld\_safe mysqld from pid file /var/lib/mysql/tree.flyheight.com.pid ended 141010 09:46:41 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql 141010 09:46:41 mysqld\_safe WSREP: Running position recovery with --log\_error=‘/var/lib/mysql/wsrep\_recovery.KI7FTN’ --pid-file=‘/var/lib/mysql/tree.flyheight.com-recover.pid’ 2014-10-10 09:46:42 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit\_defaults\_for\_timestamp server option (see documentation for more details). 141010 09:46:46 mysqld\_safe WSREP: Recovered position e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693 2014-10-10 09:46:46 0 [Note] WSREP: wsrep\_start\_position var submitted: ‘e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693’ 2014-10-10 09:46:46 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit\_defaults\_for\_timestamp server option (see documentation for more details). 2014-10-10 09:46:46 12326 [Note] WSREP: Read nil XID from storage engines, skipping position init 2014-10-10 09:46:46 12326 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 2014-10-10 09:46:46 12326 [Note] WSREP: wsrep\_load(): Galera 3.7(r7f44a18) by Codership Oy \ loaded successfully. 2014-10-10 09:46:46 12326 [Note] WSREP: CRC-32C: using hardware acceleration. 2014-10-10 09:46:46 12326 [Note] WSREP: Found saved state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:-1 2014-10-10 09:46:46 12326 [Note] WSREP: Passing config to GCS: base\_host = 192.99.19.158; base\_port = 4567; cert.log\_conflicts = no; debug = no; evs.inactive\_check\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 4; evs.stats\_report\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 2; evs.view\_forget\_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1.0; gcs.fc\_limit = 16; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignore\_quorum = false; pc.ignore\_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait\_prim = true; pc.wait\_prim\_timeout = P30S; pc.w 2014-10-10 09:46:46 12326 [Note] WSREP: Service thread queue flushed. 2014-10-10 09:46:46 12326 [Note] WSREP: Assign initial position for certification: 3555693, protocol version: -1 2014-10-10 09:46:46 12326 [Note] WSREP: wsrep\_sst\_grab() 2014-10-10 09:46:46 12326 [Note] WSREP: Start replication 2014-10-10 09:46:46 12326 [Note] WSREP: Setting initial position to e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693 2014-10-10 09:46:46 12326 [Note] WSREP: protonet asio version 0 2014-10-10 09:46:46 12326 [Note] WSREP: Using CRC-32C for message checksums. 2014-10-10 09:46:46 12326 [Note] WSREP: backend: asio 2014-10-10 09:46:46 12326 [Note] WSREP: restore pc from disk successfully 2014-10-10 09:46:46 12326 [Note] WSREP: GMCast version 0 2014-10-10 09:46:46 12326 [Note] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 2014-10-10 09:46:46 12326 [Note] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 2014-10-10 09:46:46 12326 [Note] WSREP: EVS version 0 2014-10-10 09:46:46 12326 [Note] WSREP: PC version 0 2014-10-10 09:46:46 12326 [Note] WSREP: gcomm: connecting to group ‘my\_centos\_cluster’, peer ‘192.99.19.158:,192.99.19.154:’ 2014-10-10 09:46:46 12326 [Warning] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) address ‘tcp://192.99.19.158:4567’ points to own listening address, blacklisting 2014-10-10 09:46:49 12326 [Warning] WSREP: no nodes coming from prim view, prim not possible 2014-10-10 09:46:49 12326 [Note] WSREP: gcomm: connected 2014-10-10 09:46:49 12326 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2014-10-10 09:46:49 12326 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) 2014-10-10 09:46:49 12326 [Note] WSREP: Opened channel ‘my\_centos\_cluster’ 2014-10-10 09:46:49 12326 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1 2014-10-10 09:46:49 12326 [Note] WSREP: Waiting for SST to complete. 2014-10-10 09:46:49 12326 [Note] WSREP: Flow-control interval: [16, 16] 2014-10-10 09:46:49 12326 [Note] WSREP: Received NON-PRIMARY. 2014-10-10 09:46:49 12326 [Note] WSREP: New cluster view: global state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2014-10-10 09:46:49 12326 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2014-10-10 09:46:50 12326 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50128S), skipping check 141010 09:47:37 mysqld\_safe mysqld from pid file /var/lib/mysql/tree.flyheight.com.pid ended 141010 09:47:40 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql 141010 09:47:40 mysqld\_safe Skipping wsrep-recover for e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693 pair 141010 09:47:40 mysqld\_safe Assigning e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693 to wsrep\_start\_position 2014-10-10 09:47:40 0 [Note] WSREP: wsrep\_start\_position var submitted: ‘e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693’ 2014-10-10 09:47:40 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit\_defaults\_for\_timestamp server option (see documentation for more details). ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:32am UTC](https://forums.percona.com/t/help/3809/5 "2014-10-10T08:32:34Z")
2014-10-10 09:47:40 12794 [Note] WSREP: Read nil XID from storage engines, skipping position init 2014-10-10 09:47:40 12794 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 2014-10-10 09:47:40 12794 [Note] WSREP: wsrep\_load(): Galera 3.7(r7f44a18) by Codership Oy \ loaded successfully. 2014-10-10 09:47:40 12794 [Note] WSREP: CRC-32C: using hardware acceleration. 2014-10-10 09:47:40 12794 [Note] WSREP: Found saved state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693 2014-10-10 09:47:40 12794 [Note] WSREP: Passing config to GCS: base\_host = 192.99.19.158; base\_port = 4567; cert.log\_conflicts = no; debug = no; evs.inactive\_check\_period = PT0.5S; evs.inactive\_timeout = PT15S; evs.join\_retrans\_period = PT1S; evs.max\_install\_timeouts = 3; evs.send\_window = 4; evs.stats\_report\_period = PT1M; evs.suspect\_timeout = PT5S; evs.user\_send\_window = 2; evs.view\_forget\_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1.0; gcs.fc\_limit = 16; gcs.fc\_master\_slave = no; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce\_timeout = PT3S; pc.checksum = false; pc.ignore\_quorum = false; pc.ignore\_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait\_prim = true; pc.wait\_prim\_timeout = P30S; pc.w 2014-10-10 09:47:41 12794 [Note] WSREP: Service thread queue flushed. 2014-10-10 09:47:41 12794 [Note] WSREP: Assign initial position for certification: 3555693, protocol version: -1 2014-10-10 09:47:41 12794 [Note] WSREP: wsrep\_sst\_grab() 2014-10-10 09:47:41 12794 [Note] WSREP: Start replication 2014-10-10 09:47:41 12794 [Note] WSREP: Setting initial position to e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693 2014-10-10 09:47:41 12794 [Note] WSREP: protonet asio version 0 2014-10-10 09:47:41 12794 [Note] WSREP: Using CRC-32C for message checksums. 2014-10-10 09:47:41 12794 [Note] WSREP: backend: asio 2014-10-10 09:47:41 12794 [Note] WSREP: restore pc from disk successfully 2014-10-10 09:47:41 12794 [Note] WSREP: GMCast version 0 2014-10-10 09:47:41 12794 [Note] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 2014-10-10 09:47:41 12794 [Note] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 2014-10-10 09:47:41 12794 [Note] WSREP: EVS version 0 2014-10-10 09:47:41 12794 [Note] WSREP: PC version 0 2014-10-10 09:47:41 12794 [Note] WSREP: gcomm: bootstrapping new group ‘my\_centos\_cluster’ 2014-10-10 09:47:41 12794 [Note] WSREP: start\_prim is enabled, turn off pc\_recovery 2014-10-10 09:47:41 12794 [Note] WSREP: Node 778c1134 state prim 2014-10-10 09:47:41 12794 [Note] WSREP: save pc into disk 2014-10-10 09:47:41 12794 [Note] WSREP: discarding pending addr without UUID: tcp://192.99.19.154:4567 2014-10-10 09:47:41 12794 [Note] WSREP: discarding pending addr proto entry 0x14d9930 2014-10-10 09:47:41 12794 [Note] WSREP: discarding pending addr without UUID: tcp://192.99.19.158:4567 2014-10-10 09:47:41 12794 [Note] WSREP: discarding pending addr proto entry 0x14e20f0 2014-10-10 09:47:41 12794 [Note] WSREP: clear restored view 2014-10-10 09:47:41 12794 [Note] WSREP: gcomm: connected 2014-10-10 09:47:41 12794 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2014-10-10 09:47:41 12794 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) 2014-10-10 09:47:41 12794 [Note] WSREP: Opened channel ‘my\_centos\_cluster’ 2014-10-10 09:47:41 12794 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 0, memb\_num = 1 2014-10-10 09:47:41 12794 [Note] WSREP: Waiting for SST to complete. 2014-10-10 09:47:41 12794 [Note] WSREP: STATE\_EXCHANGE: sent state UUID: 00f84f3f-5084-11e4-a61d-97674bc48724 2014-10-10 09:47:41 12794 [Note] WSREP: STATE EXCHANGE: sent state msg: 00f84f3f-5084-11e4-a61d-97674bc48724 2014-10-10 09:47:41 12794 [Note] WSREP: STATE EXCHANGE: got state msg: 00f84f3f-5084-11e4-a61d-97674bc48724 from 0 ([tree.flyheight.com](http://tree.flyheight.com)) 2014-10-10 09:47:41 12794 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf\_id = 0, members = 1/1 (joined/total), act\_id = 3555693, last\_appl. = -1, protocols = 0/6/3 (gcs/repl/appl), group UUID = e29f57c0-4ee8-11e4-9e34-f334de4c3867 2014-10-10 09:47:41 12794 [Note] WSREP: Flow-control interval: [16, 16] 2014-10-10 09:47:41 12794 [Note] WSREP: Restored state OPEN → JOINED (3555693) 2014-10-10 09:47:41 12794 [Note] WSREP: Member 0.0 ([tree.flyheight.com](http://tree.flyheight.com)) synced with group. 2014-10-10 09:47:41 12794 [Note] WSREP: Shifting JOINED → SYNCED (TO: 3555693) 2014-10-10 09:47:41 12794 [Note] WSREP: New cluster view: global state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3 2014-10-10 09:47:41 12794 [Note] WSREP: SST complete, seqno: 3555693 2014-10-10 09:47:41 12794 [Warning] Using unique option prefix innodb\_log\_group\_home instead of innodb-log-group-home-dir is deprecated and will be removed in a future release. Please use the full name instead. 2014-10-10 09:47:41 12794 [Note] Plugin ‘FEDERATED’ is disabled. 2014-10-10 09:47:41 12794 [Note] InnoDB: Using atomics to ref count buffer pool pages 2014-10-10 09:47:41 12794 [Note] InnoDB: The InnoDB memory heap is disabled 2014-10-10 09:47:41 12794 [Note] InnoDB: Mutexes and rw\_locks use GCC atomic builtins 2014-10-10 09:47:41 12794 [Note] InnoDB: Memory barrier is not used 2014-10-10 09:47:41 12794 [Note] InnoDB: Compressed tables use zlib 1.2.3 2014-10-10 09:47:41 12794 [Note] InnoDB: Using Linux native AIO 2014-10-10 09:47:41 12794 [Note] InnoDB: Using CPU crc32 instructions 2014-10-10 09:47:41 12794 [Note] InnoDB: Initializing buffer pool, size = 10.0G 2014-10-10 09:47:41 12794 [Note] InnoDB: Completed initialization of buffer pool 2014-10-10 09:47:41 12794 [Note] InnoDB: Highest supported file format is Barracuda. ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:34am UTC](https://forums.percona.com/t/help/3809/6 "2014-10-10T08:34:42Z")
2014-10-10 09:47:41 12794 [Note] InnoDB: 128 rollback segment(s) are active. 2014-10-10 09:47:41 12794 [Note] InnoDB: Waiting for purge to start 2014-10-10 09:47:41 12794 [Note] InnoDB: Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 5.6.20-68.0 started; log sequence number 2564845042 2014-10-10 09:47:41 12794 [Note] RSA private key file not found: /var/lib/mysql//private\_key.pem. Some authentication plugins will not work. 2014-10-10 09:47:41 12794 [Note] RSA public key file not found: /var/lib/mysql//public\_key.pem. Some authentication plugins will not work. 2014-10-10 09:47:41 12794 [Note] Server hostname (bind-address): ‘\*’; port: 3306 2014-10-10 09:47:41 12794 [Note] IPv6 is available. 2014-10-10 09:47:41 12794 [Note] - ‘::’ resolves to ‘::’; 2014-10-10 09:47:41 12794 [Note] Server socket created on IP: ‘::’. 2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.events\_waits\_current is wrong. Expected 19, found 16. Created with MySQL 50540, now running 50620. Please use mysql\_upgrade to fix this error. 2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.events\_waits\_history is wrong. Expected 19, found 16. Created with MySQL 50540, now running 50620. Please use mysql\_upgrade to fix this error. 2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.events\_waits\_history\_long is wrong. Expected 19, found 16. Created with MySQL 50540, now running 50620. Please use mysql\_upgrade to fix this error. 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_waits\_summary\_by\_host\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Incorrect definition of table performance\_schema.events\_waits\_summary\_by\_thread\_by\_event\_name: expected column ‘THREAD\_ID’ at position 0 to have type bigint(20), found type int(11). 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_waits\_summary\_by\_user\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_waits\_summary\_by\_account\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.file\_summary\_by\_event\_name is wrong. Expected 23, found 5. Created with MySQL 50540, now running 50620. Please use mysql\_upgrade to fix this error. 2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.file\_summary\_by\_instance is wrong. Expected 25, found 6. Created with MySQL 50540, now running 50620. Please use mysql\_upgrade to fix this error. 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘host\_cache’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Incorrect definition of table performance\_schema.mutex\_instances: expected column ‘LOCKED\_BY\_THREAD\_ID’ at position 2 to have type bigint(20), found type int(11). 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘objects\_summary\_global\_by\_type’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Incorrect definition of table performance\_schema.rwlock\_instances: expected column ‘WRITE\_LOCKED\_BY\_THREAD\_ID’ at position 2 to have type bigint(20), found type int(11). 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘setup\_actors’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘setup\_objects’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘table\_io\_waits\_summary\_by\_index\_usage’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘table\_io\_waits\_summary\_by\_table’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘table\_lock\_waits\_summary\_by\_table’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.threads is wrong. Expected 14, found 3. Created with MySQL 50540, now running 50620. Please use mysql\_upgrade to fix this error. 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_stages\_current’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_stages\_history’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_stages\_history\_long’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_stages\_summary\_by\_thread\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_stages\_summary\_by\_account\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_stages\_summary\_by\_user\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_stages\_summary\_by\_host\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_stages\_summary\_global\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_current’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_history’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_history\_long’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_summary\_by\_thread\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_summary\_by\_account\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_summary\_by\_user\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_summary\_by\_host\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_summary\_global\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘events\_statements\_summary\_by\_digest’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘users’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘accounts’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘hosts’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘socket\_instances’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘socket\_summary\_by\_instance’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘socket\_summary\_by\_event\_name’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘session\_connect\_attrs’ has the wrong structure 2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance\_schema’.‘session\_account\_connect\_attrs’ has the wrong structure 2014-10-10 09:47:41 12794 [Note] Event Scheduler: Loaded 0 events 2014-10-10 09:47:41 12794 [Note] /usr/sbin/mysqld: ready for connections. Version: ‘5.6.20-68.0-56’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel68.0, Revision 888, WSREP version 25.7, wsrep\_25.7.r4126 2014-10-10 09:47:41 12794 [Note] WSREP: inited wsrep sidno 1 2014-10-10 09:47:41 12794 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2014-10-10 09:47:41 12794 [Note] WSREP: REPL Protocols: 6 (3, 2) 2014-10-10 09:47:41 12794 [Note] WSREP: Service thread queue flushed. 2014-10-10 09:47:41 12794 [Note] WSREP: Assign initial position for certification: 3555693, protocol version: 3 2014-10-10 09:47:41 12794 [Note] WSREP: Service thread queue flushed. 2014-10-10 09:47:41 12794 [Note] WSREP: Synchronized with group, ready for connections 2014-10-10 09:47:41 12794 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_3”.“wp\_options” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_3”.“wp\_postmeta” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_3”.“wp\_posts” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_3”.“wp\_terms” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin\_3”.“wp\_term\_taxonomy” but the required system tables mysql.innodb\_table\_stats and mysql.innodb\_index\_stats are not present or have unexpected structure. Using transient stats instead. ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:41am UTC](https://forums.percona.com/t/help/3809/7 "2014-10-10T08:41:40Z")
[COLOR=#FF0000][root@tree ~]# mysql\_upgrade Looking for ‘mysql’ as: mysql Looking for ‘mysqlcheck’ as: mysqlcheck Running 'mysqlcheck with default connection arguments Warning: Using a password on the command line interface can be insecure. Running 'mysqlcheck with default connection arguments Warning: Using a password on the command line interface can be insecure. mysql.columns\_priv OK mysql.db OK mysql.event OK mysql.func OK mysql.general\_log OK mysql.help\_category OK mysql.help\_keyword OK mysql.help\_relation OK mysql.help\_topic OK mysql.host OK mysql.ndb\_binlog\_index OK mysql.plugin OK mysql.proc OK mysql.procs\_priv OK mysql.proxies\_priv OK mysql.servers OK mysql.slow\_log OK mysql.tables\_priv OK mysql.time\_zone OK mysql.time\_zone\_leap\_second OK mysql.time\_zone\_name OK mysql.time\_zone\_transition OK mysql.time\_zone\_transition\_type OK mysql.user OK Running ‘mysql\_fix\_privilege\_tables’… Warning: Using a password on the command line interface can be insecure. Running 'mysqlcheck with default connection arguments Warning: Using a password on the command line interface can be insecure. Running 'mysqlcheck with default connection arguments Warning: Using a password on the command line interface can be insecure. admin\_1.rlk\_aiowps\_failed\_logins OK admin\_1.rlk\_aiowps\_global\_meta OK admin\_1.rlk\_aiowps\_login\_activity OK admin\_1.rlk\_aiowps\_login\_lockdown OK admin\_1.rlk\_bpspro\_login\_security OK admin\_1.rlk\_bpspro\_seclog\_ignore OK admin\_1.rlk\_cntctfrm\_field OK admin\_1.rlk\_commentmeta OK admin\_1.rlk\_comments OK admin\_1.rlk\_fbthumbnails OK admin\_1.rlk\_links OK admin\_1.rlk\_lockdowns OK admin\_1.rlk\_login\_fails OK admin\_1.rlk\_ocmx\_comment\_meta OK admin\_1.rlk\_ocmx\_gallery OK admin\_1.rlk\_ocmx\_gallery\_dtl OK admin\_1.rlk\_options OK admin\_1.rlk\_pollsa OK admin\_1.rlk\_pollsip OK admin\_1.rlk\_pollsq OK admin\_1.rlk\_postlikes OK admin\_1.rlk\_postmeta OK admin\_1.rlk\_posts OK admin\_1.rlk\_pvc\_daily OK admin\_1.rlk\_pvc\_total OK admin\_1.rlk\_sabre\_table OK admin\_1.rlk\_term\_relationships OK admin\_1.rlk\_term\_taxonomy OK admin\_1.rlk\_terms OK admin\_1.rlk\_usermeta OK admin\_1.rlk\_users OK admin\_1.rlk\_wfBadLeechers OK admin\_1.rlk\_wfBlocks OK admin\_1.rlk\_wfBlocksAdv OK admin\_1.rlk\_wfConfig OK admin\_1.rlk\_wfCrawlers OK admin\_1.rlk\_wfFileMods OK admin\_1.rlk\_wfHits OK admin\_1.rlk\_wfHoover OK admin\_1.rlk\_wfIssues OK admin\_1.rlk\_wfLeechers OK admin\_1.rlk\_wfLockedOut OK admin\_1.rlk\_wfLocs OK admin\_1.rlk\_wfLogins OK admin\_1.rlk\_wfNet404s OK admin\_1.rlk\_wfReverseCache OK admin\_1.rlk\_wfScanners OK admin\_1.rlk\_wfStatus OK admin\_1.rlk\_wfThrottleLog OK admin\_1.rlk\_wfVulnScanners OK admin\_1.wpbandit\_images OK admin\_10.wp\_cntctfrm\_field OK admin\_10.wp\_commentmeta OK admin\_10.wp\_comments OK admin\_10.wp\_links OK admin\_10.wp\_nxs\_log OK admin\_10.wp\_options OK admin\_10.wp\_postmeta OK admin\_10.wp\_posts OK admin\_10.wp\_term\_relationships OK admin\_10.wp\_term\_taxonomy OK admin\_10.wp\_terms OK admin\_10.wp\_usermeta OK admin\_10.wp\_users OK admin\_10.wp\_wfBadLeechers OK admin\_10.wp\_wfBlocks OK admin\_10.wp\_wfBlocksAdv OK admin\_10.wp\_wfConfig OK admin\_10.wp\_wfCrawlers OK admin\_10.wp\_wfFileMods OK admin\_10.wp\_wfHits OK admin\_10.wp\_wfHoover OK admin\_10.wp\_wfIssues OK admin\_10.wp\_wfLeechers OK admin\_10.wp\_wfLockedOut OK admin\_10.wp\_wfLocs OK admin\_10.wp\_wfLogins OK admin\_10.wp\_wfNet404s OK admin\_10.wp\_wfReverseCache OK admin\_10.wp\_wfScanners OK admin\_10.wp\_wfStatus OK admin\_10.wp\_wfThrottleLog OK admin\_10.wp\_wfVulnScanners OK admin\_10.wp\_wpo\_campaign OK admin\_10.wp\_wpo\_campaign\_category OK admin\_10.wp\_wpo\_campaign\_feed OK admin\_10.wp\_wpo\_campaign\_post OK admin\_10.wp\_wpo\_campaign\_word OK admin\_10.wp\_wpo\_log OK admin\_10.wp\_wppa\_albums OK admin\_10.wp\_wppa\_comments OK admin\_10.wp\_wppa\_exif OK admin\_10.wp\_wppa\_index OK admin\_10.wp\_wppa\_iptc OK admin\_10.wp\_wppa\_photos OK admin\_10.wp\_wppa\_rating OK admin\_10.wpbandit\_images OK admin\_3.nuevo\_\_htmlad OK admin\_3.nuevo\_\_midroll OK admin\_3.nuevo\_\_midroll\_settings OK admin\_3.nuevo\_\_overlay OK admin\_3.nuevo\_\_peel OK admin\_3.nuevo\_\_player OK admin\_3.nuevo\_\_preroll OK admin\_3.nuevo\_\_ticker OK admin\_3.wp\_cntctfrm\_field OK admin\_3.wp\_cntctfrm\_location OK admin\_3.wp\_commentmeta OK admin\_3.wp\_comments OK admin\_3.wp\_hdflvvideoshare OK admin\_3.wp\_hdflvvideoshare\_med2play OK admin\_3.wp\_hdflvvideoshare\_playlist OK admin\_3.wp\_hdflvvideoshare\_settings OK admin\_3.wp\_hdflvvideoshare\_tags OK admin\_3.wp\_hdflvvideoshare\_vgads OK admin\_3.wp\_links OK admin\_3.wp\_login\_security\_solution\_fail OK admin\_3.wp\_nxs\_log OK admin\_3.wp\_ocmx\_comment\_meta OK admin\_3.wp\_ocmx\_gallery OK admin\_3.wp\_ocmx\_gallery\_dtl OK admin\_3.wp\_options OK admin\_3.wp\_postlikes OK admin\_3.wp\_postmeta OK admin\_3.wp\_posts OK admin\_3.wp\_pvc\_daily OK admin\_3.wp\_pvc\_total OK admin\_3.wp\_social\_users OK admin\_3.wp\_term\_relationships OK admin\_3.wp\_term\_taxonomy OK admin\_3.wp\_terms OK admin\_3.wp\_usermeta OK admin\_3.wp\_users OK admin\_3.wpvidz\_commentmeta OK admin\_3.wpvidz\_comments OK admin\_3.wpvidz\_fbthumbnails OK ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:42am UTC](https://forums.percona.com/t/help/3809/8 "2014-10-10T08:42:14Z")
admin\_3.wpvidz\_links OK admin\_3.wpvidz\_nuevo\_\_htmlad OK admin\_3.wpvidz\_nuevo\_\_midroll OK admin\_3.wpvidz\_nuevo\_\_midroll\_settings OK admin\_3.wpvidz\_nuevo\_\_overlay OK admin\_3.wpvidz\_nuevo\_\_peel OK admin\_3.wpvidz\_nuevo\_\_player OK admin\_3.wpvidz\_nuevo\_\_preroll OK admin\_3.wpvidz\_nuevo\_\_ticker OK admin\_3.wpvidz\_options OK admin\_3.wpvidz\_postlikes OK admin\_3.wpvidz\_postmeta OK admin\_3.wpvidz\_posts OK admin\_3.wpvidz\_term\_relationships OK admin\_3.wpvidz\_term\_taxonomy OK admin\_3.wpvidz\_terms OK admin\_3.wpvidz\_usermeta OK admin\_3.wpvidz\_users OK admin\_4.wpvidz\_commentmeta OK admin\_4.wpvidz\_comments OK admin\_4.wpvidz\_fbthumbnails OK admin\_4.wpvidz\_links OK admin\_4.wpvidz\_options OK admin\_4.wpvidz\_postlikes OK admin\_4.wpvidz\_postmeta OK admin\_4.wpvidz\_posts OK admin\_4.wpvidz\_term\_relationships OK admin\_4.wpvidz\_term\_taxonomy OK admin\_4.wpvidz\_terms OK admin\_4.wpvidz\_usermeta OK admin\_4.wpvidz\_users OK admin\_5.wp\_cntctfrm\_field OK admin\_5.wp\_commentmeta OK admin\_5.wp\_comments OK admin\_5.wp\_links OK admin\_5.wp\_nxs\_log OK admin\_5.wp\_options OK admin\_5.wp\_postlikes OK admin\_5.wp\_postmeta OK admin\_5.wp\_posts OK admin\_5.wp\_term\_relationships OK admin\_5.wp\_term\_taxonomy OK admin\_5.wp\_terms OK admin\_5.wp\_usermeta OK admin\_5.wp\_users OK admin\_5.wp\_wfBadLeechers OK admin\_5.wp\_wfBlocks OK admin\_5.wp\_wfBlocksAdv OK admin\_5.wp\_wfConfig OK admin\_5.wp\_wfCrawlers OK admin\_5.wp\_wfFileMods OK admin\_5.wp\_wfHits OK admin\_5.wp\_wfHoover OK admin\_5.wp\_wfIssues OK admin\_5.wp\_wfLeechers OK admin\_5.wp\_wfLockedOut OK admin\_5.wp\_wfLocs OK admin\_5.wp\_wfLogins OK admin\_5.wp\_wfNet404s OK admin\_5.wp\_wfReverseCache OK admin\_5.wp\_wfScanners OK admin\_5.wp\_wfStatus OK admin\_5.wp\_wfThrottleLog OK admin\_5.wp\_wfVulnScanners OK admin\_5.wp\_wpo\_campaign OK admin\_5.wp\_wpo\_campaign\_category OK admin\_5.wp\_wpo\_campaign\_feed OK admin\_5.wp\_wpo\_campaign\_post OK admin\_5.wp\_wpo\_campaign\_word OK admin\_5.wp\_wpo\_log OK admin\_5.wp\_wppa\_albums OK admin\_5.wp\_wppa\_comments OK admin\_5.wp\_wppa\_exif OK admin\_5.wp\_wppa\_index OK admin\_5.wp\_wppa\_iptc OK admin\_5.wp\_wppa\_photos OK admin\_5.wp\_wppa\_rating OK admin\_5.wpbandit\_images OK admin\_6.rlk\_aiowps\_failed\_logins OK admin\_6.rlk\_aiowps\_global\_meta OK admin\_6.rlk\_aiowps\_login\_activity OK admin\_6.rlk\_aiowps\_login\_lockdown OK admin\_6.rlk\_bpspro\_login\_security OK admin\_6.rlk\_bpspro\_seclog\_ignore OK admin\_6.rlk\_cntctfrm\_field OK admin\_6.rlk\_commentmeta OK admin\_6.rlk\_comments OK admin\_6.rlk\_fbthumbnails OK admin\_6.rlk\_links OK admin\_6.rlk\_lockdowns OK admin\_6.rlk\_login\_fails OK admin\_6.rlk\_ocmx\_comment\_meta OK admin\_6.rlk\_ocmx\_gallery OK admin\_6.rlk\_ocmx\_gallery\_dtl OK admin\_6.rlk\_options OK admin\_6.rlk\_pollsa OK admin\_6.rlk\_pollsip OK admin\_6.rlk\_pollsq OK admin\_6.rlk\_postlikes OK admin\_6.rlk\_postmeta OK admin\_6.rlk\_posts OK admin\_6.rlk\_pvc\_daily OK admin\_6.rlk\_pvc\_total OK admin\_6.rlk\_sabre\_table OK admin\_6.rlk\_term\_relationships OK admin\_6.rlk\_term\_taxonomy OK admin\_6.rlk\_terms OK admin\_6.rlk\_usermeta OK admin\_6.rlk\_users OK admin\_6.rlk\_websitez\_stats OK admin\_6.rlk\_wfBadLeechers OK admin\_6.rlk\_wfBlocks OK admin\_6.rlk\_wfBlocksAdv OK admin\_6.rlk\_wfConfig OK admin\_6.rlk\_wfCrawlers OK admin\_6.rlk\_wfFileMods OK admin\_6.rlk\_wfHits OK admin\_6.rlk\_wfHoover OK admin\_6.rlk\_wfIssues OK admin\_6.rlk\_wfLeechers OK admin\_6.rlk\_wfLockedOut OK admin\_6.rlk\_wfLocs OK admin\_6.rlk\_wfLogins OK admin\_6.rlk\_wfNet404s OK admin\_6.rlk\_wfReverseCache OK admin\_6.rlk\_wfScanners OK admin\_6.rlk\_wfStatus OK admin\_6.rlk\_wfThrottleLog OK admin\_6.rlk\_wfVulnScanners OK admin\_6.wpbandit\_images OK admin\_7.wp\_commentmeta OK admin\_7.wp\_comments OK admin\_7.wp\_links OK admin\_7.wp\_options OK admin\_7.wp\_postlikes OK admin\_7.wp\_postmeta OK admin\_7.wp\_posts OK admin\_7.wp\_term\_relationships OK admin\_7.wp\_term\_taxonomy OK admin\_7.wp\_terms OK admin\_7.wp\_usermeta OK admin\_7.wp\_users OK admin\_8.wp\_commentmeta OK admin\_8.wp\_comments OK admin\_8.wp\_links OK admin\_8.wp\_ocmx\_comment\_meta OK admin\_8.wp\_ocmx\_gallery OK admin\_8.wp\_ocmx\_gallery\_dtl OK admin\_8.wp\_options OK admin\_8.wp\_postlikes OK admin\_8.wp\_postmeta OK admin\_8.wp\_posts OK admin\_8.wp\_term\_relationships OK admin\_8.wp\_term\_taxonomy OK admin\_8.wp\_terms OK admin\_8.wp\_usermeta OK admin\_8.wp\_users OK admin\_9.wp\_commentmeta OK admin\_9.wp\_comments OK admin\_9.wp\_links OK admin\_9.wp\_ocmx\_comment\_meta OK admin\_9.wp\_ocmx\_gallery OK admin\_9.wp\_ocmx\_gallery\_dtl OK admin\_9.wp\_options OK admin\_9.wp\_postlikes OK admin\_9.wp\_postmeta OK admin\_9.wp\_posts OK admin\_9.wp\_term\_relationships OK admin\_9.wp\_term\_taxonomy OK admin\_9.wp\_terms OK admin\_9.wp\_usermeta OK admin\_9.wp\_users OK OK ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:43am UTC](https://forums.percona.com/t/help/3809/9 "2014-10-10T08:43:36Z")
[COLOR=#FF0000]This is my.cnf on node #1 [mysqld] innodb\_file\_per\_table datadir=/var/lib/mysql user=mysql # Path to Galera library wsrep\_provider=/usr/lib64/libgalera\_smm.so # Cluster connection URL contains IPs of node#1, node#2 and node#3 wsrep\_cluster\_address=gcomm://192.99.19.158,192.99.19.154 # In order for Galera to work correctly binlog format should be ROW binlog\_format=ROW # MyISAM storage engine has only experimental support default\_storage\_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb\_autoinc\_lock\_mode=2 # Node #2 address wsrep\_node\_address=192.99.19.158 # Cluster name wsrep\_cluster\_name=my\_centos\_cluster # SST method wsrep\_sst\_method=xtrabackup-v2 #Authentication for SST method wsrep\_sst\_auth=“sstuser:\*\*\*\*\*\*\*” innodb\_buffer\_pool\_size=10G innodb\_log\_group\_home=/var/lib/mysql innodb\_log\_file\_size = 5242880 log\_error = /var/log/mysql.log ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:45am UTC](https://forums.percona.com/t/help/3809/10 "2014-10-10T08:45:00Z")
[COLOR=#FF0000]my.cnf node #2 [mysqld] innodb\_file\_per\_table datadir=/var/lib/mysql user=mysql # Path to Galera library wsrep\_provider=/usr/lib64/libgalera\_smm.so # Cluster connection URL contains IPs of node#1, node#2 and node#3 wsrep\_cluster\_address=gcomm://192.99.19.158,192.99.19.154 # In order for Galera to work correctly binlog format should be ROW binlog\_format=ROW # MyISAM storage engine has only experimental support default\_storage\_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb\_autoinc\_lock\_mode=2 # Node #2 address wsrep\_node\_address=192.99.19.154 # Cluster name wsrep\_cluster\_name=my\_centos\_cluster # SST method wsrep\_sst\_method=xtrabackup-v2 #Authentication for SST method wsrep\_sst\_auth=“sstuser:\*\*\*\*\*\*\*\*” innodb\_buffer\_pool\_size=10G innodb\_log\_group\_home=/var/lib/mysql innodb\_log\_file\_size = 5242880 log\_error = /var/log/mysql.log ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 10, 2014, 8:48am UTC](https://forums.percona.com/t/help/3809/11 "2014-10-10T08:48:36Z")
Server specs: [TABLE=“class: wysiwyg\_table\_full wysiwyg\_table\_noBg”] [TR] [TD]Intel Xeon 1x E5-2650v2 8c/16t 2,6 GHz+/3,4 GHz+[/TD] [/TR] [TR] [TD]128 GB DDR3 ECC 1600MHz[/TD] [/TR] [TR] [TD]2x 3 SSD[/TD] [/TR] [/TABLE] ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 11, 2014, 4:20am UTC](https://forums.percona.com/t/help/3809/12 "2014-10-11T04:20:38Z")
[COLOR=#FF0000]Went down again: 2014-10-10 10:06:32 7ff829c7a700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 10:08:23 7ff829c7a700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 10:08:23 7ff829c7a700 InnoDB: Recalculation of persistent statistics requested for table “admin\_1”.“rlk\_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 10:09:33 7ff829c7a700 InnoDB: Error: Table “mysql”.“innodb\_table\_stats” not found. 2014-10-10 10:09:33 7ff829c7a700 InnoDB: Recalculation of persistent statistics requested for table “admin\_6”.“rlk\_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead. 2014-10-10 10:21:08 12794 [Note] WSREP: Provider paused at e29f57c0-4ee8-11e4-9e34-f334de4c3867:3630097 (75162) 2014-10-10 10:21:10 12794 [Note] WSREP: resuming provider at 75162 2014-10-10 10:21:10 12794 [Note] WSREP: Provider resumed. 2014-10-10 10:21:32 12794 [Note] WSREP: Provider paused at e29f57c0-4ee8-11e4-9e34-f334de4c3867:3630465 (75534) 2014-10-10 10:21:34 12794 [Note] WSREP: resuming provider at 75534 2014-10-10 10:21:34 12794 [Note] WSREP: Provider resumed. 2014-10-10 12:06:38 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:06:48 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:06:58 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:07:09 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:07:19 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:07:29 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 13:35:01 12794 [Warning] IP address ‘122.0.114.157’ could not be resolved: Name or service not known 2014-10-11 05:11:35 12794 [Warning] IP address ‘115.231.81.35’ could not be resolved: Name or service not known 2014-10-11 05:11:36 12794 [Warning] Too many connections 2014-10-11 05:11:36 12794 [Warning] Too many connections 2014-10-11 05:11:36 12794 [Warning] Too many connections 2014-10-11 05:11:36 12794 [Warning] Too many connections 2014-10-11 05:11:36 12794 [Warning] Too many connections 2014-10-11 05:11:37 12794 [Warning] Too many connections 2014-10-11 05:11:38 12794 [Warning] Too many connections 2014-10-11 05:11:38 12794 [Warning] Too many connections 2014-10-11 05:11:39 12794 [Warning] Too many connections 2014-10-11 05:11:39 12794 [Warning] Too many connections 2014-10-11 05:11:39 12794 [Warning] Too many connections 2014-10-11 05:11:39 12794 [Warning] Too many connections 2014-10-11 05:11:39 12794 [Warning] Too many connections 2014-10-11 05:11:41 12794 [Warning] Too many connections 2014-10-11 05:11:41 12794 [Warning] Too many connections 2014-10-11 05:11:41 12794 [Warning] Too many connections 2014-10-11 05:11:41 12794 [Warning] Too many connections 2014-10-11 05:11:43 12794 [Warning] Too many connections 2014-10-11 05:11:44 12794 [Warning] Too many connections 2014-10-11 05:11:44 12794 [Warning] Too many connections 2014-10-11 05:11:44 12794 [Warning] Too many connections 2014-10-11 05:11:45 12794 [Warning] Too many connections 2014-10-11 05:11:45 12794 [Warning] Too many connections 2014-10-11 05:11:46 12794 [Warning] Too many connections 2014-10-11 05:11:48 12794 [Warning] Too many connections 2014-10-11 05:11:48 12794 [Warning] Too many connections 2014-10-11 05:12:15 12794 [Warning] Too many connections 2014-10-11 05:12:15 12794 [Warning] Too many connections 2014-10-11 05:12:15 12794 [Warning] Too many connections 2014-10-11 05:12:15 12794 [Warning] Too many connections 2014-10-11 05:12:15 12794 [Warning] Too many connections 2014-10-11 05:12:15 12794 [Warning] Too many connections 2014-10-11 05:12:15 12794 [Warning] Too many connections 2014-10-11 05:12:19 12794 [Warning] Too many connections 2014-10-11 05:12:32 12794 [Warning] Too many connections 2014-10-11 05:12:39 12794 [Warning] Too many connections 2014-10-11 05:12:44 12794 [Warning] Too many connections 2014-10-11 05:12:44 12794 [Warning] Too many connections 2014-10-11 05:13:01 12794 [Warning] Too many connections 2014-10-11 05:13:05 12794 [Warning] Too many connections 2014-10-11 05:16:59 12794 [Warning] Too many connections 2014-10-11 05:17:26 12794 [Warning] Too many connections 2014-10-11 05:17:26 12794 [Warning] Too many connections 2014-10-11 05:17:26 12794 [Warning] Too many connections 2014-10-11 05:17:26 12794 [Warning] Too many connections 09:18:53 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at [URL=“[https://bugs.launchpad.net/percona-xtrad/usr/sbin/mysqld](https://bugs.launchpad.net/percona-xtrad/usr/sbin/mysqld)(\_Z10sub\_selectP4JOINP13st\_join\_tableb+0x1c1”][https://bugs.launchpad.net/percona-x...n\_tableb+0x1c1[/URL]](https://bugs.launchpad.net/percona-x...n_tableb+0x1c1%5B/URL%5D))[0x6c8141] /usr/sbin/mysqld[0x6c7c11] /usr/sbin/mysqld(\_Z10sub\_selectP4JOINP13st\_join\_tableb+0x1c1 )[0x6c8141] /usr/sbin/mysqld(\_ZN4JOIN4execEv+0x2fa)[0x6c89aa] /usr/sbin/mysqld(_Z12mysql\_selectP3THDP10TABLE\_LISTjR4ListI4 ItemEPS4\_P10SQL\_I\_ListI8st\_orderESB\_S7\_yP13select_ resultP18st\_select\_lex\_unitP13st\_select\_lex+0x250)[0x710b50] /usr/sbin/mysqld(\_Z13handle\_selectP3THDP13select\_resultm+0x1 87)[0x7113d7] /usr/sbin/mysqld[0x6e70bd] /usr/sbin/mysqld(\_Z21mysql\_execute\_commandP3THD+0x38b0)[0x6ebe30] /usr/sbin/mysqld(\_Z11mysql\_parseP3THDPcjP12Parser\_state+0x65 8)[0x6eedc8] /usr/sbin/mysqld[0x6eef21] /usr/sbin/mysqld(\_Z16dispatch\_command19enum\_server\_commandP3 THDPcj+0x1362)[0x6f0a92] /usr/sbin/mysqld(\_Z10do\_commandP3THD+0x20a)[0x6f261a] /usr/sbin/mysqld(\_Z24do\_handle\_one\_connectionP3THD+0x17f)[0x6bb06f] /usr/sbin/mysqld(handle\_one\_connection+0x47)[0x6bb247] /usr/sbin/mysqld(pfs\_spawn\_thread+0x12a)[0xaee54a] /lib64/libpthread.so.0(+0x79d1)[0x7ffb1036d9d1] /lib64/libc.so.6(clone+0x6d)[0x7ffb0e87486d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7ff69c398c50): is an invalid pointer Connection ID (thread ID): 431692 Status: NOT\_KILLED You may download the Percona XtraDB Cluster operations manual by visiting [URL][http://www.percona.com/software/percona-xtradb-cluster/[/URL]](http://www.percona.com/software/percona-xtradb-cluster/%5B/URL%5D). You may find information in the manual which will help you identify the cause of the crash. 141011 05:18:53 mysqld\_safe Number of processes running now: 0 141011 05:18:53 mysqld\_safe WSREP: not restarting wsrep node automatically 141011 05:18:53 mysqld\_safe mysqld from pid file /var/lib/mysql/tree.flyheight.com.pid ended ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 11, 2014, 4:22am UTC](https://forums.percona.com/t/help/3809/13 "2014-10-11T04:22:27Z")
Who are those ips, they are not mine IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:06:48 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:06:58 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:07:09 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:07:19 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 12:07:29 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution 2014-10-10 13:35:01 12794 [Warning] IP address ‘122.0.114.157’ could not be resolved: Name or service not known 2014-10-11 05:11:35 12794 [Warning] IP address ‘115.231.81.35’ could not be resolved: Name or service not known ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 12, 2014, 7:19am UTC](https://forums.percona.com/t/help/3809/14 "2014-10-12T07:19:27Z")
[COLOR=#FF0000]CRASHED AGAIN THIS THING DON’T LAST A DAY? DOES ANYBODY KNOWS? IF NOT LET ME KNOW TO DROP YOUR PRODUCT, NOT GONNA PAY 15K FOR SUPPORT FOR ONLY 2 NODES ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 12, 2014, 7:20am UTC](https://forums.percona.com/t/help/3809/15 "2014-10-12T07:20:23Z")
2014-10-12 08:38:46 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 8130 2014-10-12 08:39:31 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 8160 2014-10-12 08:40:16 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 8190 2014-10-12 08:41:01 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 8220 2014-10-12 08:41:43 20658 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-12 08:41:43 20658 [Note] WSREP: Stop replication 2014-10-12 08:41:43 20658 [Note] WSREP: Closing send monitor… 2014-10-12 08:41:43 20658 [Note] WSREP: Closed send monitor. 2014-10-12 08:41:43 20658 [Note] WSREP: gcomm: terminating thread 2014-10-12 08:41:43 20658 [Note] WSREP: gcomm: joining thread 2014-10-12 08:41:43 20658 [Note] WSREP: gcomm: closing backend 2014-10-12 08:41:43 20658 [Note] WSREP: Received self-leave message. 2014-10-12 08:41:43 20658 [Note] WSREP: gcomm: closed 2014-10-12 08:41:43 20658 [Note] WSREP: Flow-control interval: [0, 0] 2014-10-12 08:41:43 20658 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2014-10-12 08:41:43 20658 [Note] WSREP: Shifting OPEN → CLOSED (TO: 7699803) 2014-10-12 08:41:43 20658 [Note] WSREP: RECV thread exiting 0: Success 2014-10-12 08:41:43 20658 [Note] WSREP: New cluster view: global state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:7699803, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3 2014-10-12 08:41:43 20658 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2014-10-12 08:41:43 20658 [Note] WSREP: applier thread exiting (code:0) 2014-10-12 08:41:43 20658 [Note] WSREP: recv\_thread() joined. 2014-10-12 08:41:43 20658 [Note] WSREP: Closing replication queue. 2014-10-12 08:41:43 20658 [Note] WSREP: Closing slave action queue. 2014-10-12 08:41:45 20658 [Note] WSREP: rollbacker thread exiting 2014-10-12 08:41:45 20658 [Note] Giving 0 client threads a chance to die gracefully 2014-10-12 08:41:45 20658 [Note] Event Scheduler: Purging the queue. 0 events 2014-10-12 08:41:45 20658 [Note] Shutting down slave threads 2014-10-12 08:41:45 20658 [Note] Forcefully disconnecting 0 remaining clients 2014-10-12 08:41:45 20658 [Note] WSREP: dtor state: CLOSED 2014-10-12 08:41:45 20658 [Note] WSREP: mon: entered 2063776 oooe fraction 0 oool fraction 0.000787392 2014-10-12 08:41:45 20658 [Note] WSREP: mon: entered 2063776 oooe fraction 0.0380894 oool fraction 0.000788361 2014-10-12 08:41:45 20658 [Note] WSREP: mon: entered 2082237 oooe fraction 0 oool fraction 0.000219956 2014-10-12 08:41:45 20658 [Note] WSREP: cert index usage at exit 0 2014-10-12 08:41:45 20658 [Note] WSREP: cert trx map usage at exit 5644 2014-10-12 08:41:45 20658 [Note] WSREP: deps set usage at exit 0 2014-10-12 08:41:45 20658 [Note] WSREP: avg deps dist 48.432 2014-10-12 08:41:45 20658 [Note] WSREP: avg cert interval 0.156547 2014-10-12 08:41:45 20658 [Note] WSREP: cert index size 1161 2014-10-12 08:41:45 20658 [Note] WSREP: Service thread queue flushed. 2014-10-12 08:41:45 20658 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0 2014-10-12 08:41:45 20658 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0.999768, misses: 361, in use: 0, in pool: 361 2014-10-12 08:41:45 20658 [Note] WSREP: MemPool(SlaveTrxHandle): hit ratio: 0.987945, misses: 6149, in use: 0, in pool: 3797 2014-10-12 08:41:45 20658 [Note] WSREP: Shifting CLOSED → DESTROYED (TO: 7699803) 2014-10-12 08:41:45 20658 [Note] WSREP: Flushing memory map to disk… 2014-10-12 08:41:45 20658 [Note] Binlog end 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘partition’ ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 12, 2014, 7:21am UTC](https://forums.percona.com/t/help/3809/16 "2014-10-12T07:21:10Z")
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘ARCHIVE’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘BLACKHOLE’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘PERFORMANCE\_SCHEMA’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_CHANGED\_PAGES’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_DATAFILES’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESPACES’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN\_COLS’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_FOREIGN’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_FIELDS’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_COLUMNS’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_INDEXES’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLESTATS’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_SYS\_TABLES’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_TABLE’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_FT\_INDEX\_CACHE’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_FT\_CONFIG’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_FT\_BEING\_DELETED’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_FT\_DELETED’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_FT\_DEFAULT\_STOPWORD’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_METRICS’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_BUFFER\_POOL\_STATS’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE\_LRU’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_BUFFER\_PAGE’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX\_RESET’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_CMP\_PER\_INDEX’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_CMPMEM\_RESET’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_CMPMEM’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_CMP\_RESET’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_CMP’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_LOCK\_WAITS’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_LOCKS’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB\_TRX’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘XTRADB\_RSEG’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘XTRADB\_INTERNAL\_HASH\_TABLES’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘XTRADB\_READ\_VIEW’ 2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘InnoDB’ 2014-10-12 08:41:45 20658 [Note] InnoDB: FTS optimize thread exiting. 2014-10-12 08:41:45 20658 [Note] InnoDB: Starting shutdown… 2014-10-12 08:41:50 20658 [Note] InnoDB: Shutdown completed; log sequence number 5267192027 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘CSV’ 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘MyISAM’ 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘MRG\_MYISAM’ 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘MEMORY’ 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘sha256\_password’ 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘mysql\_old\_password’ 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘mysql\_native\_password’ 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘wsrep’ 2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘binlog’ 2014-10-12 08:41:50 20658 [Note] /usr/sbin/mysqld: Shutdown complete 141012 08:41:51 mysqld\_safe mysqld from pid file /var/lib/mysql/tree.flyheight.com.pid ended 141012 08:42:01 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql 141012 08:42:01 mysqld\_safe Skipping wsrep-recover for e29f57c0-4ee8-11e4-9e34-f334de4c3867:7699803 pair 141012 08:42:01 mysqld\_safe Assigning e29f57c0-4ee8-11e4-9e34-f334de4c3867:7699803 to wsrep\_start\_position 2014-10-12 08:42:01 0 [Note] WSREP: wsrep\_start\_position var submitted: ‘e29f57c0-4ee8-11e4-9e34-f334de4c3867:7699803’ 2014-10-12 08:42:01 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit\_defaults\_for\_timestamp server option (see documentation for more details). ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 12, 2014, 7:31am UTC](https://forums.percona.com/t/help/3809/17 "2014-10-12T07:31:19Z")
[COLOR=#FF0000]HERE IS THE WHOLE LOG: [Dropbox - mysql.log - Simplify your life](https://www.dropbox.com/s/pw3azqmywv0lh1s/mysql.log?dl=0) IF YOU CAN NOT HELP PLEASE LET ME KNOW ASAP TO FIND ANOTHER SOLUTION. THANKS ERICK ---
**Author:** ![iraton](https://avatars.discourse-cdn.com/v4/letter/i/9de053/32.png) [@iraton](https://forums.percona.com/u/iraton)\ **Post date:** [October 13, 2014, 8:00am UTC](https://forums.percona.com/t/help/3809/18 "2014-10-13T08:00:13Z")
Still went down again, can anyone help? ---
**Author:** ![przemek](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/przemek/32/3_2.png) [@przemek](https://forums.percona.com/u/przemek)\ **Post date:** [October 28, 2014, 6:19pm UTC](https://forums.percona.com/t/help/3809/19 "2014-10-28T18:19:37Z")
In the log I can see dozens of service shutdowns: ```auto grep 'Normal shutdown' /var/tmp/mysql.log 2014-10-08 18:23:06 12724 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-08 18:42:31 13216 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-08 18:42:55 19088 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-11 05:57:09 19467 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-11 06:04:00 21143 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-11 06:05:23 23334 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-11 06:06:35 24372 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-11 06:08:16 25107 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-11 06:09:56 25904 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-11 06:20:30 26981 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-11 08:03:43 29955 [Note] /usr/sbin/mysqld: Normal shutdown 2014-10-12 08:41:43 20658 [Note] /usr/sbin/mysqld: Normal shutdown ``` which are not making much sense to me - were they caused by some automation tool or intentionally? Also many failed SSTs in the history due to wrong sst auth it seems “–auth ‘(null)’”. Then once proper SST was done (–auth ‘sstuser:s3cret’) we can see later many “WSREP: Provider paused at / WSREP: Provider resumed” which is usually indication of flush tables with read lock being issued, and then shutdown again. At the end there is some network issue between the nodes or just the second node was forcibly killed: 2014-10-12 05:15:26 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://192.99.19.154:4567 2014-10-12 05:17:43 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 90 node is in non-primary state as a result. Given all these, I think there is something messing with the cluster rather then the cluster being the source of problem itself. “Normal shutdown” is not something that PXC server triggers on itself.