Help

Hi, guys

i have succesfully setup Xtradb cluster with a total of 2 nodes

however it ramdonly crashes, can anyone help?

so far the error log has the following:

2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_6”.“rlk_users” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_6”.“rlk_usermeta” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-08 18:46:23 7fdb180a1700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_6”.“rlk_comments” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-08 18:47:16 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-08 18:47:16 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_9”.“wp_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-08 18:47:25 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-08 18:47:25 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.

why would this happen?

thank you for all your help

For some reason this system table seems missing. Did you do any manual work on mysql database? How was the second node created? Can you try with mysql_upgrade script if it fixes it?

Hi, No i have not, they are both fresh with same versions of everything, it lasts 1 or 2 days and then crashes. Last time i ended up restoring database. This time crashed but didnt damage the database here is the error log:

2014-10-10 05:05:57 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:05:57 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_1”.“rlk_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:06:00 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:06:00 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_9”.“wp_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:06:18 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:06:18 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:06:35 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:06:35 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_7”.“wp_commentmeta” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:10:19 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:10:19 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:15 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:16 19556 [Warning] Too many connections
2014-10-10 05:12:17 19556 [Warning] Too many connections
2014-10-10 05:12:17 19556 [Warning] Too many connections
2014-10-10 05:12:17 19556 [Warning] Too many connections
2014-10-10 05:12:17 19556 [Warning] Too many connections
2014-10-10 05:12:17 19556 [Warning] Too many connections
2014-10-10 05:12:17 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:22 19556 [Warning] Too many connections
2014-10-10 05:12:55 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:12:55 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:13:35 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:13:35 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_9”.“wp_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_6”.“rlk_wfHoover” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_6”.“rlk_wfHoover” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-10 05:17:29 7fd8561b9700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:19:00 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:19:00 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_wfScanners” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:19:00 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:19:00 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:21:40 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:21:40 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_1”.“rlk_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 05:23:27 7fd85bfff700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 05:23:27 7fd85bfff700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
09:23:40 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at [url]https://bugs.launchpad.net/percona-xtradb-cluster[/url]

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=152
max_threads=153
thread_count=3
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69257 K bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.

Thread pointer: 0x124f84e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7fd855f2ed38 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8f5835]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x664384]
/lib64/libpthread.so.0(+0xf710)[0x7fdb46db1710]
/usr/sbin/mysqld(_ZNK7handler22ha_statistic_incrementEM17system_status_vary+0x8)[0x59f768]
/usr/sbin/mysqld(_ZN9ha_myisam9write_rowEPh+0x20)141010 05:23:40 mysqld_safe Number of processes running now: 0
141010 05:23:40 mysqld_safe WSREP: not restarting wsrep node automatically
141010 05:23:40 mysqld_safe mysqld from pid file /var/lib/mysql/tree.flyheight.com.pid ended
141010 09:46:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141010 09:46:41 mysqld_safe WSREP: Running position recovery with --log_error=‘/var/lib/mysql/wsrep_recovery.KI7FTN’ --pid-file=‘/var/lib/mysql/tree.flyheight.com-recover.pid’
2014-10-10 09:46:42 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
141010 09:46:46 mysqld_safe WSREP: Recovered position e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693
2014-10-10 09:46:46 0 [Note] WSREP: wsrep_start_position var submitted: ‘e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693’
2014-10-10 09:46:46 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-10-10 09:46:46 12326 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-10-10 09:46:46 12326 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’
2014-10-10 09:46:46 12326 [Note] WSREP: wsrep_load(): Galera 3.7(r7f44a18) by Codership Oy <info@codership.com> loaded successfully.
2014-10-10 09:46:46 12326 [Note] WSREP: CRC-32C: using hardware acceleration.
2014-10-10 09:46:46 12326 [Note] WSREP: Found saved state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:-1
2014-10-10 09:46:46 12326 [Note] WSREP: Passing config to GCS: base_host = 192.99.19.158; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.w
2014-10-10 09:46:46 12326 [Note] WSREP: Service thread queue flushed.
2014-10-10 09:46:46 12326 [Note] WSREP: Assign initial position for certification: 3555693, protocol version: -1
2014-10-10 09:46:46 12326 [Note] WSREP: wsrep_sst_grab()
2014-10-10 09:46:46 12326 [Note] WSREP: Start replication
2014-10-10 09:46:46 12326 [Note] WSREP: Setting initial position to e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693
2014-10-10 09:46:46 12326 [Note] WSREP: protonet asio version 0
2014-10-10 09:46:46 12326 [Note] WSREP: Using CRC-32C for message checksums.
2014-10-10 09:46:46 12326 [Note] WSREP: backend: asio
2014-10-10 09:46:46 12326 [Note] WSREP: restore pc from disk successfully
2014-10-10 09:46:46 12326 [Note] WSREP: GMCast version 0
2014-10-10 09:46:46 12326 [Note] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
2014-10-10 09:46:46 12326 [Note] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
2014-10-10 09:46:46 12326 [Note] WSREP: EVS version 0
2014-10-10 09:46:46 12326 [Note] WSREP: PC version 0
2014-10-10 09:46:46 12326 [Note] WSREP: gcomm: connecting to group ‘my_centos_cluster’, peer ‘192.99.19.158:,192.99.19.154:’
2014-10-10 09:46:46 12326 [Warning] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) address ‘tcp://192.99.19.158:4567’ points to own listening address, blacklisting
2014-10-10 09:46:49 12326 [Warning] WSREP: no nodes coming from prim view, prim not possible
2014-10-10 09:46:49 12326 [Note] WSREP: gcomm: connected
2014-10-10 09:46:49 12326 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2014-10-10 09:46:49 12326 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0)
2014-10-10 09:46:49 12326 [Note] WSREP: Opened channel ‘my_centos_cluster’
2014-10-10 09:46:49 12326 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2014-10-10 09:46:49 12326 [Note] WSREP: Waiting for SST to complete.
2014-10-10 09:46:49 12326 [Note] WSREP: Flow-control interval: [16, 16]
2014-10-10 09:46:49 12326 [Note] WSREP: Received NON-PRIMARY.
2014-10-10 09:46:49 12326 [Note] WSREP: New cluster view: global state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1
2014-10-10 09:46:49 12326 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-10-10 09:46:50 12326 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50128S), skipping check
141010 09:47:37 mysqld_safe mysqld from pid file /var/lib/mysql/tree.flyheight.com.pid ended
141010 09:47:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141010 09:47:40 mysqld_safe Skipping wsrep-recover for e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693 pair
141010 09:47:40 mysqld_safe Assigning e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693 to wsrep_start_position
2014-10-10 09:47:40 0 [Note] WSREP: wsrep_start_position var submitted: ‘e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693’
2014-10-10 09:47:40 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

2014-10-10 09:47:40 12794 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-10-10 09:47:40 12794 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’
2014-10-10 09:47:40 12794 [Note] WSREP: wsrep_load(): Galera 3.7(r7f44a18) by Codership Oy <info@codership.com> loaded successfully.
2014-10-10 09:47:40 12794 [Note] WSREP: CRC-32C: using hardware acceleration.
2014-10-10 09:47:40 12794 [Note] WSREP: Found saved state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693
2014-10-10 09:47:40 12794 [Note] WSREP: Passing config to GCS: base_host = 192.99.19.158; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.w
2014-10-10 09:47:41 12794 [Note] WSREP: Service thread queue flushed.
2014-10-10 09:47:41 12794 [Note] WSREP: Assign initial position for certification: 3555693, protocol version: -1
2014-10-10 09:47:41 12794 [Note] WSREP: wsrep_sst_grab()
2014-10-10 09:47:41 12794 [Note] WSREP: Start replication
2014-10-10 09:47:41 12794 [Note] WSREP: Setting initial position to e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693
2014-10-10 09:47:41 12794 [Note] WSREP: protonet asio version 0
2014-10-10 09:47:41 12794 [Note] WSREP: Using CRC-32C for message checksums.
2014-10-10 09:47:41 12794 [Note] WSREP: backend: asio
2014-10-10 09:47:41 12794 [Note] WSREP: restore pc from disk successfully
2014-10-10 09:47:41 12794 [Note] WSREP: GMCast version 0
2014-10-10 09:47:41 12794 [Note] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
2014-10-10 09:47:41 12794 [Note] WSREP: (778c1134, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
2014-10-10 09:47:41 12794 [Note] WSREP: EVS version 0
2014-10-10 09:47:41 12794 [Note] WSREP: PC version 0
2014-10-10 09:47:41 12794 [Note] WSREP: gcomm: bootstrapping new group ‘my_centos_cluster’
2014-10-10 09:47:41 12794 [Note] WSREP: start_prim is enabled, turn off pc_recovery
2014-10-10 09:47:41 12794 [Note] WSREP: Node 778c1134 state prim
2014-10-10 09:47:41 12794 [Note] WSREP: save pc into disk
2014-10-10 09:47:41 12794 [Note] WSREP: discarding pending addr without UUID: tcp://192.99.19.154:4567
2014-10-10 09:47:41 12794 [Note] WSREP: discarding pending addr proto entry 0x14d9930
2014-10-10 09:47:41 12794 [Note] WSREP: discarding pending addr without UUID: tcp://192.99.19.158:4567
2014-10-10 09:47:41 12794 [Note] WSREP: discarding pending addr proto entry 0x14e20f0
2014-10-10 09:47:41 12794 [Note] WSREP: clear restored view
2014-10-10 09:47:41 12794 [Note] WSREP: gcomm: connected
2014-10-10 09:47:41 12794 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2014-10-10 09:47:41 12794 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0)
2014-10-10 09:47:41 12794 [Note] WSREP: Opened channel ‘my_centos_cluster’
2014-10-10 09:47:41 12794 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2014-10-10 09:47:41 12794 [Note] WSREP: Waiting for SST to complete.
2014-10-10 09:47:41 12794 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 00f84f3f-5084-11e4-a61d-97674bc48724
2014-10-10 09:47:41 12794 [Note] WSREP: STATE EXCHANGE: sent state msg: 00f84f3f-5084-11e4-a61d-97674bc48724
2014-10-10 09:47:41 12794 [Note] WSREP: STATE EXCHANGE: got state msg: 00f84f3f-5084-11e4-a61d-97674bc48724 from 0 (tree.flyheight.com)
2014-10-10 09:47:41 12794 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 0,
members = 1/1 (joined/total),
act_id = 3555693,
last_appl. = -1,
protocols = 0/6/3 (gcs/repl/appl),
group UUID = e29f57c0-4ee8-11e4-9e34-f334de4c3867
2014-10-10 09:47:41 12794 [Note] WSREP: Flow-control interval: [16, 16]
2014-10-10 09:47:41 12794 [Note] WSREP: Restored state OPEN → JOINED (3555693)
2014-10-10 09:47:41 12794 [Note] WSREP: Member 0.0 (tree.flyheight.com) synced with group.
2014-10-10 09:47:41 12794 [Note] WSREP: Shifting JOINED → SYNCED (TO: 3555693)
2014-10-10 09:47:41 12794 [Note] WSREP: New cluster view: global state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:3555693, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3
2014-10-10 09:47:41 12794 [Note] WSREP: SST complete, seqno: 3555693
2014-10-10 09:47:41 12794 [Warning] Using unique option prefix innodb_log_group_home instead of innodb-log-group-home-dir is deprecated and will be removed in a future release. Please use the full name instead.
2014-10-10 09:47:41 12794 [Note] Plugin ‘FEDERATED’ is disabled.
2014-10-10 09:47:41 12794 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-10-10 09:47:41 12794 [Note] InnoDB: The InnoDB memory heap is disabled
2014-10-10 09:47:41 12794 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-10-10 09:47:41 12794 [Note] InnoDB: Memory barrier is not used
2014-10-10 09:47:41 12794 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-10-10 09:47:41 12794 [Note] InnoDB: Using Linux native AIO
2014-10-10 09:47:41 12794 [Note] InnoDB: Using CPU crc32 instructions
2014-10-10 09:47:41 12794 [Note] InnoDB: Initializing buffer pool, size = 10.0G
2014-10-10 09:47:41 12794 [Note] InnoDB: Completed initialization of buffer pool
2014-10-10 09:47:41 12794 [Note] InnoDB: Highest supported file format is Barracuda.

2014-10-10 09:47:41 12794 [Note] InnoDB: 128 rollback segment(s) are active.
2014-10-10 09:47:41 12794 [Note] InnoDB: Waiting for purge to start
2014-10-10 09:47:41 12794 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 2564845042
2014-10-10 09:47:41 12794 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2014-10-10 09:47:41 12794 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2014-10-10 09:47:41 12794 [Note] Server hostname (bind-address): ‘*’; port: 3306
2014-10-10 09:47:41 12794 [Note] IPv6 is available.
2014-10-10 09:47:41 12794 [Note] - ‘::’ resolves to ‘::’;
2014-10-10 09:47:41 12794 [Note] Server socket created on IP: ‘::’.
2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.events_waits_current is wrong. Expected 19, found 16. Created with MySQL 50540, now running 50620. Please use mysql_upgrade to fix this error.
2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.events_waits_history is wrong. Expected 19, found 16. Created with MySQL 50540, now running 50620. Please use mysql_upgrade to fix this error.
2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.events_waits_history_long is wrong. Expected 19, found 16. Created with MySQL 50540, now running 50620. Please use mysql_upgrade to fix this error.
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_waits_summary_by_host_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Incorrect definition of table performance_schema.events_waits_summary_by_thread_by_event_name: expected column ‘THREAD_ID’ at position 0 to have type bigint(20), found type int(11).
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_waits_summary_by_user_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_waits_summary_by_account_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.file_summary_by_event_name is wrong. Expected 23, found 5. Created with MySQL 50540, now running 50620. Please use mysql_upgrade to fix this error.
2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.file_summary_by_instance is wrong. Expected 25, found 6. Created with MySQL 50540, now running 50620. Please use mysql_upgrade to fix this error.
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘host_cache’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Incorrect definition of table performance_schema.mutex_instances: expected column ‘LOCKED_BY_THREAD_ID’ at position 2 to have type bigint(20), found type int(11).
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘objects_summary_global_by_type’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Incorrect definition of table performance_schema.rwlock_instances: expected column ‘WRITE_LOCKED_BY_THREAD_ID’ at position 2 to have type bigint(20), found type int(11).
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘setup_actors’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘setup_objects’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘table_io_waits_summary_by_index_usage’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘table_io_waits_summary_by_table’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘table_lock_waits_summary_by_table’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Column count of mysql.threads is wrong. Expected 14, found 3. Created with MySQL 50540, now running 50620. Please use mysql_upgrade to fix this error.
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_stages_current’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_stages_history’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_stages_history_long’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_stages_summary_by_thread_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_stages_summary_by_account_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_stages_summary_by_user_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_stages_summary_by_host_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_stages_summary_global_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_current’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_history’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_history_long’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_summary_by_thread_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_summary_by_account_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_summary_by_user_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_summary_by_host_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_summary_global_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘events_statements_summary_by_digest’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘users’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘accounts’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘hosts’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘socket_instances’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘socket_summary_by_instance’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘socket_summary_by_event_name’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘session_connect_attrs’ has the wrong structure
2014-10-10 09:47:41 12794 [ERROR] Native table ‘performance_schema’.‘session_account_connect_attrs’ has the wrong structure
2014-10-10 09:47:41 12794 [Note] Event Scheduler: Loaded 0 events
2014-10-10 09:47:41 12794 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘5.6.20-68.0-56’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel68.0, Revision 888, WSREP version 25.7, wsrep_25.7.r4126
2014-10-10 09:47:41 12794 [Note] WSREP: inited wsrep sidno 1
2014-10-10 09:47:41 12794 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-10-10 09:47:41 12794 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-10-10 09:47:41 12794 [Note] WSREP: Service thread queue flushed.
2014-10-10 09:47:41 12794 [Note] WSREP: Assign initial position for certification: 3555693, protocol version: 3
2014-10-10 09:47:41 12794 [Note] WSREP: Service thread queue flushed.
2014-10-10 09:47:41 12794 [Note] WSREP: Synchronized with group, ready for connections
2014-10-10 09:47:41 12794 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_3”.“wp_options” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_3”.“wp_postmeta” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_3”.“wp_posts” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_3”.“wp_terms” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 09:47:48 7ff828c37700 InnoDB: Error: Fetch of persistent statistics requested for table “admin_3”.“wp_term_taxonomy” but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.

[COLOR=#FF0000][root@tree ~]# mysql_upgrade
Looking for ‘mysql’ as: mysql
Looking for ‘mysqlcheck’ as: mysqlcheck
Running 'mysqlcheck with default connection arguments
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck with default connection arguments
Warning: Using a password on the command line interface can be insecure.
mysql.columns_priv OK
mysql.db OK
mysql.event OK
mysql.func OK
mysql.general_log OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.proxies_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user OK
Running ‘mysql_fix_privilege_tables’…
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck with default connection arguments
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck with default connection arguments
Warning: Using a password on the command line interface can be insecure.
admin_1.rlk_aiowps_failed_logins OK
admin_1.rlk_aiowps_global_meta OK
admin_1.rlk_aiowps_login_activity OK
admin_1.rlk_aiowps_login_lockdown OK
admin_1.rlk_bpspro_login_security OK
admin_1.rlk_bpspro_seclog_ignore OK
admin_1.rlk_cntctfrm_field OK
admin_1.rlk_commentmeta OK
admin_1.rlk_comments OK
admin_1.rlk_fbthumbnails OK
admin_1.rlk_links OK
admin_1.rlk_lockdowns OK
admin_1.rlk_login_fails OK
admin_1.rlk_ocmx_comment_meta OK
admin_1.rlk_ocmx_gallery OK
admin_1.rlk_ocmx_gallery_dtl OK
admin_1.rlk_options OK
admin_1.rlk_pollsa OK
admin_1.rlk_pollsip OK
admin_1.rlk_pollsq OK
admin_1.rlk_postlikes OK
admin_1.rlk_postmeta OK
admin_1.rlk_posts OK
admin_1.rlk_pvc_daily OK
admin_1.rlk_pvc_total OK
admin_1.rlk_sabre_table OK
admin_1.rlk_term_relationships OK
admin_1.rlk_term_taxonomy OK
admin_1.rlk_terms OK
admin_1.rlk_usermeta OK
admin_1.rlk_users OK
admin_1.rlk_wfBadLeechers OK
admin_1.rlk_wfBlocks OK
admin_1.rlk_wfBlocksAdv OK
admin_1.rlk_wfConfig OK
admin_1.rlk_wfCrawlers OK
admin_1.rlk_wfFileMods OK
admin_1.rlk_wfHits OK
admin_1.rlk_wfHoover OK
admin_1.rlk_wfIssues OK
admin_1.rlk_wfLeechers OK
admin_1.rlk_wfLockedOut OK
admin_1.rlk_wfLocs OK
admin_1.rlk_wfLogins OK
admin_1.rlk_wfNet404s OK
admin_1.rlk_wfReverseCache OK
admin_1.rlk_wfScanners OK
admin_1.rlk_wfStatus OK
admin_1.rlk_wfThrottleLog OK
admin_1.rlk_wfVulnScanners OK
admin_1.wpbandit_images OK
admin_10.wp_cntctfrm_field OK
admin_10.wp_commentmeta OK
admin_10.wp_comments OK
admin_10.wp_links OK
admin_10.wp_nxs_log OK
admin_10.wp_options OK
admin_10.wp_postmeta OK
admin_10.wp_posts OK
admin_10.wp_term_relationships OK
admin_10.wp_term_taxonomy OK
admin_10.wp_terms OK
admin_10.wp_usermeta OK
admin_10.wp_users OK
admin_10.wp_wfBadLeechers OK
admin_10.wp_wfBlocks OK
admin_10.wp_wfBlocksAdv OK
admin_10.wp_wfConfig OK
admin_10.wp_wfCrawlers OK
admin_10.wp_wfFileMods OK
admin_10.wp_wfHits OK
admin_10.wp_wfHoover OK
admin_10.wp_wfIssues OK
admin_10.wp_wfLeechers OK
admin_10.wp_wfLockedOut OK
admin_10.wp_wfLocs OK
admin_10.wp_wfLogins OK
admin_10.wp_wfNet404s OK
admin_10.wp_wfReverseCache OK
admin_10.wp_wfScanners OK
admin_10.wp_wfStatus OK
admin_10.wp_wfThrottleLog OK
admin_10.wp_wfVulnScanners OK
admin_10.wp_wpo_campaign OK
admin_10.wp_wpo_campaign_category OK
admin_10.wp_wpo_campaign_feed OK
admin_10.wp_wpo_campaign_post OK
admin_10.wp_wpo_campaign_word OK
admin_10.wp_wpo_log OK
admin_10.wp_wppa_albums OK
admin_10.wp_wppa_comments OK
admin_10.wp_wppa_exif OK
admin_10.wp_wppa_index OK
admin_10.wp_wppa_iptc OK
admin_10.wp_wppa_photos OK
admin_10.wp_wppa_rating OK
admin_10.wpbandit_images OK
admin_3.nuevo__htmlad OK
admin_3.nuevo__midroll OK
admin_3.nuevo__midroll_settings OK
admin_3.nuevo__overlay OK
admin_3.nuevo__peel OK
admin_3.nuevo__player OK
admin_3.nuevo__preroll OK
admin_3.nuevo__ticker OK
admin_3.wp_cntctfrm_field OK
admin_3.wp_cntctfrm_location OK
admin_3.wp_commentmeta OK
admin_3.wp_comments OK
admin_3.wp_hdflvvideoshare OK
admin_3.wp_hdflvvideoshare_med2play OK
admin_3.wp_hdflvvideoshare_playlist OK
admin_3.wp_hdflvvideoshare_settings OK
admin_3.wp_hdflvvideoshare_tags OK
admin_3.wp_hdflvvideoshare_vgads OK
admin_3.wp_links OK
admin_3.wp_login_security_solution_fail OK
admin_3.wp_nxs_log OK
admin_3.wp_ocmx_comment_meta OK
admin_3.wp_ocmx_gallery OK
admin_3.wp_ocmx_gallery_dtl OK
admin_3.wp_options OK
admin_3.wp_postlikes OK
admin_3.wp_postmeta OK
admin_3.wp_posts OK
admin_3.wp_pvc_daily OK
admin_3.wp_pvc_total OK
admin_3.wp_social_users OK
admin_3.wp_term_relationships OK
admin_3.wp_term_taxonomy OK
admin_3.wp_terms OK
admin_3.wp_usermeta OK
admin_3.wp_users OK
admin_3.wpvidz_commentmeta OK
admin_3.wpvidz_comments OK
admin_3.wpvidz_fbthumbnails OK

admin_3.wpvidz_links OK
admin_3.wpvidz_nuevo__htmlad OK
admin_3.wpvidz_nuevo__midroll OK
admin_3.wpvidz_nuevo__midroll_settings OK
admin_3.wpvidz_nuevo__overlay OK
admin_3.wpvidz_nuevo__peel OK
admin_3.wpvidz_nuevo__player OK
admin_3.wpvidz_nuevo__preroll OK
admin_3.wpvidz_nuevo__ticker OK
admin_3.wpvidz_options OK
admin_3.wpvidz_postlikes OK
admin_3.wpvidz_postmeta OK
admin_3.wpvidz_posts OK
admin_3.wpvidz_term_relationships OK
admin_3.wpvidz_term_taxonomy OK
admin_3.wpvidz_terms OK
admin_3.wpvidz_usermeta OK
admin_3.wpvidz_users OK
admin_4.wpvidz_commentmeta OK
admin_4.wpvidz_comments OK
admin_4.wpvidz_fbthumbnails OK
admin_4.wpvidz_links OK
admin_4.wpvidz_options OK
admin_4.wpvidz_postlikes OK
admin_4.wpvidz_postmeta OK
admin_4.wpvidz_posts OK
admin_4.wpvidz_term_relationships OK
admin_4.wpvidz_term_taxonomy OK
admin_4.wpvidz_terms OK
admin_4.wpvidz_usermeta OK
admin_4.wpvidz_users OK
admin_5.wp_cntctfrm_field OK
admin_5.wp_commentmeta OK
admin_5.wp_comments OK
admin_5.wp_links OK
admin_5.wp_nxs_log OK
admin_5.wp_options OK
admin_5.wp_postlikes OK
admin_5.wp_postmeta OK
admin_5.wp_posts OK
admin_5.wp_term_relationships OK
admin_5.wp_term_taxonomy OK
admin_5.wp_terms OK
admin_5.wp_usermeta OK
admin_5.wp_users OK
admin_5.wp_wfBadLeechers OK
admin_5.wp_wfBlocks OK
admin_5.wp_wfBlocksAdv OK
admin_5.wp_wfConfig OK
admin_5.wp_wfCrawlers OK
admin_5.wp_wfFileMods OK
admin_5.wp_wfHits OK
admin_5.wp_wfHoover OK
admin_5.wp_wfIssues OK
admin_5.wp_wfLeechers OK
admin_5.wp_wfLockedOut OK
admin_5.wp_wfLocs OK
admin_5.wp_wfLogins OK
admin_5.wp_wfNet404s OK
admin_5.wp_wfReverseCache OK
admin_5.wp_wfScanners OK
admin_5.wp_wfStatus OK
admin_5.wp_wfThrottleLog OK
admin_5.wp_wfVulnScanners OK
admin_5.wp_wpo_campaign OK
admin_5.wp_wpo_campaign_category OK
admin_5.wp_wpo_campaign_feed OK
admin_5.wp_wpo_campaign_post OK
admin_5.wp_wpo_campaign_word OK
admin_5.wp_wpo_log OK
admin_5.wp_wppa_albums OK
admin_5.wp_wppa_comments OK
admin_5.wp_wppa_exif OK
admin_5.wp_wppa_index OK
admin_5.wp_wppa_iptc OK
admin_5.wp_wppa_photos OK
admin_5.wp_wppa_rating OK
admin_5.wpbandit_images OK
admin_6.rlk_aiowps_failed_logins OK
admin_6.rlk_aiowps_global_meta OK
admin_6.rlk_aiowps_login_activity OK
admin_6.rlk_aiowps_login_lockdown OK
admin_6.rlk_bpspro_login_security OK
admin_6.rlk_bpspro_seclog_ignore OK
admin_6.rlk_cntctfrm_field OK
admin_6.rlk_commentmeta OK
admin_6.rlk_comments OK
admin_6.rlk_fbthumbnails OK
admin_6.rlk_links OK
admin_6.rlk_lockdowns OK
admin_6.rlk_login_fails OK
admin_6.rlk_ocmx_comment_meta OK
admin_6.rlk_ocmx_gallery OK
admin_6.rlk_ocmx_gallery_dtl OK
admin_6.rlk_options OK
admin_6.rlk_pollsa OK
admin_6.rlk_pollsip OK
admin_6.rlk_pollsq OK
admin_6.rlk_postlikes OK
admin_6.rlk_postmeta OK
admin_6.rlk_posts OK
admin_6.rlk_pvc_daily OK
admin_6.rlk_pvc_total OK
admin_6.rlk_sabre_table OK
admin_6.rlk_term_relationships OK
admin_6.rlk_term_taxonomy OK
admin_6.rlk_terms OK
admin_6.rlk_usermeta OK
admin_6.rlk_users OK
admin_6.rlk_websitez_stats OK
admin_6.rlk_wfBadLeechers OK
admin_6.rlk_wfBlocks OK
admin_6.rlk_wfBlocksAdv OK
admin_6.rlk_wfConfig OK
admin_6.rlk_wfCrawlers OK
admin_6.rlk_wfFileMods OK
admin_6.rlk_wfHits OK
admin_6.rlk_wfHoover OK
admin_6.rlk_wfIssues OK
admin_6.rlk_wfLeechers OK
admin_6.rlk_wfLockedOut OK
admin_6.rlk_wfLocs OK
admin_6.rlk_wfLogins OK
admin_6.rlk_wfNet404s OK
admin_6.rlk_wfReverseCache OK
admin_6.rlk_wfScanners OK
admin_6.rlk_wfStatus OK
admin_6.rlk_wfThrottleLog OK
admin_6.rlk_wfVulnScanners OK
admin_6.wpbandit_images OK
admin_7.wp_commentmeta OK
admin_7.wp_comments OK
admin_7.wp_links OK
admin_7.wp_options OK
admin_7.wp_postlikes OK
admin_7.wp_postmeta OK
admin_7.wp_posts OK
admin_7.wp_term_relationships OK
admin_7.wp_term_taxonomy OK
admin_7.wp_terms OK
admin_7.wp_usermeta OK
admin_7.wp_users OK
admin_8.wp_commentmeta OK
admin_8.wp_comments OK
admin_8.wp_links OK
admin_8.wp_ocmx_comment_meta OK
admin_8.wp_ocmx_gallery OK
admin_8.wp_ocmx_gallery_dtl OK
admin_8.wp_options OK
admin_8.wp_postlikes OK
admin_8.wp_postmeta OK
admin_8.wp_posts OK
admin_8.wp_term_relationships OK
admin_8.wp_term_taxonomy OK
admin_8.wp_terms OK
admin_8.wp_usermeta OK
admin_8.wp_users OK
admin_9.wp_commentmeta OK
admin_9.wp_comments OK
admin_9.wp_links OK
admin_9.wp_ocmx_comment_meta OK
admin_9.wp_ocmx_gallery OK
admin_9.wp_ocmx_gallery_dtl OK
admin_9.wp_options OK
admin_9.wp_postlikes OK
admin_9.wp_postmeta OK
admin_9.wp_posts OK
admin_9.wp_term_relationships OK
admin_9.wp_term_taxonomy OK
admin_9.wp_terms OK
admin_9.wp_usermeta OK
admin_9.wp_users OK
OK

[COLOR=#FF0000]This is my.cnf on node #1
[mysqld]
innodb_file_per_table
datadir=/var/lib/mysql
user=mysql

Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

Cluster connection URL contains IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://192.99.19.158,192.99.19.154

In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

Node #2 address

wsrep_node_address=192.99.19.158

Cluster name

wsrep_cluster_name=my_centos_cluster

SST method

wsrep_sst_method=xtrabackup-v2

#Authentication for SST method
wsrep_sst_auth=“sstuser:*******”

innodb_buffer_pool_size=10G
innodb_log_group_home=/var/lib/mysql
innodb_log_file_size = 5242880

log_error = /var/log/mysql.log

[COLOR=#FF0000]my.cnf node #2
[mysqld]
innodb_file_per_table
datadir=/var/lib/mysql
user=mysql

Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

Cluster connection URL contains IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://192.99.19.158,192.99.19.154

In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

Node #2 address

wsrep_node_address=192.99.19.154

Cluster name

wsrep_cluster_name=my_centos_cluster

SST method

wsrep_sst_method=xtrabackup-v2

#Authentication for SST method
wsrep_sst_auth=“sstuser:********”

innodb_buffer_pool_size=10G
innodb_log_group_home=/var/lib/mysql
innodb_log_file_size = 5242880

log_error = /var/log/mysql.log

Server specs: [TABLE=“class: wysiwyg_table_full wysiwyg_table_noBg”]
[TR]
[TD]Intel Xeon
1x E5-2650v2
8c/16t 2,6 GHz+/3,4 GHz+[/TD]
[/TR]
[TR]
[TD]128 GB DDR3 ECC 1600MHz[/TD]
[/TR]
[TR]
[TD]2x 3 SSD[/TD]
[/TR]
[/TABLE]

[COLOR=#FF0000]Went down again:

2014-10-10 10:06:32 7ff829c7a700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 10:08:23 7ff829c7a700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 10:08:23 7ff829c7a700 InnoDB: Recalculation of persistent statistics requested for table “admin_1”.“rlk_options” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 10:09:33 7ff829c7a700 InnoDB: Error: Table “mysql”.“innodb_table_stats” not found.
2014-10-10 10:09:33 7ff829c7a700 InnoDB: Recalculation of persistent statistics requested for table “admin_6”.“rlk_wfNet404s” but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2014-10-10 10:21:08 12794 [Note] WSREP: Provider paused at e29f57c0-4ee8-11e4-9e34-f334de4c3867:3630097 (75162)
2014-10-10 10:21:10 12794 [Note] WSREP: resuming provider at 75162
2014-10-10 10:21:10 12794 [Note] WSREP: Provider resumed.
2014-10-10 10:21:32 12794 [Note] WSREP: Provider paused at e29f57c0-4ee8-11e4-9e34-f334de4c3867:3630465 (75534)
2014-10-10 10:21:34 12794 [Note] WSREP: resuming provider at 75534
2014-10-10 10:21:34 12794 [Note] WSREP: Provider resumed.
2014-10-10 12:06:38 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:06:48 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:06:58 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:07:09 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:07:19 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:07:29 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 13:35:01 12794 [Warning] IP address ‘122.0.114.157’ could not be resolved: Name or service not known
2014-10-11 05:11:35 12794 [Warning] IP address ‘115.231.81.35’ could not be resolved: Name or service not known
2014-10-11 05:11:36 12794 [Warning] Too many connections
2014-10-11 05:11:36 12794 [Warning] Too many connections
2014-10-11 05:11:36 12794 [Warning] Too many connections
2014-10-11 05:11:36 12794 [Warning] Too many connections
2014-10-11 05:11:36 12794 [Warning] Too many connections
2014-10-11 05:11:37 12794 [Warning] Too many connections
2014-10-11 05:11:38 12794 [Warning] Too many connections
2014-10-11 05:11:38 12794 [Warning] Too many connections
2014-10-11 05:11:39 12794 [Warning] Too many connections
2014-10-11 05:11:39 12794 [Warning] Too many connections
2014-10-11 05:11:39 12794 [Warning] Too many connections
2014-10-11 05:11:39 12794 [Warning] Too many connections
2014-10-11 05:11:39 12794 [Warning] Too many connections
2014-10-11 05:11:41 12794 [Warning] Too many connections
2014-10-11 05:11:41 12794 [Warning] Too many connections
2014-10-11 05:11:41 12794 [Warning] Too many connections
2014-10-11 05:11:41 12794 [Warning] Too many connections
2014-10-11 05:11:43 12794 [Warning] Too many connections
2014-10-11 05:11:44 12794 [Warning] Too many connections
2014-10-11 05:11:44 12794 [Warning] Too many connections
2014-10-11 05:11:44 12794 [Warning] Too many connections
2014-10-11 05:11:45 12794 [Warning] Too many connections
2014-10-11 05:11:45 12794 [Warning] Too many connections
2014-10-11 05:11:46 12794 [Warning] Too many connections
2014-10-11 05:11:48 12794 [Warning] Too many connections
2014-10-11 05:11:48 12794 [Warning] Too many connections
2014-10-11 05:12:15 12794 [Warning] Too many connections
2014-10-11 05:12:15 12794 [Warning] Too many connections
2014-10-11 05:12:15 12794 [Warning] Too many connections
2014-10-11 05:12:15 12794 [Warning] Too many connections
2014-10-11 05:12:15 12794 [Warning] Too many connections
2014-10-11 05:12:15 12794 [Warning] Too many connections
2014-10-11 05:12:15 12794 [Warning] Too many connections
2014-10-11 05:12:19 12794 [Warning] Too many connections
2014-10-11 05:12:32 12794 [Warning] Too many connections
2014-10-11 05:12:39 12794 [Warning] Too many connections
2014-10-11 05:12:44 12794 [Warning] Too many connections
2014-10-11 05:12:44 12794 [Warning] Too many connections
2014-10-11 05:13:01 12794 [Warning] Too many connections
2014-10-11 05:13:05 12794 [Warning] Too many connections
2014-10-11 05:16:59 12794 [Warning] Too many connections
2014-10-11 05:17:26 12794 [Warning] Too many connections
2014-10-11 05:17:26 12794 [Warning] Too many connections
2014-10-11 05:17:26 12794 [Warning] Too many connections
2014-10-11 05:17:26 12794 [Warning] Too many connections
09:18:53 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at [URL=“https://bugs.launchpad.net/percona-xtrad/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1c1”]https://bugs.launchpad.net/percona-x...n_tableb+0x1c1[/URL])[0x6c8141]
/usr/sbin/mysqld[0x6c7c11]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1c1 )[0x6c8141]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x2fa)[0x6c89aa]
/usr/sbin/mysqld(Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4 ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select resultP18st_select_lex_unitP13st_select_lex+0x250)[0x710b50]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x1 87)[0x7113d7]
/usr/sbin/mysqld[0x6e70bd]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x38b0)[0x6ebe30]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x65 8)[0x6eedc8]
/usr/sbin/mysqld[0x6eef21]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3 THDPcj+0x1362)[0x6f0a92]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x20a)[0x6f261a]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x17f)[0x6bb06f]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6bb247]
/usr/sbin/mysqld(pfs_spawn_thread+0x12a)[0xaee54a]
/lib64/libpthread.so.0(+0x79d1)[0x7ffb1036d9d1]
/lib64/libc.so.6(clone+0x6d)[0x7ffb0e87486d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7ff69c398c50): is an invalid pointer
Connection ID (thread ID): 431692
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
[URL]http://www.percona.com/software/percona-xtradb-cluster/[/URL]. You may find information
in the manual which will help you identify the cause of the crash.
141011 05:18:53 mysqld_safe Number of processes running now: 0
141011 05:18:53 mysqld_safe WSREP: not restarting wsrep node automatically
141011 05:18:53 mysqld_safe mysqld from pid file /var/lib/mysql/tree.flyheight.com.pid ended

Who are those ips, they are not mine
IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:06:48 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:06:58 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:07:09 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:07:19 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 12:07:29 12794 [Warning] IP address ‘104.194.20.90’ could not be resolved: Temporary failure in name resolution
2014-10-10 13:35:01 12794 [Warning] IP address ‘122.0.114.157’ could not be resolved: Name or service not known
2014-10-11 05:11:35 12794 [Warning] IP address ‘115.231.81.35’ could not be resolved: Name or service not known

[COLOR=#FF0000]CRASHED AGAIN THIS THING DON’T LAST A DAY? DOES ANYBODY KNOWS? IF NOT LET ME KNOW TO DROP YOUR PRODUCT, NOT GONNA PAY 15K FOR SUPPORT FOR ONLY 2 NODES

2014-10-12 08:38:46 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 8130
2014-10-12 08:39:31 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 8160
2014-10-12 08:40:16 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 8190
2014-10-12 08:41:01 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 8220
2014-10-12 08:41:43 20658 [Note] /usr/sbin/mysqld: Normal shutdown

2014-10-12 08:41:43 20658 [Note] WSREP: Stop replication
2014-10-12 08:41:43 20658 [Note] WSREP: Closing send monitor…
2014-10-12 08:41:43 20658 [Note] WSREP: Closed send monitor.
2014-10-12 08:41:43 20658 [Note] WSREP: gcomm: terminating thread
2014-10-12 08:41:43 20658 [Note] WSREP: gcomm: joining thread
2014-10-12 08:41:43 20658 [Note] WSREP: gcomm: closing backend
2014-10-12 08:41:43 20658 [Note] WSREP: Received self-leave message.
2014-10-12 08:41:43 20658 [Note] WSREP: gcomm: closed
2014-10-12 08:41:43 20658 [Note] WSREP: Flow-control interval: [0, 0]
2014-10-12 08:41:43 20658 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2014-10-12 08:41:43 20658 [Note] WSREP: Shifting OPEN → CLOSED (TO: 7699803)
2014-10-12 08:41:43 20658 [Note] WSREP: RECV thread exiting 0: Success
2014-10-12 08:41:43 20658 [Note] WSREP: New cluster view: global state: e29f57c0-4ee8-11e4-9e34-f334de4c3867:7699803, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3
2014-10-12 08:41:43 20658 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-10-12 08:41:43 20658 [Note] WSREP: applier thread exiting (code:0)
2014-10-12 08:41:43 20658 [Note] WSREP: recv_thread() joined.
2014-10-12 08:41:43 20658 [Note] WSREP: Closing replication queue.
2014-10-12 08:41:43 20658 [Note] WSREP: Closing slave action queue.
2014-10-12 08:41:45 20658 [Note] WSREP: rollbacker thread exiting
2014-10-12 08:41:45 20658 [Note] Giving 0 client threads a chance to die gracefully
2014-10-12 08:41:45 20658 [Note] Event Scheduler: Purging the queue. 0 events
2014-10-12 08:41:45 20658 [Note] Shutting down slave threads
2014-10-12 08:41:45 20658 [Note] Forcefully disconnecting 0 remaining clients
2014-10-12 08:41:45 20658 [Note] WSREP: dtor state: CLOSED
2014-10-12 08:41:45 20658 [Note] WSREP: mon: entered 2063776 oooe fraction 0 oool fraction 0.000787392
2014-10-12 08:41:45 20658 [Note] WSREP: mon: entered 2063776 oooe fraction 0.0380894 oool fraction 0.000788361
2014-10-12 08:41:45 20658 [Note] WSREP: mon: entered 2082237 oooe fraction 0 oool fraction 0.000219956
2014-10-12 08:41:45 20658 [Note] WSREP: cert index usage at exit 0
2014-10-12 08:41:45 20658 [Note] WSREP: cert trx map usage at exit 5644
2014-10-12 08:41:45 20658 [Note] WSREP: deps set usage at exit 0
2014-10-12 08:41:45 20658 [Note] WSREP: avg deps dist 48.432
2014-10-12 08:41:45 20658 [Note] WSREP: avg cert interval 0.156547
2014-10-12 08:41:45 20658 [Note] WSREP: cert index size 1161
2014-10-12 08:41:45 20658 [Note] WSREP: Service thread queue flushed.
2014-10-12 08:41:45 20658 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
2014-10-12 08:41:45 20658 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0.999768, misses: 361, in use: 0, in pool: 361
2014-10-12 08:41:45 20658 [Note] WSREP: MemPool(SlaveTrxHandle): hit ratio: 0.987945, misses: 6149, in use: 0, in pool: 3797
2014-10-12 08:41:45 20658 [Note] WSREP: Shifting CLOSED → DESTROYED (TO: 7699803)
2014-10-12 08:41:45 20658 [Note] WSREP: Flushing memory map to disk…
2014-10-12 08:41:45 20658 [Note] Binlog end
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘partition’

2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘ARCHIVE’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘BLACKHOLE’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘PERFORMANCE_SCHEMA’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_CHANGED_PAGES’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_DATAFILES’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_TABLESPACES’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN_COLS’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_FIELDS’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_COLUMNS’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_INDEXES’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_TABLESTATS’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_SYS_TABLES’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_FT_INDEX_TABLE’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_FT_INDEX_CACHE’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_FT_CONFIG’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_FT_BEING_DELETED’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_FT_DELETED’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_FT_DEFAULT_STOPWORD’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_METRICS’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_BUFFER_POOL_STATS’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE_LRU’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX_RESET’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_CMPMEM_RESET’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_CMPMEM’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_CMP_RESET’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_CMP’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_LOCK_WAITS’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_LOCKS’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘INNODB_TRX’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘XTRADB_RSEG’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘XTRADB_INTERNAL_HASH_TABLES’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘XTRADB_READ_VIEW’
2014-10-12 08:41:45 20658 [Note] Shutting down plugin ‘InnoDB’
2014-10-12 08:41:45 20658 [Note] InnoDB: FTS optimize thread exiting.
2014-10-12 08:41:45 20658 [Note] InnoDB: Starting shutdown…
2014-10-12 08:41:50 20658 [Note] InnoDB: Shutdown completed; log sequence number 5267192027
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘CSV’
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘MyISAM’
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘MRG_MYISAM’
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘MEMORY’
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘sha256_password’
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘mysql_old_password’
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘mysql_native_password’
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘wsrep’
2014-10-12 08:41:50 20658 [Note] Shutting down plugin ‘binlog’
2014-10-12 08:41:50 20658 [Note] /usr/sbin/mysqld: Shutdown complete

141012 08:41:51 mysqld_safe mysqld from pid file /var/lib/mysql/tree.flyheight.com.pid ended
141012 08:42:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141012 08:42:01 mysqld_safe Skipping wsrep-recover for e29f57c0-4ee8-11e4-9e34-f334de4c3867:7699803 pair
141012 08:42:01 mysqld_safe Assigning e29f57c0-4ee8-11e4-9e34-f334de4c3867:7699803 to wsrep_start_position
2014-10-12 08:42:01 0 [Note] WSREP: wsrep_start_position var submitted: ‘e29f57c0-4ee8-11e4-9e34-f334de4c3867:7699803’
2014-10-12 08:42:01 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

[COLOR=#FF0000]HERE IS THE WHOLE LOG: Dropbox - mysql.log - Simplify your life

IF YOU CAN NOT HELP PLEASE LET ME KNOW ASAP TO FIND ANOTHER SOLUTION.

THANKS
ERICK

Still went down again, can anyone help?

In the log I can see dozens of service shutdowns:

grep 'Normal shutdown' /var/tmp/mysql.log
2014-10-08 18:23:06 12724 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-08 18:42:31 13216 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-08 18:42:55 19088 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-11 05:57:09 19467 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-11 06:04:00 21143 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-11 06:05:23 23334 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-11 06:06:35 24372 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-11 06:08:16 25107 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-11 06:09:56 25904 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-11 06:20:30 26981 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-11 08:03:43 29955 [Note] /usr/sbin/mysqld: Normal shutdown
2014-10-12 08:41:43 20658 [Note] /usr/sbin/mysqld: Normal shutdown


which are not making much sense to me - were they caused by some automation tool or intentionally?
Also many failed SSTs in the history due to wrong sst auth it seems “–auth ‘(null)’”.
Then once proper SST was done (–auth ‘sstuser:s3cret’) we can see later many “WSREP: Provider paused at / WSREP: Provider resumed” which is usually indication of flush tables with read lock being issued, and then shutdown again. At the end there is some network issue between the nodes or just the second node was forcibly killed:
2014-10-12 05:15:26 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://192.99.19.154:4567
2014-10-12 05:17:43 20658 [Note] WSREP: (b045ef7f, ‘tcp://0.0.0.0:4567’) reconnecting to ce59deee (tcp://192.99.19.154:4567), attempt 90
node is in non-primary state as a result.

Given all these, I think there is something messing with the cluster rather then the cluster being the source of problem itself. “Normal shutdown” is not something that PXC server triggers on itself.