Not the answer you need?
Register and ask your own question!

percona server 5.5.33 response very slow and many error in mysql error file

chenzhe07chenzhe07 EntrantCurrent User Role Beginner
my database env:
MySQL Server | Percona-Server-5.5.33-rel31.1
# Percona Toolkit System Summary Report ######################
Date | 2017-08-30 11:46:02 UTC (local TZ: CST +0800)
Hostname | database9
Uptime | 370 days, 2:29, 2 users, load average: 0.08, 0.12, 0.16
System | Dell Inc.; PowerEdge R630; vNot Specified (Rack Mount Chassis)
Service Tag | FTD5DF2
Platform | Linux
Release | CentOS release 6.8 (Final)
Kernel | 2.6.32-642.4.2.el6.x86_64
Architecture | CPU = 64-bit, OS = 64-bit
Threading | NPTL 2.12
SELinux | Disabled
Virtualized | No virtualization detected
# Processor ##################################################
Processors | physical = 2, cores = 20, virtual = 40, hyperthreading = yes
Speeds | 40x2199.942
Models | 40xIntel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Caches | 40x25600 KB
# Memory ################################################## ###
Total | 125.9G


last sunday, one of my database response to application suddenly very slow, and the mysql thread running and cpu load is very high, such as:

[HTML]top - 21:16:17 up 367 days, 4:00, 0 users, load average: 906.18, 623.71, 300.
Tasks: 1149 total, 3 running, 1146 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 0.3%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 132042600k total, 131641800k used, 400800k free, 0k buffers
Swap: 41929644k total, 3606772k used, 38322872k free, 24683400k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
38391 mysql 20 0 111g 99g 5636 D 3403.4 78.8 147443:28 mysqld

TS 1503839784.028255754 2017-08-27 21:16:24
MemTotal: 132042600 kB
MemFree: 435984 kB
Buffers: 0 kB
Cached: 24602272 kB
SwapCached: 383772 kB[/HTML]


and have many repeate message in error log, such as:
[HTML]Aug 27 21:00:00 database9 mysqld-3309: Memory status:
Aug 27 21:00:00 database9 mysqld-3309: Non-mmapped space allocated from system: 0
Aug 27 21:00:00 database9 mysqld-3309: Number of free chunks:#011#011#011 1
Aug 27 21:00:00 database9 mysqld-3309: Number of fastbin blocks:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Number of mmapped regions:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Space in mmapped regions:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Maximum total allocated space:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Space available in freed fastbin blocks: 0
Aug 27 21:00:00 database9 mysqld-3309: Total allocated space:#011#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Total free space:#011#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Top-most, releasable space:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Estimated memory (with thread stack): 16515072
Aug 27 21:00:00 database9 mysqld-3309: Events status:
Aug 27 21:00:00 database9 mysqld-3309: LLA = Last Locked At LUA = Last Unlocked At
Aug 27 21:00:00 database9 mysqld-3309: WOC = Waiting On Condition DL = Data Locked
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: Event scheduler status:
Aug 27 21:00:00 database9 mysqld-3309: State : INITIALIZED
Aug 27 21:00:00 database9 mysqld-3309: Thread id : 0
Aug 27 21:00:00 database9 mysqld-3309: LLA : n/a:0
Aug 27 21:00:00 database9 mysqld-3309: LUA : n/a:0
Aug 27 21:00:00 database9 mysqld-3309: WOC : NO
Aug 27 21:00:00 database9 mysqld-3309: Workers : 0
Aug 27 21:00:00 database9 mysqld-3309: Executed : 0
Aug 27 21:00:00 database9 mysqld-3309: Data locked: NO
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: Event queue status:
Aug 27 21:00:00 database9 mysqld-3309: Element count : 0
Aug 27 21:00:00 database9 mysqld-3309: Data locked : NO
Aug 27 21:00:00 database9 mysqld-3309: Attempting lock : NO
Aug 27 21:00:00 database9 mysqld-3309: LLA : init_queue:139
Aug 27 21:00:00 database9 mysqld-3309: LUA : init_queue:149
Aug 27 21:00:00 database9 mysqld-3309: WOC : NO
Aug 27 21:00:00 database9 mysqld-3309: Next activation : never
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: Status information:
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: Current dir: /web/mysql/node3309/data/
Aug 27 21:00:00 database9 mysqld-3309: Running threads: 61 Stack size: 262144
Aug 27 21:00:00 database9 mysqld-3309: Current locks:
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcbe4c41e38:
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcc35bf82c0: read
Aug 27 21:00:00 database9 mysqld-3309: read : 0x7fcc01689398 (29404920:2);
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcc35bf8180: read
Aug 27 21:00:00 database9 mysqld-3309: read : 0x7fcc01687398 (29404920:2);
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcc35bf7f00: read
Aug 27 21:00:00 database9 mysqld-3309: read : 0x7fcd1bcc8398 (29404920:2);
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcc35bf7dc0: read
Aug 27 21:00:00 database9 mysqld-3309: read : 0x7fcc017b2398 (29404920:2); [/HTML]

it seems that mysql server receives many SIGHUP, but there is no operate to MySQL server by admin or application, I have no idea for this error message, thanks a lot for everyone help.

Comments

  • chenzhe07chenzhe07 Entrant Current User Role Beginner
    does linux kernel or glib send sighup to MySQL process, or MySQL server send sighup to itself?
  • chenzhe07chenzhe07 Entrant Current User Role Beginner
    I found the pt-stalk will trigger to send `debug` command to MySQL server, then MySQL will print many debug message:
    local mysql_error_log="$(awk '/log_error/{print $2}' "$d/$p-variables")"
       if [ -z "$mysql_error_log" -a "$mysqld_pid" ]; then
          mysql_error_log="$(ls -l /proc/$mysqld_pid/fd | awk '/ 2 ->/{print $NF}')"
       fi
    
       local tail_error_log_pid=""
       if [ "$mysql_error_log" ]; then
          log "The MySQL error log seems to be $mysql_error_log"
          tail -f "$mysql_error_log" >"$d/$p-log_error" &
          tail_error_log_pid=$!
    
          $CMD_MYSQLADMIN $EXT_ARGV debug 
       else
          log "Could not find the MySQL error log"
       fi
    

    and in percona server 5.5.33 version , COM_DEBUG will invoke mysql_print_status function:
    case COM_DEBUG:
        status_var_increment(thd->status_var.com_other);
        if (check_global_access(thd, SUPER_ACL))
          break;                                    /* purecov: inspected */
        mysql_print_status();
        general_log_print(thd, command, NullS);
        my_eof(thd);
        break;
    
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.